
You've built a high-performance Python application using asyncio, but you've noticed a bottleneck. The culprit? Your logging. This is a common issue as logging seems to be a a trivial operation, but it can be a significant performance killer.

By default, Python's logging module uses blocking handlers, which means your application pauses every time it writes a log message.
Before we delve right in, it's important to say that in most cases, synchronous logging is perfectly fine. It's simple, reliable, and the default behavior. It works well for applications that aren't I/O-bound or where logging operations are infrequent.

But if you're building a high-performance I/O-bound application, you'll want to consider asynchronous logging. This article will walk you through why traditional logging can slow down your code and how to implement a simple yet effective asynchronous logging solution using Python's standard library.

Why do we even need to care about async logging? Here's a simple example:

```python
import logging
import time

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(message)s')

def process_data(item):
    """Simulates a blocking data processing task."""
    logging.info(f"Starting to process item: {item}")
    time.sleep(0.1)  # Simulates work
    logging.info(f"Finished processing item: {item}")

if __name__ == "__main__":
    data_items = range(5)
    for item in data_items:
        process_data(item)

    logging.info("All data processed.")
```

When you move into a high-concurrency environment, this approach falls apart.

With the Async logging approach, instead of writing directly to a file or a remote service, log messages are placed in a queue and processed in a separate thread. This allows your primary application code to continue executing without waiting for the I/O to complete.

Imagine a web server handling thousands of concurrent requests. If each request logs synchronously, the application spends valuable time waiting for log files to be written. This increases latency and drastically reduces the number of requests it can handle per second. Asynchronous logging solves this by offloading the logging work to a background thread.

The good part, is you don't need a third-party library to achieve this. Python's standard library provides the `logging.handlers.QueueHandler` and `logging.handlers.QueueListener` for this exact purpose.

First, create a Queue, then use QueueHandler.
By using these, you'll create a thread-safe queue that will hold the log messages, then, the your main application logger will use the QueueHandler to put log records into the queue. This is a non-blocking operation, so your app remains fast.

Lastly, use QueueListener. It runs in a separate thread. It reads messages from the queue and passes them to a "worker handler" (e.g., a FileHandler) that performs the actual, slow I/O operation.

Here's another code example:

```python
import logging
from logging.handlers import QueueHandler, QueueListener
from queue import Queue
import threading
import time
import asyncio

# 1. Setup a queue for logging
log_queue = Queue(-1)

# 2. Setup the worker handler that will actually write the logs
worker_handler = logging.FileHandler("async_app.log")
worker_handler.setFormatter(logging.Formatter('%(asctime)s - %(message)s'))

# 3. Setup the listener in a separate thread
listener = QueueListener(log_queue, worker_handler)
listener.start()

# 4. Setup the main application logger to use the QueueHandler
async_handler = QueueHandler(log_queue)
logger = logging.getLogger('my_async_app')
logger.addHandler(async_handler)
logger.setLevel(logging.INFO)

async def handle_request(request_id):
    """Simulates a fast, asynchronous request handler."""
    logger.info(f"Async: Starting request {request_id}")
    await asyncio.sleep(0.01)  # Simulate non-blocking work
    logger.info(f"Async: Finishing request {request_id}")

async def main():
    """Simulates a high-concurrency environment."""
    tasks = [handle_request(i) for i in range(10)]
    await asyncio.gather(*tasks)

if __name__ == "__main__":
    start_time = time.time()
    asyncio.run(main())
    end_time = time.time()
    logger.info(f"Total time elapsed: {end_time - start_time:.4f} seconds")

    # It's important to stop the listener gracefully
    listener.stop()
```

Happy logging!


Focusing on high performance I/O bound Python Applications? Do not forget Asynchronous logging to prevent bottlenecks in your app.

Python Async Logging


Couple of months ago at work, we thought about a testing solution for handling the testability of our common react components being used in many different applications. While it might be trivial these days achieving it with Storybook (using vitest or even playwright ct library), working with legacy Storybook v7, I found out that it might require some tweaking. The annoying part was actually getting it working both in Storybook's interactions tab (which is a storybook addon) and in the CLI as in almost any common test runner.

When you're stuck on Storybook v7 (often due to enterprise constraints), setting up automated interaction testing becomes a bit more complex than the modern v8+ experience. Some of the challenges I faced were limited tooling compatibility, some extra configurations, error messages and debugging, and more.

The goal was simple: execute the interaction tests defined in `play` functions directly from the CLI or from Storybook's UI.

## The Component Context: Complex Interactions Need Complex Testing

To understand why this testing setup was so crucial, let me briefly introduce the component that drove this need. A Suggestion component which is a common component in many applications, and has a lot of different states and interactions.

- **Multi-step user flows** (type → filter → select → validate)
- **Keyboard navigation** through options with proper focus management
- **Dynamic option management** (add, edit, delete suggestions in real-time)
- **Complex validation states** with async feedback
- **Multiple selection modes** with chip management
- **Loading states** during async operations

Before finding the solution, I tried several approaches that are commonly recommended but fail with v7:

### Attempt 1: @storybook/addon-vitest

```bash
npm install @storybook/addon-vitest
```

I cannot emphasize on how vitest is a quality of life game changer in terms of performance, therefore it was my first choice, but this addon is designed for Storybook v8+ and caused immediate startup crashes. Sadly the addon expects internal APIs that don't exist in v7's architecture.

### Attempt 2: Newer @storybook/test-runner Versions

```bash
npm install @storybook/test-runner@latest
```

Runtime errors and compatibility issues. The newer versions expect v8 features and fail with cryptic Playwright errors (which is its dependency).

---

After extensive trial and error, I found the sweet spot: **@storybook/test-runner version 0.13.0**. This specific version is the last one with solid Storybook v7 compatibility.

### Complete Installation Steps

1. **Install the test runner and dependencies**:

```bash
yarn add -D @storybook/test-runner@0.13.0
yarn add -D @storybook/addon-interactions@^7.6.0
yarn add -D @storybook/jest@^0.2.0
yarn add -D @storybook/testing-library@^0.2.0
```

2. **Add the script to package.json**:

```json
{
  "scripts": {
    "test:component": "test-storybook"
  }
}
```

3. **Eject the configuration** (critical step):

```bash
yarn test:component --eject
```

### Why This Version Matters

Version 0.13.0 is crucial because:

- **Last v7-compatible release** before the v8 transition
- **Stable Playwright integration** without modern API dependencies
- **Proven Jest configuration** that works with v7's architecture
- **Mature error handling** for complex interaction scenarios
- **No dependency on internal v8 APIs** that don't exist in v7

Because the default configuration doesn't work out of the box, you must eject and customize:

This creates the essential `test-runner-jest.config.js`:

```javascript
const { getJestConfig } = require('@storybook/test-runner')

/**
 * @type {import('@jest/types').Config.InitialOptions}
 */
module.exports = {
  // The default configuration comes from @storybook/test-runner
  ...getJestConfig(),
  /** Add your own overrides below
   * @see https://jestjs.io/docs/configuration
   */
  testEnvironmentOptions: {
    'jest-playwright': {
      collectCoverage: true,
      contextOptions: {},
      exitOnPageError: false, // Critical for preventing non-critical page errors from failing tests
    },
  },
}
```

Without this configuration, you'll encounter the infamous error:

```
TypeError: Cannot read properties of undefined (reading 'goto')
```

This happens because the Playwright `page` object isn't properly initialized in the default v7 environment setup.

### Essential Storybook Configuration

Your `.storybook/main.ts` must include the interactions addon:

```typescript
import type { StorybookConfig } from '@storybook/react-vite'

const config: StorybookConfig = {
  stories: [
    '../src/**/*.stories.mdx',
    '../src/**/*.stories.@(js|jsx|mjs|ts|tsx)',
  ],
  addons: [
    '@storybook/addon-links',
    '@storybook/addon-essentials',
    '@storybook/addon-docs',
    '@storybook/addon-interactions', // ← Critical for interaction testing
  ],
  framework: {
    name: '@storybook/react-vite', // or '@storybook/react-webpack5'
    options: {},
  },
}
export default config
```

### Required Dependencies

Your `package.json` needs these exact versions:

```json
{
  "devDependencies": {
    "@storybook/addon-interactions": "^7.6.0",
    "@storybook/jest": "^0.2.0",
    "@storybook/test-runner": "0.13.0",
    "@storybook/testing-library": "^0.2.0"
  }
}
```

**Version compatibility is critical**:

- `@storybook/test-runner@0.13.0` - Last v7-compatible version
- `@storybook/jest@0.2.0` - Provides Jest matchers for Storybook
- `@storybook/testing-library@0.2.0` - Testing utilities for interactions

### Story File Setup

Your story files need proper imports for interaction testing:

```typescript
import type { Meta, StoryObj } from '@storybook/react'
import {
  configure,
  screen,
  within,
  userEvent,
} from '@storybook/testing-library'
import { expect } from '@storybook/jest'

// Configure testing library
configure({ testIdAttribute: 'data-testid' })

// Your component and story setup
const meta: Meta<typeof YourComponent> = {
  title: 'Components/YourComponent',
  component: YourComponent,
}

export default meta
type Story = StoryObj<typeof YourComponent>
```

## How the Test Runner Works Under the Hood

Understanding the mechanics helps debug issues and optimize performance:

1. **Browser Automation**: Uses Playwright to launch a headless browser (Chromium by default)
2. **Story Navigation**: Automatically discovers and navigates to each story's dedicated URL
3. **Test Execution**:
   - Runs a basic "smoke test" for every story (ensures rendering without errors) - By default.
   - Executes `play` function interactions for stories that have them
4. **Result Aggregation**: Collects results from all tests and reports them to the CLI

![test runner component tests](/images/storybook-test-runner.png)

## Writing Interaction Tests

Here's the complete test structure that handles all the complex scenarios:

```typescript
export const InteractionTests: StoryObj<typeof InteractionTestComponent> = {
  render: () => <InteractionTestComponent />,
  play: async ({ canvasElement, step }) => {
    const canvas = within(canvasElement)

    await step('Interactive Typing Test', async () => {
      const wrapper = await canvas.findByTestId('default-suggestion-input')
      const input = within(wrapper).getByRole('textbox')
      await userEvent.clear(input)
      await userEvent.type(input, 'Marg')
      await expect(input).toHaveValue('Marg')
    })

    await step('Keyboard Navigation Test', async () => {
      const wrapper = await canvas.findByTestId('default-suggestion-input')
      const input = within(wrapper).getByRole('textbox')
      await userEvent.click(input)
      await expect(input).toHaveFocus()
      await userEvent.keyboard('{ArrowDown}')
      await userEvent.keyboard('{ArrowDown}')
      await userEvent.keyboard('{ArrowUp}')
      await userEvent.keyboard('{Enter}')
      await expect(input).toHaveFocus()
    })

    await step('Multiple Selection Test', async () => {
      const wrapper = await canvas.findByTestId(
        'states-suggestion-multiple-input',
      )
      const input = within(wrapper).getByRole('textbox')
      await userEvent.click(input)

      await userEvent.type(input, 'React')
      await userEvent.keyboard('{ArrowDown}')
      await userEvent.keyboard('{Enter}')

      await userEvent.type(input, 'Vue')
      await userEvent.keyboard('{ArrowDown}')
      await userEvent.keyboard('{Enter}')

      const reactChip = await canvas.findByTestId(
        'states-suggestion-multiple-input.tag.react',
      )
      const vueChip = await canvas.findByTestId(
        'states-suggestion-multiple-input.tag.vue',
      )

      expect(reactChip).toBeInTheDocument()
      expect(vueChip).toBeInTheDocument()
    })

    await step('Component States Test', async () => {
      const loadingWrapper = await canvas.findByTestId(
        'loading-state-suggestion',
      )
      const progress = within(loadingWrapper).getByRole('progressbar')
      expect(progress).toBeInTheDocument()

      const errorWrapper = await canvas.findByTestId('error-state-suggestion')
      const warningIcon = within(errorWrapper).getByText('warning')
      expect(warningIcon).toBeInTheDocument()
    })

    await step('Accessibility Test', async () => {
      const wrapper = await canvas.findByTestId('default-suggestion-input')
      const input = within(wrapper).getByRole('textbox')
      expect(input).toHaveAttribute('aria-autocomplete', 'list')
      const label = await canvas.findByText('Default Suggestion')
      expect(label).toBeInTheDocument()
    })
  },
}
```

## The Execution Workflow: Two-Terminal Setup

The testing workflow requires careful orchestration:

```bash
# Terminal 1: Start Storybook (and keep it running)
yarn storybook
# Wait for "Local: http://localhost:6006/"

# Terminal 2: Run the tests
yarn test:component
```

**Critical timing**: The test runner expects Storybook to be fully loaded before starting. Rushing this step causes connection errors.

## Debugging Test Failures in Storybook v7

Storybook v7's error messages can be overwhelming. Here's a real failure example and how to decode it (Just changed the role to a wrong role for the sake of the example):

```bash
FAIL   browser: chromium  src/schemaRender/stories/base/suggestion/index.stories.tsx
  Development/Base/Suggestion
    InteractionTests
      ✕ play-test (617 ms)

  ● Development/Base/Suggestion › InteractionTests › play-test

    page.evaluate: StorybookTestRunnerError:
    An error occurred in the following story. Access the link for full output:
    http://127.0.0.1:6006/?path=/story/development-base-suggestion--interaction-tests&addonPanel=storybook/interactions/panel

    Message:
     Unable to find an accessible element with the role "textbo"

    Here are the accessible roles:
      textbox:
      Name "":
      <input
        id="someidinput"
        placeholder=""
        type="text"
        value="Pizza"
      />
```

### Debugging Strategy:

1. **Use the provided URL**: The error includes a direct link to the failing story
2. **Check element selectors**: The error shows available roles - use these for your queries
3. **Verify timing**: Often failures are due to elements not being ready
4. **Test manually first**: Run the interaction manually in Storybook before automating

### The Screen vs Canvas Problem

A common mistake in Storybook v7 interaction tests:

```typescript
// ❌ Wrong - causes warnings
const element = screen.getByRole('textbox')

// ✅ Correct - use canvas
const canvas = within(canvasElement)
const element = canvas.getByRole('textbox')
```

The `screen` object works but generates warnings about Testing Library best practices.

## Storybook v7 vs v8: What You're Missing

If you're stuck on v7, here's what v8+ offers that you can't easily replicate:

- **Built-in Vitest integration** with `@storybook/addon-vitest`
- **Improved error messages** with better stack traces
- **Faster test execution** through optimized browser handling
- **Better TypeScript support** in test configurations
- **Streamlined setup** with working defaults

But with the setup I've outlined, you can achieve 90% of the functionality with v7.

So what we achieved here?

### Automated Test Coverage

- **Interaction testing** for complex user flows
- **Accessibility validation** built into every test
- **Regression prevention** through CI integration
- **Visual testing** combined with functional testing

### Development Benefits

- **Faster debugging** with direct Storybook links in failures
- **Confident refactoring** knowing interactions are covered
- **Documentation through tests** showing expected behavior
- **Onboarding acceleration** for new team members

## Integrating with CI/CD

For production use, add this to your CI pipeline:

```yaml
# .github/workflows/storybook-tests.yml
name: Storybook Tests
on: [push, pull_request]

jobs:
  interaction-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install dependencies
        run: yarn install --frozen-lockfile

      - name: Build Storybook
        run: yarn build-storybook --quiet

      - name: Serve Storybook and run tests
        run: |
          npx concurrently -k -s first -n "SB,TEST" -c "magenta,blue" \
            "npx http-server storybook-static --port 6006 --silent" \
            "npx wait-on http://127.0.0.1:6006 && yarn test:component"
```

## Best Practices for Storybook v7 Testing

### 1. Test Structure

```typescript
// ✅ Good: Clear steps with descriptive names
await step(
  'User can select multiple options with keyboard navigation',
  async () => {
    // Test implementation
  }
)

// ❌ Bad: Vague step descriptions
await step('Test selection', async () => {
  // Test implementation
})
```

### 2. Element Selection

```typescript
// ✅ Good: Use data-testid attributes
const input = canvas.getByTestId('suggestion-input')

// ❌ Bad: Fragile CSS selectors
const input = canvas.querySelector('.someCssClass')
```

### 3. Async Handling

```typescript
// ✅ Good: Proper async waiting
await userEvent.type(input, 'search term')
await waitFor(() => expect(dropdown).toBeVisible())

// ❌ Bad: Not waiting for async operations
userEvent.type(input, 'search term')
expect(dropdown).toBeVisible() // Might fail due to timing
```

![storybook interactions tests in storybook ui](/images/interactions-tests.gif)

---

That's it.
I would love to know and hear if you have any other tips or tricks for Storybook v7 testing or even better solutions for such cases.


Using storybook 7? Having trouble implementing component/interaction testing? This is the guide for you.

Storybook Component/Interactions Testing


## The Problem with Flaky Tests

We've all been there - you have a test suite that works perfectly in isolation, but occasionally fails due to external factors beyond your control (like a third-party service being down). Maybe it's a network hiccup, a temporary service unavailability, or leftover test data from previous runs. While Playwright offers built-in retry mechanisms, sometimes you need more control over what happens between retry attempts.

In this post, I'll show you how to leverage Playwright's retry system with custom fixtures to perform cleanup operations between retries, ensuring each retry attempt starts with a clean slate.

## Understanding Playwright's Built-in Retry Mechanism

Playwright provides a simple retry configuration that you can set in your `playwright.config.ts`:

```typescript
// playwright.config.ts
export default defineConfig({
  // Retry failed tests up to 2 times
  retries: process.env.CI ? 2 : 0,

  // Or configure per project
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
      retries: 2,
    },
  ],
})
```

While this works great for simple flaky tests, it doesn't help when your test failures are caused by leftover data or state from previous attempts. This is where custom retry handling becomes invaluable.

## The Solution: Custom Retry Fixture

The key insight is to create a custom fixture that can detect when a test is being retried and perform necessary cleanup operations. Here's how I implemented this pattern in one of my projects:

### Step 1: Create a RetryHandler Class

First, let's create a `RetryHandler` class that manages the cleanup logic.
In this example, we are cleaning up test data that could be stored in various systems like MongoDB, Redis, files, or other external services.

```typescript
// utils/retry-handler.ts
import { logger } from '@moonactive/admin-common-logger'
import { Fixtures } from 'fixture/base.fixture'
import { TestInfo } from 'playwright-decorators'

export class RetryHandler {
  async retryHandler(
    testInfo: TestInfo,
    context: Fixtures, // Where you pass other fixtures you want to clean up
    testData: any[] // Where you pass the test data you want to clean up
  ): Promise<void> {
    // Only perform cleanup if this is a retry attempt (retry > 0)
    if (testInfo.retry > 0) {
      await RetryHandler.retryCleaner(context, testData)
    }
  }

  static async retryCleaner(
    context: Partial<Fixtures>,
    testData: any[]
  ): Promise<void> {
    try {
      const { mongo, redis } = context
      const dataIds = testData.map(
        item => item.id || item._id || item.identifier
      )

      const contextDetails = {
        mongo: !!mongo,
        redis: !!redis,
      }

      logger.info(
        `Triggering retry cleanup for data: ${dataIds}, context: ${JSON.stringify(
          contextDetails
        )}`
      )

      // Clean up database records if MongoDB is used
      if (mongo) {
        await mongo.deleteTestDataFromDB(testData)
      }

      // Clean up Redis records if Redis is used
      if (redis) {
        await redis.deleteTestDataFromRedis(testData)
      }

      if (!mongo && !redis) {
        logger.error('Missing context for retryCleaner')
        throw new Error('Missing context for retryCleaner')
      }
    } catch (err) {
      logger.error(`Failed to clean up after retry: ${err}`)
      throw err
    }
  }
}
```

### Step 2: Integrate with Playwright Fixtures

Next, integrate the RetryHandler into your Playwright fixture system:

```typescript
// fixture/base.fixture.ts
import { RetryHandler } from '../utils/retry-handler'
// ... other imports

export type Fixtures = {
  logger: Types.SomeLogger
  app: App
  expect: typeof expect
  mongo: DatabaseHelper
  redis: RedisHelper
  bq: BQHelper
  api: APIHelper
  sqs: SqsHelper
  retry: RetryHandler // Our custom retry handler
}

export const test = base.extend<Fixtures>({
  // ... other fixtures

  retry: async ({}, use) => {
    const retryHandler = new RetryHandler()
    await use(retryHandler)
  },

  // ... other fixture implementations
})
```

### Step 3: Use the Retry Handler in Tests

Now you can use the retry handler in your tests:

```typescript
// tests/example.spec.ts

test('Create and validate test data', async ({
  app,
  api,
  mongo,
  redis,
  retry,
}, testInfo) => {
  // Define test data that might need cleanup
  const testData = [
    { id: 'test-item-1', name: 'Test Item 1', type: 'example' },
    { id: 'test-item-2', name: 'Test Item 2', type: 'example' },
  ]

  // Call retry handler at the beginning of the test
  // This will clean up any leftover data if this is a retry attempt
  await retry.retryHandler(testInfo, { mongo, redis }, testData)

  // Your actual test logic
  await app.createTestItems(testData)

  // Validate the created items
  await api.validateTestItems(testData)

  // Perform additional test actions
  await app.performActionsOnItems(testData)
})
```

## How Does It Work?

The magic happens in the `testInfo.retry` property. Playwright automatically increments this value for each retry attempt:

- **First attempt**: `testInfo.retry = 0` (no cleanup performed)
- **First retry**: `testInfo.retry = 1` (cleanup is performed)
- **Second retry**: `testInfo.retry = 2` (cleanup is performed again)

This ensures that cleanup only happens when needed, avoiding unnecessary operations on the first test run.

## Key Benefits

### 1. **Intelligent Cleanup**

Only performs cleanup operations when actually retrying, not on the initial test run.

### 2. **Flexible Context**

You can pass different fixtures and resources based on what your specific test needs to clean up.

### 3. **Comprehensive Coverage**

The example shows cleanup across multiple systems (database, GitHub, configuration), but you can adapt it to your needs.

### 4. **Proper Error Handling**

Includes logging and error handling to help debug cleanup issues.

### 5. **Type Safety**

Leverages TypeScript for better IDE support and catch errors at compile time.

## Advanced Usage Patterns

### Multi-Service Cleanup

You can extend this pattern to handle different types of cleanup based on the test context:

```typescript
await retry.retryHandler(
  testInfo,
  {
    mongo,
    redis,
    elasticsearch,
    fileSystem,
    api,
  },
  testData
)
```

### Conditional Cleanup

Add logic to perform different cleanup based on test metadata:

```typescript
async retryHandler(
  testInfo: TestInfo,
  context: Fixtures,
  data: any[],
  options?: { skipRedis?: boolean; skipDB?: boolean; skipFiles?: boolean }
): Promise<void> {
  if (testInfo.retry > 0) {
    await RetryHandler.retryCleaner(context, data, options)
  }
}
```

### Custom Retry Limits

You can even implement custom retry logic based on specific conditions:

```typescript
async retryHandler(
  testInfo: TestInfo,
  context: Fixtures,
  data: any[],
  maxRetries: number = 2
): Promise<void> {
  if (testInfo.retry > 0 && testInfo.retry <= maxRetries) {
    await RetryHandler.retryCleaner(context, data)
  } else if (testInfo.retry > maxRetries) {
    logger.warn(`Exceeded max retries (${maxRetries}), skipping cleanup`)
  }
}
```

## Best Practices

1. **Call Early**: Always call the retry handler at the very beginning of your test, before any test actions.

2. **Be Specific**: Only clean up the specific data your test creates to avoid affecting other tests.

3. **Log Everything**: Include comprehensive logging to help debug retry scenarios.

4. **Handle Failures**: Make sure cleanup failures don't prevent the retry from proceeding.

5. **Test Your Cleanup**: Consider writing separate tests to verify your cleanup logic works correctly.

## Conclusion

This pattern transforms Playwright's basic retry mechanism into a powerful tool for handling complex test scenarios. By combining custom fixtures with intelligent cleanup logic, you can ensure that flaky tests get a truly fresh start on each retry attempt.

The approach is particularly valuable for integration tests that interact with external services, databases, or file systems where leftover state can cause cascading failures.

Have you implemented similar retry patterns in your test suites? I'd love to hear about your experiences and any creative solutions you've developed!


Do you have a flaky test that is out of your control? Do you have some functionality you need to apply between retries? This is the tip for you.

Playwright Tips: Leverage Playwright's Retry Mechanism as a Custom Fixture


So a little bit context first - As our test suites grow in complexity and execution parallelism, pinpointing the origin of specific network requests and responses becomes increasingly crucial dude to some random flakiness.

In our Playwright automation framework, we recently encountered a scenario where we needed to intercept a response from a particular worker instance in our parallel test execution. This led us to devise a solution leveraging Playwright's request interception capabilities and custom headers.

When running Playwright tests in parallel using multiple workers, identifying which worker initiated a specific network request can be challenging. This is especially true when dealing with APIs that might behave slightly differently based on the context of the worker or the test being executed.
In our case, we needed to verify the response of a specific API call triggered by an action within a particular test running on a specific worker. Using the builtin `page.waitForResponse()` with a URL matcher wasn't sufficient because multiple workers could potentially trigger the same API call around the same time. We needed a more precise way to target the response we were interested in.

Therefore, our approach was to inject a custom header into the outgoing request from a specific worker instance. This unique header would then allow us to identify the corresponding response. Playwright's `page.route()` API provides a powerful mechanism to intercept and modify requests before they are sent.

Here's an example of how we implemented this solution:

```typescript
import { Page, Response } from '@playwright/test'

async function waitForResponseOfSpecificWorkerInstance(
  page: Page,
  partialRoute: string,
  uniqueIdentifier: string,
  timeout = 21000
): Promise<Response> {
  const regexRoute = new RegExp(partialRoute)

  await page.route('**/*', route => {
    const url = route.request().url()
    const urlObj = new URL(url)
    const pathname = urlObj.pathname

    if (regexRoute.test(pathname)) {
      const newHeaders = { ...route.request().headers() }
      newHeaders['X-Unique-Identifier'] = uniqueIdentifier
      route.continue({ headers: newHeaders })
    } else {
      route.continue()
    }
  })

  const response = await page.waitForResponse(
    response => {
      const requestHeaders = response.request().headers()
      return (
        regexRoute.test(response.url()) &&
        requestHeaders['x-unique-identifier'] === uniqueIdentifier
      )
    },
    { timeout }
  )
  return response
}
```

Let's break down what this simple function does:

```typescript
waitForResponseOfSpecificWorkerInstance(page, partialRoute, uniqueIdentifier, timeout):
```

- Takes the Playwright Page object, a partialRoute (a string or regular expression to match the URL), a uniqueIdentifier (a string to identify the worker instance), and an optional timeout as arguments.
- Returns a Promise that resolves with the intercepted Response object.

```typescript
const regexRoute = new RegExp(partialRoute)
```

Creates a regex from the partialRoute variable to allow for flexible URL matching.

```typescript
await page.route('**/*', route => { ... });
```

This is the heart of the request interception. `page.route('**/*')` intercepts all network requests made by the page.
The provided callback function is executed for each intercepted request.

```typescript
const url = route.request().url()
const pathname = new URL(url).pathname

if (regexRoute.test(pathname)) { ... }
```

Extracts the URL and then specifically the pathname from the intercepted request. We often focus on the pathname for route matching.
Then we check if the pathname of the current request matches the provided partialRoute.

```typescript
const newHeaders = { ...route.request().headers() }

newHeaders['X-Unique-Identifier'] = uniqueIdentifier
```

Creates a copy of the existing request headers. It's important to copy the headers to avoid modifying the original object directly. Then we inject our custom header. We add a new header named `X-Unique-Identifier` with the value of the uniqueIdentifier variable that was passed to the function as an argument.

```typescript
route.continue({ headers: newHeaders })
else { route.continue() }

```

Allows the intercepted request to continue to the server, but now with our newly added custom header.
If the request URL doesn't match our target partialRoute, we simply allow the request to continue without modification.

```typescript
const response = await page.waitForResponse(response => { ... }, { timeout })

```

This is where we wait for the specific response we are interested in. The callback function within `waitForResponse` checks two conditions:

- `regexRoute.test(response.url())` Ensures the response URL matches our target route.
  `requestHeaders['x-unique-identifier'] === uniqueIdentifier` Checks if the request that led to this response had our custom X-Unique-Identifier header with the expected value.

And finally, once a matching response is found (both URL and custom header match), the promise resolves with the Response object and returns it.

Now, lets take a look at how we can use this function in a test:

```typescript
import { test, expect } from '@playwright/test'
import { waitForResponseOfSpecificWorkerInstance } from './utils' // Assuming the function is in a utils.ts file

test('Edit refresh lock', async ({ app, api }, testInfo) => {
  // Assuming 'app' and 'api' is a fixture object containing your page objects and api helper function like waitForResponseOfSpecificWorkerInstance

  const responsePromise = api.waitForResponseOfSpecificWorkerInstance(
    app.myPage,
    '/api/v1/example/',
    `worker_${testInfo.workerIndex}`
  )

  // Perform some action that will trigger the API call

  const response = await responsePromise
  expect(response.status()).toBe(200)
})
```

In this example:

- We use `testInfo.workerIndex` (testInfo is a built-in fixture of Playwright) to get a unique identifier for the current worker running the test. This ensures that each parallel test execution will have a distinct X-Unique-Identifier header.
- We call our `waitForResponseOfSpecificWorkerInstance` function before performing the action that triggers the API call.
- We pass the page object, the partial route of the API endpoint we're interested in (/api/v1/example/), and the unique worker identifier as a string.
- The test then proceeds to perform the action on the browser.
- We await the `responsePromise`, which will resolve only when a response from the /api/v1/example/ endpoint is received and its corresponding request had the X-Unique-Identifier header matching the current worker's index.

By doing that simple trick, we can now precisely target and handle responses from specific worker instances in our parallel test execution, reduce flakiness (even in parallel execution) and enhanced debugging abilities.


Target specific API responses with Playwright parallel execution using custom headers for reliable automation.

Playwright Tips: Using Custom Headers in Playwright for Worker-Specific Response Handling


## The data-test-id Debate: To Use or Not to Use?

For those of us who have a passion for code testing, automated testing plays a crucial role in web development. When talking about end-to-end or functional testing of web applications, one technique that often sparks a debate is the use of data-test-id attributes in HTML elements. This article delves into the pros and cons of using these attributes.

### So What Are data-test-id Attributes?

data-test-id is a custom HTML attribute used to uniquely identify elements for automated testing. It serves as a stable hook for testing tools, allowing them to locate and interact easily with specific elements regardless of changes to styling, layout, structure, or any other non-functional aspects of the application.

So let's start by examining the arguments for and against using data-test-id attributes in web applications.

### The Case For data-test-id

1. **Stability and Resilience:**

Using data-test-id provides a stable selector, mitigating the risk of tests breaking or flakiness due to changes in CSS classes or DOM structure, which usually happens quite frequently on modern web applications being constantly developed.
It is also especially vital in complex dynamic applications with frequent updates and state changes.

2. **Insulation from Style Changes:**

Due to the fact that almost any new web application is being developed using some kind of a framework or a library (for example, tailwind), CSS classes are often subject to change for styling purposes in a more frequent way then it used to be changed in the past. Relying on them for testing often creates flaky tests.

3. **Improved Testing Efficiency:**

Whatever you think on data-test-id usage, you can't deny its the most efficient method of building an automation testing framework from scratch.
Due to the fact that it is a custom attribute, it is not used for any other purpose other than testing, which makes it a perfect candidate for testing and of course it makes our locators more quicker to find with less of an effort and time to figure out which selectors we should use.

4. **Improved Debugging:**

During test failures, data-test-id attributes make it easier to pinpoint the problematic element, streamlining the debugging process which saves us time and effort.

5. **Dynamic Addition and Removal:**

data-test-id attributes can be dynamically added or removed based on the environment using build tools, server-side rendering, or client-side JavaScript. This allows for their removal in production builds, addressing potential performance and security concerns (spoiler for some of the cons).

### The Case Against data-test-id

1. **Potential Clutter and Performance Impact:**

Leaving data-test-id attributes in production can increase the size of the DOM, potentially leading to slight performance degradation.

2. **Maintenance Overhead:**

Maintaining unique and meaningful data-test-id values can become challenging in large applications, requiring consistent naming conventions and careful management.

3. **Security Concerns:**

On some cases, if left in production, data-test-id attributes might inadvertently expose sensitive information about the application's structure or functionality for potential attackers.

4. **Scraping:**

Simply put, these attributes can make web scraping much easier. While true, it is not the only way to scrape a web application.

5. **Discourage of User Facing Locators:**

Using data-test-id encourages testing implementation details rather than user behavior.
For example, if you look at [Playwright's documentation, under best practices,](https://playwright.dev/docs/best-practices#prefer-user-facing-attributes-to-xpath-or-css-selectors) they encourage using selectors that are visible to the user, such as text content, and other user-visible attributes. Although we need to be fair and say, they were mainly aiming at the usage of CSS selectors, and XPath which these days are considered as unreliable.
You can find similar recommendations in the React Testing Library.

---

To conclude, let's go over of some of the **best practices** and considerations when using data-test-id attributes.

**Naming Conventions:** Establish clear and consistent naming conventions for data-test-id values (e.g., component-element-action).

**Environment-Specific Handling:** Use build tools or server-side rendering to remove data-test-id attributes in production.

**Balanced Approach:** Use data-test-id judiciously, complementing it with user-centric testing approaches that target user-visible attributes.

**Avoid Over-Reliance:** Do not rely solely on data-test-id. Test user flows and user visible information.


In this article, I will discuss the pros and cons of using test-ids in your web applications projects.

The Case For and Against test-id Usage


I'm using Mac for several years now as my daily driver at different workplaces, and during those years I've selected a few tools that goes with me anywhere.

I will skip the obvious ones such as code editors / terminals, etc and focus on the tools that I think are less known but can improve your productivity.

### Homebrew

Can't define [homebrew](https://brew.sh/) as less known, but I still think it's worth mentioning because its the first thing I install when I start working on a Mac.

Homebrew is a package manager for macOS (not only) that allows you to install and manage software packages from the command line effortlessly. By the way, it's also a great way to install tools that are not available in the App Store.
It is free and open source, and it's a must-have tool for any developer working on a Mac in my point of view.

Some of the tools recommended in this article can be installed using Homebrew.

### Docker Desktop

[Docker desktop](https://www.docker.com/products/docker-desktop/) is the easiest and most strait forward way I know to run containers, monitor and manage locally on your Mac.
It's a great tool for developers who want to run and test their applications in a containerized environment.
I use it daily to run databases, message brokers, and other services that I need for my development environment.

### Raycast

[Raycast](https://www.raycast.com/) is a productivity tool that helps you to perform tasks faster on your Mac. It's a great tool for developers who want to automate repetitive tasks and access their favorite tools and services quickly using custom scripts, hotkeys and more.

### Jumpcut

[Jumpcut](https://snark.github.io/jumpcut/) is a clipboard manager for macOS that allows you to access your clipboard history and paste previously copied items with ease. It's a great tool for developers who want to copy and paste multiple items quickly without losing track of their clipboard history.
You can browse your clipboard history using hotkeys, and it supports text, images, and other types of content.

### AltTab

[AltTab](https://alt-tab-macos.netlify.app/) is a macOS app that enhances the built-in app switcher with more features and customization options. It allows you to switch between apps and windows more efficiently using keyboard shortcuts and gestures. It's a great tool for developers who want to navigate between apps and windows quickly and easily.
Personally, I replaced the default macOS app switcher with AltTab, and I find it more intuitive and user-friendly.
For example, when you want to switch between windows of the same app, you can use the arrow keys to navigate and select the desired window while it provides you a preview of the window you're about to switch to.

### UTM

[UTM](https://mac.getutm.app/) is a macOS app that allows you to create and manage virtual machines on your Mac. It's a great tool for developers who want to run multiple operating systems and environments on their Mac for testing and development purposes. UTM supports a wide range of operating systems and platforms, and it provides one of the most user-friendly interfaces I encountered up until now for creating and managing virtual machines.

### TinkerTool

[TinkerTool](https://www.bresink.com/osx/TinkerTool.html) is a macOS app that allows you to customize and tweak various system settings and preferences on your Mac. It's a great tool for developers who want to personalize their Mac experience and optimize their workflow. TinkerTool provides a wide range of options and settings that are not available in the standard macOS preferences, and it allows you to customize your Mac to suit your needs and preferences.

### Stats

[Stats](https://github.com/exelban/stats) is a macOS system monitor that provides real-time information about your Mac's performance and resource usage. It's a great tool for developers who want to keep an eye on their system's performance and identify any bottlenecks or issues that may affect their work. Stats displays information about CPU, memory, disk, network, and battery usage in a user-friendly interface, and it allows you to monitor your Mac's performance at a glance while also providing the option to customize the display and layout of the information.

---

Let me know if you have any other tools that you think are worth mentioning, and I'll be happy to check them out!


In this blog post I will share some of the tools I use daily on my Mac to improve my productivity and efficiency as a developer.

My Mac Dev Toolkit For Better Efficiency


Managing a monorepo with multiple Python packages presents unique challenges, particularly in ensuring efficient CI/CD. In this article, I will share how I created CI/CD workflows for our monorepo of Python packages using GitHub Actions.

We will delve into the workflows, including matrix complexity, and scripts for checking changed paths, alongside other custom scripts and composite actions that facilitate this process.

Our primary CI workflow (`on-pr-build-push.yaml`) triggers on pull requests, ensuring that all changes are validated before merging. This workflow comprises several jobs: `ChangedPaths`, `Lint`, `Typecheck`, `Test`, and `Build-Push`.

**Job: ChangedPaths**

This job determines which packages have been modified and outputs a matrix that drives the subsequent jobs. So for example, if a PR modifies `package1` and `package2`, the matrix will contain two rows: `package1` and `package2`.

```yaml
name: Build & Push Python Package

on:
  pull_request:

jobs:
  ChangedPaths:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.get_changed_paths.outputs.matrix }}
    timeout-minutes: 10
    permissions:
      id-token: write
      contents: read

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          token: ${{ secrets.GLOBAL_GITHUB_TOKEN }}
          fetch-depth: 0

      - name: Run init
        uses: ./.github/actions/init # Custom composite action for initializing the environment

      - name: Get Changed Paths
        id: get_changed_paths
        shell: bash
        run: |
          python $GITHUB_WORKSPACE/.github/scripts/internal/get_changed_paths.py
```

The `get_changed_paths.py` python script identifies the changed paths and filters them to include only those within the packages directory.
This script is quite straightforward and uses the `git diff` command by using the built-in `subprocess` module in order to identify the changed paths.

```python
import subprocess
import os

output_file = os.getenv('GITHUB_OUTPUT')


def get_changed_paths():
    cmd = 'git diff --name-only $(git rev-parse HEAD) HEAD~1'
    result = subprocess.run(cmd, shell=True, capture_output=True)
    if result.returncode != 0:
        raise Exception("Error getting changed paths")

    changed_paths = result.stdout.decode("utf-8").strip().split("\n")
    print(f"Changed paths: {changed_paths}")
    return changed_paths


def get_filtered_paths(changed_paths):
    filtered_paths = set()
    for path in changed_paths:
        if path.startswith("packages/"):
            filtered_paths.add(path.split("/")[1])
    print(f"Filtered paths: {filtered_paths}")
    return list(filtered_paths)


if __name__ == "__main__":
    changed_paths = get_changed_paths()
    filtered_paths = get_filtered_paths(changed_paths)
    print(filtered_paths)
    with open(output_file, "a") as file:
        file.write(f"matrix={filtered_paths}")
```

Note the `capture_output=True` argument allows us to capture the command's output. We then decode and split the output to get a list of changed paths. The `get_filtered_paths` function filters these paths to include only those that start with `packages/` which is my path in my repos case.

Also note that GitHub Actions uses environment files to pass data between steps and jobs. By writing the output matrix to a file (GITHUB_OUTPUT), we can ensure that the list of changed paths is available as an output in the ChangedPaths job. This allows subsequent jobs to access and iterate over the paths efficiently.

**Job: Lint, Typecheck, and Test**

These jobs runs composite actions for linting (flake8), type checking (mypy), and tests (pytest) on the changed packages using a matrix strategy. They depend on the ChangedPaths job.

```yaml
Lint:
  needs: ChangedPaths
  if: ${{ needs.ChangedPaths.outputs.matrix != '[]' }}
  strategy:
    matrix:
      path: ${{ fromJson(needs.ChangedPaths.outputs.matrix) }}
  runs-on: ubuntu-latest
  timeout-minutes: 10
  permissions:
    id-token: write
    contents: read

  steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Get path
      run: echo ${{ matrix.path }}

    - name: Run init
      uses: ./.github/actions/init
      with:
        change_path: ${{ matrix.path }}

    - name: Run Lint action
      uses: ./.github/actions/lint
      env:
        paths: ${{ matrix.path }}
```

**Job: Build-Push**

The final job in the CI pipeline builds and pushes the packages if all previous jobs succeed.

```yaml
Build-Push:
  needs: [ChangedPaths, Lint, Typecheck, Test]
  if: ${{ needs.ChangedPaths.outputs.matrix != '[]' }}
  strategy:
    matrix:
      path: ${{ fromJson(needs.ChangedPaths.outputs.matrix) }}
  runs-on: ubuntu-latest
  timeout-minutes: 15
  permissions:
    id-token: write
    contents: read

  steps:
    - name: Checkout
      uses: actions/checkout@v3
      id: checkout

    - name: Run init
      id: run-init
      uses: ./.github/actions/init
      with:
        change_path: ${{ matrix.path }}

    - name: Build & Publish Python Packages
      uses: ./.github/actions/build-push-package
      env:
        paths: ${{ matrix.path }}
```

This job calls a custom composite action (`build-push-package`) that builds and upload the package to GitHub cache, so the user can install it from there.

```yaml
name: Build & Release Package
description: 'Build & Push Python Package'
author: 'Or Kazaz'

runs:
  using: composite

  steps:
    - name: Checkout
      uses: actions/checkout@v3
      with:
        fetch-depth: 0

    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'

    - name: Install Python Dependencies
      shell: bash
      run: |
        pip install --upgrade pip
        pip install poetry twine check-wheel-contents

    - name: Run Poetry Check for pyproject.toml
      working-directory: ./packages/${{ env.paths }}
      shell: bash
      run: poetry check

    - name: Build Package Distribution
      working-directory: ./packages/${{ env.paths }}
      shell: bash
      run: poetry build

    - name: Check Distribution Descriptions
      working-directory: ./packages/${{ env.paths }}
      shell: bash
      run: |
        twine check dist/*

    - name: Check Wheel Contents
      working-directory: ./packages/${{ env.paths }}
      shell: bash
      run: |
        check-wheel-contents dist/*.whl

    - name: Cache Package Distribution
      uses: actions/upload-artifact@v3
      with:
        name: ${{ env.paths }}
        path: ./packages/${{ env.paths }}/dist/
```

The secondary workflow (`tag-and-release.yaml`) is triggered on a manually on demand by using `workflow_dispatch`. It tags the package, updates its version is pyproject.toml.toml, and publish the packages into our internal PyPI repository (on Jfrog Artifactory). It consists of multiple jobs to ensure a smooth release process.

**Job: Tag and Release**

This job tags the repository and creates a release on GitHub by zipping it and using an action to create a release to be available for download.

```yaml
jobs:
  tag-and-release:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    outputs:
      package_version: ${{ steps.dynamic_version.outputs.package_version }}

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          ref: ${{ github.event.inputs.commit_hash }}

      - name: Run Dynamic-Versioning
        id: dynamic_version
        uses: ./.github/actions/dynamic-versioning
        with:
          package_name: ${{ github.event.inputs.package_name }}

      - name: Archive package directory
        working-directory: ./packages
        run: |
          zip -r ${{ github.event.inputs.package_name }}.zip ./${{ github.event.inputs.package_name }}
          echo "ARTIFACT_FILE_PATH=./packages/${{ github.event.inputs.package_name }}.zip" >> $GITHUB_ENV
        env:
          ARTIFACT_FILE_PATH: ${{ env.ARTIFACT_FILE_PATH }}

      - name: Create Release
        uses: ncipollo/release-action@v1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          artifacts: '${{ env.ARTIFACT_FILE_PATH }}'
          commit: ${{ github.event.inputs.commit_hash }}
          tag: ${{ github.event.inputs.package_name }}-v${{ steps.dynamic_version.outputs.package_version }}
          name: ${{ github.event.inputs.package_name }}-v${{ steps.dynamic_version.outputs.package_version }}
          token: ${{ secrets.GLOBAL_GITHUB_TOKEN }}
          skipIfReleaseExists: true
```

Note that the `dynamic-versioning` action is a custom action that updates the version in the `pyproject.toml` file.

```yaml
name: 'Dynamic Versioning'
description: 'Bump version in python package'
author: 'Or Kazaz'

inputs:
  package_name:
    description: 'Package name'
    required: true

outputs:
  package_version:
    description: 'bumped package version'
    value: ${{ steps.bump_version.outputs.package_version }}

runs:
  using: composite

  steps:
    - name: Checkout
      uses: actions/checkout@v3
      with:
        fetch-depth: 0

    - name: Bump version
      id: bump_version
      shell: bash
      run: |
        PACKAGE_VERSION=$($GITHUB_WORKSPACE/.github/scripts/internal/bump_version.sh ${{ inputs.package_name }})
        echo "${{ inputs.package_name }} package will be bumped to version: ${PACKAGE_VERSION}"
        echo "package_version=$PACKAGE_VERSION" >> $GITHUB_OUTPUT
```

Bump version is a very simple bash script the increments the package version in the `pyproject.toml` file.

```bash
#!/bin/bash

PACKAGE_NAME=$1

VERSION_LINE=$(grep version ./packages/$PACKAGE_NAME/pyproject.toml)
OLD_VERSION=$(echo "$VERSION_LINE" | awk -F'"' '{print $2}')
NEW_VERSION=$(echo "$OLD_VERSION" | awk -F. -v OFS=. '{$3=$3+1}1')

sed -i "s/$VERSION_LINE/version = \"$NEW_VERSION\"/g" ./packages/$PACKAGE_NAME/pyproject.toml

VERSION=$(grep version ./packages/$PACKAGE_NAME/pyproject.toml | awk -F'\"' '{print $2}')
echo $VERSION
```

**Job: Build**

This job builds the package distribution using Poetry.

```yaml
build:
  needs: tag-and-release
  name: Build Python Package
  runs-on: ubuntu-latest
  timeout-minutes: 30

  steps:
    - name: Checkout
      uses: actions/checkout@v3
      with:
        ref: ${{ github.event.inputs.commit_hash }}

    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'

    - name: Install Python Dependencies
      working-directory: ./packages/${{ github.event.inputs.package_name }}
      run: |
        pip install --upgrade pip
        pip install --upgrade poetry twine check-wheel-contents

    - name: Run Dynamic-Versioning
      id: dynamic_version
      uses: ./.github/actions/dynamic-versioning
      with:
        package_name: ${{ github.event.inputs.package_name }}

    - name: Build Package Distribution
      working-directory: ./packages/${{ github.event.inputs.package_name }}
      run: |
        poetry build
```

**Job: Publish to JFrog**

This job publishes the built package to JFrog by retrieving the cached package distribution and using the official `gh-action-pypi-publish` action.

```yaml
publish-jfrog:
  needs: build
  name: Publish Package to JFrog
  runs-on: ubuntu-latest
  timeout-minutes: 15

  steps:
    - name: Retrieve Cached Package Distribution
      uses: actions/download-artifact@v3
      with:
        name: ${{ github.event.inputs.package_name }}
        path: ./packages/${{ github.event.inputs.package_name }}/dist

    - name: Publish to JFrog
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        packages-dir: ./packages/${{ github.event.inputs.package_name }}/dist
        user: ${{ secret.secret_user }}
        password: ${{ secret.secret_password }}
        repository-url: https://the.internal.repo.url/artifactory/api/pypi/pypi
```

**Job: Commit Updated Version**

Finally, this job commits the updated version to the repository while skipping CI.

```yaml
commit-version:
  needs: [tag-and-release, build, publish-jfrog]
  name: Commit updated version
  runs-on: ubuntu-latest
  timeout-minutes: 15

  steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Commit updated version
      shell: bash
      run: |
        git config --global user.name $GITHUB_USER
        git config --global user.email $GITHUB_EMAIL
        git pull
        bash $GITHUB_WORKSPACE/.github/scripts/internal/bump_version.sh ${{ github.event.inputs.package_name }}
        git add ./packages/${{ github.event.inputs.package_name }}/pyproject.toml
        git commit -m "Bump ${{ github.event.inputs.package_name }} to version ${{ needs.tag-and-release.outputs.package_version }} [skip ci]"
        git push origin ${GITHUB_REF#refs/heads/}
```

![CI-ON-PR](https://res.cloudinary.com/dtt2usukl/image/upload/v1718703406/Image_18-06-2024_at_12.36_ge6oku.jpg)_On Pull request CI of 3 packages modified_

This setup ensures that only the affected packages are processed, reducing build times and resource usage. The use of matrix strategies, custom scripts, and composite actions provides a flexible, generic and scalable approach to managing CI/CD workflows of such use case.


In this article i'll share how I handheld the CI/CD in a mono repo with multiple python packages using GitHub Actions.

Handling CI/CD in a Mono Repo With Multiple Python Packages


We've implemented extensive improvements, including expanding device capacity, enhancing monitoring systems, introducing features like automated recovery mechanisms, chaging regions using internal network VPN and more.
These advancements will streamline testing processes, optimize resource utilization, and provide a seamless experience for developers and testers across global locations.
With ~250 users, supporting 25 different platforms and 150 testing devices, we are well-positioned for continued success.

For those of you who are not familiar with DAZN, here's a brief overview:

> _DAZN is the world’s leading live sports BTC streaming service. With DAZN, fans in any country can watch sport in any language, on any device, live and on-demand. Purpose built to stream live sports globally, DAZN is the market leader in Italy, Spain, Germany, and Japan, where it holds the top-tier domestic football rights — Serie A, La Liga, Bundesliga and J League, respectively — as well as the richest catalogue of live and on-demand content. DAZN is also available in over 200 other territories, where it boasts the most extensive library of elite and emerging sports content._

DAZN develops both HTML-based and native applications designed to run on a variety of platforms, including TVs, set-top boxes, gaming consoles, and native devices. Ensuring high quality across these diverse platforms necessitates thorough testing at every stage of development, ranging from unit tests and component tests to emulator tests and real-device testing.

The global pandemic in 2020 prompted a paradigm shift in work practices, underscoring the importance of enabling engineers to access diverse resources and conduct testing remotely. Quality remains paramount, necessitating comprehensive testing methodologies despite remote work environments.

To address these challenges, we adopted a proactive “Lab as a Service” approach, culminating in the establishment of an in-house lab facility in Katowice, Poland. This strategic decision aimed to centralize resources and provide engineers with the necessary tools and infrastructure to conduct testing efficiently.

However, establishing and operating an in-house lab posed unique challenges that required careful consideration and resolution. These challenges encompassed logistical, technical, and organizational aspects, underscoring the need for adaptive solutions and collaborative efforts.

In response to these challenges, ongoing initiatives are focused on optimizing the efficiency and effectiveness of the in-house lab, streamlining workflows, enhancing collaboration among teams, and leveraging technology to overcome geographical barriers. By embracing innovation and agility, DAZN aims to continuously improve its testing capabilities and deliver exceptional quality across its diverse range of applications and platforms.

## Challenges

DAZN employs a diverse team of developers and test engineers spread across different global locations such as Poland, India, Israel, and the UK. Each team comes with its unique set of needs and requirements, necessitating a robust, highly available, and cost-effective platform to accommodate them effectively.

Key points include:

- Diverse Platform Support:
  The platform must ensure comprehensive testing across all platforms supported by DAZN, emphasizing thoroughness and quality assurance in our testing procedures.
- Tailored Solutions:
  Teams may require different tools and resources for their testing processes; for instance, some teams may rely on manual UI/UX testing devices, while others prioritize automation solutions.
- Integration Challenges:
  Incorporating in-house tools and services into our platform is essential to ensure seamless continuity for engineers, requiring careful integration efforts.
- Scalability and Maintenance:
  The platform needs to support a large number of engineers on a daily basis across multiple global sites, balancing scalability with efficient maintenance to avoid operational bottlenecks.

- Cost Considerations:
  Maintaining devices, infrastructure, and manpower across various locations can incur significant expenses, necessitating a cost-effective approach to resource allocation and management.

- Ownership and Accountability:
  Clear ownership and accountability structures are crucial to prevent issues such as maintenance gaps or device mismanagement.

To address these challenges, ongoing efforts are focused on streamlining processes, enhancing collaboration between teams, and leveraging automation wherever possible to optimize resource utilization and improve efficiency in our testing and development workflows.

## Solution — Lab as a Service

Our team, composed of seasoned experts in DevOps, Automation Infrastructure, and IT, recognized the need for a novel solution to address the challenges faced in managing Device Under Test (DUT) usage at DAZN. In the first quarter of 2023, we conducted a comprehensive assessment to identify and map out the major pain points associated with DUT utilization within the organization. This assessment culminated in the development of a pioneering approach coined “Lab as a Service,” aimed at alleviating the burden on developers and testers by handling all DUT-related tasks and physical infrastructure management.

During the second quarter of 2023, we introduced the Minimum Viable Product (MVP) setup, leveraging the Suitest framework — a vendor offering remote control and automation capabilities tailored for both HTML-based and native platforms. Our team meticulously orchestrated the deployment of all requisite physical infrastructure, assembling a dedicated lab housing 50 test devices within a spacious room at the DAZN Katowice office.

![DUT lab v1.0 (QA room — May 2023)](https://miro.medium.com/v2/resize:fit:640/format:webp/1*MDbJo7nq77qwWrRCpNvsrA.jpeg)_DUT lab v1.0 (QA room — May 2023)_

To bolster reliability and resilience, we implemented a comprehensive monitoring and alerting system, with Grafana serving as the frontend interface. Additionally, we integrated DUT recovery mechanisms to enhance the robustness of our testing environment.

![Grafana monitoring dashboard](https://miro.medium.com/v2/resize:fit:640/format:webp/1*8a9dpHjMLH1FZCQptCNplw.png)_Grafana monitoring dashboard_

The efficacy of these enhancements became apparent swiftly, as an increasing number of engineers gravitated towards utilizing the centralized lab infrastructure instead of configuring local devices. Leveraging the valuable feedback provided by users, we iteratively refined and enhanced the lab environment, ensuring that it continues to meet the evolving needs of our development and testing teams.

## Lab as a service 2.0

As the user base and the number of devices in the DAZN lab continued to surge, spurred by the establishment of a new DAZN site in India and the expansion of supported platforms throughout the latter half of 2023, we undertook further enhancements to the Katowice lab. These adjustments were aimed at accommodating the organization’s growth and fortifying its capacity to scale even further. Throughout the first quarter, we initiated a series of improvements targeting both the physical infrastructure and supporting features:

### Infrastructure enhancements

- **Relocation and Expansion:**

The DUT lab underwent relocation to a larger space, facilitating increased scalability and the seamless onboarding of additional platforms and devices. Leveraging existing racks more efficiently, we enabled the mounting of devices on both sides of each rack, resulting in a 50% boost in the number of TVs and a 40% increase in the number of set-top boxes (STBs), totaling 120 DUTs overall, with ample room for further expansion.

![DUT lab v2.0 (open space — January 2024)](https://miro.medium.com/v2/resize:fit:640/format:webp/1*IKUuw6aU2wQMzTBKCKS_bA.jpeg)_DUT lab v2.0 (open space — January 2024)_

- **Streaming Solutions:**

Collaborating with Suitest, we identified a cost-effective solution enabling the streaming of device HDMI output for STBs and gaming consoles. Additionally, we enhanced the TV streaming setup by refining the phone streaming app.

![Live DUT Streaming Phone camera setup](https://miro.medium.com/v2/resize:fit:720/format:webp/1*K460jjXRPqXUVQf8PVg0ng.jpeg)_Live DUT streaming mobile phone camera setup_

![HDMI grabber/splitter combo](https://miro.medium.com/v2/resize:fit:640/format:webp/1*s-cKLE8Oh1h5mliovKc91Q.jpeg)_HDMI grabber/splitter combo_

### Features enhancements

- **Collaboration with Suitest:**

Over the course of the year, we actively collaborated with Suitest to influence their roadmap, contributing to improvements in user experience and interface. A significant milestone was the introduction of a Unified UI integrating video streaming, remote control, and logs within the same browser window, enhancing productivity for lab users.

![Suitest Unified UI](https://miro.medium.com/v2/resize:fit:640/format:webp/1*Wg96k859tBYZwz13vVXG3Q.png)_Suitest Unified UI_

- **LR (Living Room) Launcher**:

In collaboration with the DAZN LivingRoom development team, we introduced LR Launcher, simplifying the workflow for manual testers and streamlining the onboarding process for partner platforms such as Sky and Movistar for the sake of ease of use when testing different development environments.

![LR Launcher](https://miro.medium.com/v2/resize:fit:640/format:webp/1*5AObQs1_Vdyd9cXhf5qF3w.png)_LR Launcher_

- **Enhanced Monitoring**:

With the growing user base and device count, robust monitoring became paramount. We revamped our monitoring service, streamlining processes and transitioning from an API-based solution to WSS notifications. Furthermore, we expanded monitoring coverage and implemented alerting for control units and streaming devices.

- **Automated Recovery**:

Leveraging smart plugs across all TV platforms, we implemented remote manual and automated recovery mechanisms for malfunctioning DUTs. Efforts are underway to develop similar mechanisms for cameras and control units, bolstering the resilience and efficiency of our lab environment.

- **Local Tunnel**:

A local tunnel solution providing exposure of a local development server allows developers and testers to seamlessly execute their code on a real physical lab device. This capability not only streamlines the testing process but also provides a practical and authentic environment for validating local code during development.

- **Lab Insights Report (BI)**:

An automated report detailing lab usage and testing trends. This report will provide valuable insights into usage patterns, enabling us to further optimize our services based on real data.

![BI Report](https://res.cloudinary.com/dtt2usukl/image/upload/v1716458627/Image_27-08-2023_at_11.58_sas9dh.jpg)_BI Report_

**VPN Change Region & VPN Console**:

A VPN solution that will empower lab users to seamlessly transfer DUTs between regions remotely. Using firewalls controlled by OPNSense, a lambda backend and a UI portal written in React (NextJS) users are able to testing of region-dependent content across all lab devices by switching regions as needed.

![VPN Console](https://res.cloudinary.com/dtt2usukl/image/upload/v1716458833/Image_23-05-2024_at_13.06_oxbech.jpg)_VPN Console_

## Service & Support

- **Continuous Improvement:**

Since the launch of the DUT lab in May 2023, we have continually bolstered our service and support infrastructure. This includes expanding our support engineering team, enhancing documentation and guides, and maintaining accessible support channels during working hours. Our commitment to swift issue resolution is underscored by our expanded remote resolution capabilities, facilitated by proactive monitoring and device error-based recovery processes.

- **Onboarding Optimization:**

Through the introduction of “Lab as a Service 2.0” and tailored training sessions, we have streamlined the onboarding process for new users, ensuring a seamless transition and maximizing productivity from the outset using official docs portal (using github pages and backstage).

These enhancements collectively represent our ongoing commitment to optimizing the DAZN lab environment, fostering innovation, and empowering our engineering teams to deliver exceptional quality across all platforms and devices.

## Acknowledgements

Special thanks to the amazing AP Nexus team for their dedication and hard work, it was a pleasure working with each one of you!

[Dennis Groisman](https://www.linkedin.com/in/dennis-groisman-9811b335/), [Daniel-Shahar Cohen](https://www.linkedin.com/in/danielshaharcohen/), [Rafal Muszalik](https://www.linkedin.com/in/rafa%C5%82muszalik/), [Blazej Milewski](https://www.linkedin.com/in/blazej-milewski/), [Alex Rivkin](https://www.linkedin.com/in/alex-rivkin-b037a6a1/), [Yoav Brown](https://www.linkedin.com/in/yoav-brown-970b55265/) and the one and only [Gal Moran](https://www.linkedin.com/in/gal-moran-a3454095/).

[![DAZN Lab as a Service](https://res.cloudinary.com/dtt2usukl/image/upload/v1716460903/play_video_lab_fnkajn.jpg)](https://res.cloudinary.com/dtt2usukl/video/upload/v1716460249/tvlab_v2_withmusic_twifar.mp4 'DAZN Lab as a Service - Nexus Team')_My team - Lab as a Service 2.0_

Article on [Medium (DAZN Engineering Blog)](https://medium.com/dazn-tech/lab-as-a-service-in-dazn-9bcc9d549970)


I'm incredibly proud to share the outstanding work our team has accomplished recently in upgrading our testing lab infrastructure.

Welcome toKazi's Dev Blog

Latest Articles