Playwright Tips: Leverage Playwright's Retry Mechanism as a Custom Fixture

The Problem with Flaky Tests

We've all been there - you have a test suite that works perfectly in isolation, but occasionally fails due to external factors beyond your control (like a third-party service being down). Maybe it's a network hiccup, a temporary service unavailability, or leftover test data from previous runs. While Playwright offers built-in retry mechanisms, sometimes you need more control over what happens between retry attempts.

In this post, I'll show you how to leverage Playwright's retry system with custom fixtures to perform cleanup operations between retries, ensuring each retry attempt starts with a clean slate.

Understanding Playwright's Built-in Retry Mechanism

Playwright provides a simple retry configuration that you can set in your playwright.config.ts:

// playwright.config.ts
export default defineConfig({
  // Retry failed tests up to 2 times
  retries: process.env.CI ? 2 : 0,

  // Or configure per project
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
      retries: 2,
    },
  ],
})

While this works great for simple flaky tests, it doesn't help when your test failures are caused by leftover data or state from previous attempts. This is where custom retry handling becomes invaluable.

The Solution: Custom Retry Fixture

The key insight is to create a custom fixture that can detect when a test is being retried and perform necessary cleanup operations. Here's how I implemented this pattern in one of my projects:

Step 1: Create a RetryHandler Class

First, let's create a RetryHandler class that manages the cleanup logic. In this example, we are cleaning up test data that could be stored in various systems like MongoDB, Redis, files, or other external services.

// utils/retry-handler.ts
import { logger } from '@moonactive/admin-common-logger'
import { Fixtures } from 'fixture/base.fixture'
import { TestInfo } from 'playwright-decorators'

export class RetryHandler {
  async retryHandler(
    testInfo: TestInfo,
    context: Fixtures, // Where you pass other fixtures you want to clean up
    testData: any[] // Where you pass the test data you want to clean up
  ): Promise<void> {
    // Only perform cleanup if this is a retry attempt (retry > 0)
    if (testInfo.retry > 0) {
      await RetryHandler.retryCleaner(context, testData)
    }
  }

  static async retryCleaner(
    context: Partial<Fixtures>,
    testData: any[]
  ): Promise<void> {
    try {
      const { mongo, redis } = context
      const dataIds = testData.map(
        item => item.id || item._id || item.identifier
      )

      const contextDetails = {
        mongo: !!mongo,
        redis: !!redis,
      }

      logger.info(
        `Triggering retry cleanup for data: ${dataIds}, context: ${JSON.stringify(
          contextDetails
        )}`
      )

      // Clean up database records if MongoDB is used
      if (mongo) {
        await mongo.deleteTestDataFromDB(testData)
      }

      // Clean up Redis records if Redis is used
      if (redis) {
        await redis.deleteTestDataFromRedis(testData)
      }

      if (!mongo && !redis) {
        logger.error('Missing context for retryCleaner')
        throw new Error('Missing context for retryCleaner')
      }
    } catch (err) {
      logger.error(`Failed to clean up after retry: ${err}`)
      throw err
    }
  }
}

Step 2: Integrate with Playwright Fixtures

Next, integrate the RetryHandler into your Playwright fixture system:

// fixture/base.fixture.ts
import { RetryHandler } from '../utils/retry-handler'
// ... other imports

export type Fixtures = {
  logger: Types.SomeLogger
  app: App
  expect: typeof expect
  mongo: DatabaseHelper
  redis: RedisHelper
  bq: BQHelper
  api: APIHelper
  sqs: SqsHelper
  retry: RetryHandler // Our custom retry handler
}

export const test = base.extend<Fixtures>({
  // ... other fixtures

  retry: async ({}, use) => {
    const retryHandler = new RetryHandler()
    await use(retryHandler)
  },

  // ... other fixture implementations
})

Step 3: Use the Retry Handler in Tests

Now you can use the retry handler in your tests:

// tests/example.spec.ts

test('Create and validate test data', async ({
  app,
  api,
  mongo,
  redis,
  retry,
}, testInfo) => {
  // Define test data that might need cleanup
  const testData = [
    { id: 'test-item-1', name: 'Test Item 1', type: 'example' },
    { id: 'test-item-2', name: 'Test Item 2', type: 'example' },
  ]

  // Call retry handler at the beginning of the test
  // This will clean up any leftover data if this is a retry attempt
  await retry.retryHandler(testInfo, { mongo, redis }, testData)

  // Your actual test logic
  await app.createTestItems(testData)

  // Validate the created items
  await api.validateTestItems(testData)

  // Perform additional test actions
  await app.performActionsOnItems(testData)
})

How Does It Work?

The magic happens in the testInfo.retry property. Playwright automatically increments this value for each retry attempt:

• First attempt: testInfo.retry = 0 (no cleanup performed)
• First retry: testInfo.retry = 1 (cleanup is performed)
• Second retry: testInfo.retry = 2 (cleanup is performed again)

This ensures that cleanup only happens when needed, avoiding unnecessary operations on the first test run.

Key Benefits

1. Intelligent Cleanup

Only performs cleanup operations when actually retrying, not on the initial test run.

2. Flexible Context

You can pass different fixtures and resources based on what your specific test needs to clean up.

3. Comprehensive Coverage

The example shows cleanup across multiple systems (database, GitHub, configuration), but you can adapt it to your needs.

4. Proper Error Handling

Includes logging and error handling to help debug cleanup issues.

5. Type Safety

Leverages TypeScript for better IDE support and catch errors at compile time.

Advanced Usage Patterns

Multi-Service Cleanup

You can extend this pattern to handle different types of cleanup based on the test context:

await retry.retryHandler(
  testInfo,
  {
    mongo,
    redis,
    elasticsearch,
    fileSystem,
    api,
  },
  testData
)

Conditional Cleanup

Add logic to perform different cleanup based on test metadata:

async retryHandler(
  testInfo: TestInfo,
  context: Fixtures,
  data: any[],
  options?: { skipRedis?: boolean; skipDB?: boolean; skipFiles?: boolean }
): Promise<void> {
  if (testInfo.retry > 0) {
    await RetryHandler.retryCleaner(context, data, options)
  }
}

Custom Retry Limits

You can even implement custom retry logic based on specific conditions:

async retryHandler(
  testInfo: TestInfo,
  context: Fixtures,
  data: any[],
  maxRetries: number = 2
): Promise<void> {
  if (testInfo.retry > 0 && testInfo.retry <= maxRetries) {
    await RetryHandler.retryCleaner(context, data)
  } else if (testInfo.retry > maxRetries) {
    logger.warn(`Exceeded max retries (${maxRetries}), skipping cleanup`)
  }
}

Best Practices

1. Call Early: Always call the retry handler at the very beginning of your test, before any test actions.

2. Be Specific: Only clean up the specific data your test creates to avoid affecting other tests.

3. Log Everything: Include comprehensive logging to help debug retry scenarios.

4. Handle Failures: Make sure cleanup failures don't prevent the retry from proceeding.

5. Test Your Cleanup: Consider writing separate tests to verify your cleanup logic works correctly.

Conclusion

This pattern transforms Playwright's basic retry mechanism into a powerful tool for handling complex test scenarios. By combining custom fixtures with intelligent cleanup logic, you can ensure that flaky tests get a truly fresh start on each retry attempt.

The approach is particularly valuable for integration tests that interact with external services, databases, or file systems where leftover state can cause cascading failures.

Have you implemented similar retry patterns in your test suites? I'd love to hear about your experiences and any creative solutions you've developed!