Retry Analyzer for Flaky Tests

IRetryAnalyzer

Sometimes a test fails because of the environment (a slow QA server, a network blip), not a bug. A retry analyzer re-runs a failed test a limited number of times. Use it as a safety net, not as a fix: a test that often needs retries must be investigated.

public class RetryAnalyzer implements IRetryAnalyzer {
    private static final int MAX_RETRIES = 2;
    private int attempt = 0;

    @Override
    public boolean retry(ITestResult result) {
        if (attempt < MAX_RETRIES) {
            attempt++;
            System.out.println("Retrying " + result.getName() + " (" + attempt + "/" + MAX_RETRIES + ")");
            return true;      // run the test again
        }
        return false;
    }
}

// apply to one test
@Test(retryAnalyzer = RetryAnalyzer.class)
public void placeOrderWithUpi() { ... }

Applying It to Every Test

public class RetryTransformer implements IAnnotationTransformer {
    @Override
    public void transform(ITestAnnotation annotation, Class testClass,
                          Constructor testConstructor, Method testMethod) {
        annotation.setRetryAnalyzer(RetryAnalyzer.class);
    }
}

Example

mvn test

Example output

Retrying placeOrderWithUpi (1/2)
[WARNING] Tests run: 13, Failures: 0, Errors: 0, Skipped: 1
[INFO] BUILD SUCCESS

Simulating Retries

Example

A flaky step fails on the first two attempts and passes on the third. The retry loop works the same way as TestNG's:

import java.util.function.*;

public class Main {
    static int calls = 0;

    static boolean flakyPaymentStep() {
        calls++;
        return calls >= 3;                 // QA payment sandbox is slow at first
    }

    static boolean runWithRetry(String name, BooleanSupplier test, int maxRetries) {
        for (int attempt = 0; attempt <= maxRetries; attempt++) {
            boolean passed = test.getAsBoolean();
            System.out.println(name + " attempt " + (attempt + 1) + ": " + (passed ? "PASS" : "FAIL"));
            if (passed) return true;
        }
        return false;
    }

    public static void main(String[] args) {
        boolean ok = runWithRetry("placeOrderWithUpi", Main::flakyPaymentStep, 2);
        System.out.println("Final result: " + (ok ? "PASSED (flaky, needed retries)" : "FAILED"));
        calls = 0;
        ok = runWithRetry("placeOrderWithCard", Main::flakyPaymentStep, 1);
        System.out.println("Final result: " + (ok ? "PASSED" : "FAILED after retries"));
    }
}

Output

placeOrderWithUpi attempt 1: FAIL
placeOrderWithUpi attempt 2: FAIL
placeOrderWithUpi attempt 3: PASS
Final result: PASSED (flaky, needed retries)
placeOrderWithCard attempt 1: FAIL
placeOrderWithCard attempt 2: FAIL
Final result: FAILED after retries