Real World Example

Product companies such as LinkedIn and Netflix track flakiness as a metric. A test that fails and then passes on retry in the same build is flagged as flaky. The CI job writes a line per test; a small job calculates the flaky rate and quarantines tests above a threshold.

Example

The same calculation on a week of fixed results (P = pass, F = fail, R = failed then passed on retry):

import java.util.*;

public class Main {
    public static void main(String[] args) {
        Map<String, String> lastSevenRuns = new LinkedHashMap<>();
        lastSevenRuns.put("validLogin",       "PPPPPPP");
        lastSevenRuns.put("searchSortByPrice", "PRPRRPP");
        lastSevenRuns.put("applyCoupon",       "PPPPFFF");
        lastSevenRuns.put("checkoutUpi",       "PPRPPPP");

        System.out.printf("%-18s %-8s %-6s %s%n", "test", "runs", "flaky", "action");
        lastSevenRuns.forEach((test, runs) -> {
            long retried = runs.chars().filter(c -> c == 'R').count();
            double flakyRate = retried * 100.0 / runs.length();
            String action;
            if (runs.endsWith("FFF")) action = "real failure - raise a bug";
            else if (flakyRate > 20) action = "QUARANTINE + ticket";
            else if (retried > 0) action = "watch";
            else action = "ok";
            System.out.printf("%-18s %-8s %5.0f%% %s%n", test, runs, flakyRate, action);
        });
    }
}

Output

test               runs     flaky  action
validLogin         PPPPPPP      0% ok
searchSortByPrice  PRPRRPP     43% QUARANTINE + ticket
applyCoupon        PPPPFFF      0% real failure - raise a bug
checkoutUpi        PPRPPPP     14% watch

Flaky Test Dashboard at a Product Company

Example

Why Separate Flaky From Failing