What a Good Test Report Shows

Who Reads the Report

The surefire console output is fine for you. Your manager, the developers and the client want a web page: how many tests passed, which failed, why, with a screenshot and the steps. A good report answers "can we release?" in ten seconds.

Summary: total, passed, failed, skipped, pass percentage, duration.
Environment: app URL, build number, browser and version, OS.
Per test: name, group/feature, steps logged, error message, screenshot.
History (Allure): is this test newly failing or failing every night?

Building a Summary in Plain Java

Example

Report tools calculate this summary from the test results. Here the results are a fixed list:

import java.util.*;
import java.util.stream.*;

public class Main {
    record Result(String test, String status, long millis) {}

    public static void main(String[] args) {
        List<Result> results = List.of(
                new Result("validLogin", "PASS", 4210),
                new Result("wrongPassword", "PASS", 3120),
                new Result("searchHeadphones", "PASS", 5530),
                new Result("addToCart", "FAIL", 8040),
                new Result("applyCoupon", "SKIP", 0),
                new Result("checkoutUpi", "PASS", 11250));

        Map<String, Long> counts = results.stream()
                .collect(Collectors.groupingBy(Result::status, TreeMap::new, Collectors.counting()));
        long total = results.size();
        long passed = counts.getOrDefault("PASS", 0L);
        long totalMs = results.stream().mapToLong(Result::millis).sum();

        System.out.println("Counts: " + counts);
        System.out.printf("Pass rate: %.1f%%%n", passed * 100.0 / total);
        System.out.printf("Duration: %dm %02ds%n", totalMs / 60000, (totalMs / 1000) % 60);
        results.stream().filter(r -> r.status().equals("FAIL"))
                .forEach(r -> System.out.println("Investigate: " + r.test()));
    }
}

Output

Counts: {FAIL=1, PASS=4, SKIP=1}
Pass rate: 66.7%
Duration: 0m 32s
Investigate: addToCart