Parallel Execution With TestNG

Why Parallel

A ShopEase regression suite of 300 tests at 20 seconds each takes 100 minutes in one browser. With 8 browsers in parallel it takes about 13 minutes. TestNG runs tests in parallel threads with two attributes in testng.xml.

Example

testng.xml

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="ShopEase Regression" parallel="methods" thread-count="4" data-provider-thread-count="2">
  <listeners>
    <listener class-name="com.shopease.listeners.TestListener"/>
  </listeners>
  <test name="UI">
    <classes>
      <class name="com.shopease.tests.LoginTest"/>
      <class name="com.shopease.tests.SearchTest"/>
      <class name="com.shopease.tests.CartTest"/>
    </classes>
  </test>
</suite>

Parallel Modes

`parallel=`	What runs at the same time	Note
`methods`	individual @Test methods	fastest; every test must be independent
`classes`	test classes (methods of a class run in one thread)	safe when a class shares state
`tests`	`<test>` blocks	e.g. one <test> per browser
`instances`	instances created by a @Factory	rare

Note: Parallel runs only work if the driver is per thread (the ThreadLocal DriverFactory from the framework module) and tests do not share data such as the same user's cart.

How Much Faster

Example

A simple scheduler: each free thread takes the next test from the queue. The test durations are fixed so the result is repeatable:

import java.util.*;

public class Main {
    static int runTime(int[] durations, int threads) {
        PriorityQueue<Integer> busyUntil = new PriorityQueue<>();
        for (int i = 0; i < threads; i++) busyUntil.add(0);
        for (int d : durations) {
            int free = busyUntil.poll();       // the thread that finishes first takes the next test
            busyUntil.add(free + d);
        }
        int end = 0;
        for (int t : busyUntil) end = Math.max(end, t);
        return end;
    }

    public static void main(String[] args) {
        int[] seconds = {20, 35, 18, 42, 25, 30, 15, 22, 40, 28, 19, 33};
        for (int threads : new int[]{1, 2, 4, 8}) {
            System.out.printf("%d thread(s): %3d s%n", threads, runTime(seconds, threads));
        }
    }
}

Output

1 thread(s): 327 s
2 thread(s): 175 s
4 thread(s):  97 s
8 thread(s):  55 s