Running tests in parallel with ParallelTestRunner

Mosè Giordano

UCL

2026-08-14

The problem

Using the Test standard library

using MyPackage, Test

@testset "MyPackage" begin
    include("core.jl")
    include("linalg.jl")
    include("io.jl")
    include("serialization.jl")
    include("integration.jl")
    # ...
end

Limitations

  • One process
  • Wall-clock time grows linearly with the size of the test suite
  • CI machines have many cores… mostly idle
  • State leaks between included files (using, globals, method redefinitions)
  • Running only linalg.jl means editing runtests.jl

Various packages in the ecosystem already tried to address some of these shortcomings (more on this later)

ParallelTestRunner

History

  • Julia’s Base.runtests parallel harness
  • adapted by Tim Besard (@maleadt) for CUDA.jl’s test infrastructure…
  • …and copied in all other packages of the GPU ecosystem
  • extracted and generalised as JuliaTesting/ParallelTestRunner.jl by Valentin Churavy (@vchuravy) + many other contributors
  • Purpose-built for large, resource-hungry suites
  • Used by CUDA.jl, AMDGPU.jl, Enzyme.jl, and more

Migration is a deletion

Before:

using MyPackage, Test

@testset "MyPackage" begin
    include("core.jl")
    include("linalg.jl")
    include("io.jl")
    # ...
end

After:

using MyPackage
using ParallelTestRunner

runtests(MyPackage, ARGS)
  • Every .jl file in test/ (except runtests.jl) is autodiscovered as an independent test
  • No macros, no annotations, no rewrite of the test files themselves

Execution model

  • Each test file runs in a separate worker process (JuliaPluto/Malt.jl)
  • …inside a freshly generated sandbox module: isolation for free
  • Results are streamed back to the main process as they complete
  • --jobs=N controls the number of parallel workers

File-level parallelism: coarser than ReTestItems.jl’s per-@testitem scheduling, but requires no code changes

A command-line interface

Usage: runtests.jl [--help] [--list] [--jobs=N] [TESTS...]

   --help             Show this text.
   --list             List all available tests.
   --verbose          Print more information during testing.
   --quickfail        Fail the entire run as soon as a single test errored.
   --jobs=N           Launch `N` processes to perform tests.

Remaining arguments filter the tests that will be executed.
Prefix your argument with '!' to instead exclude those tests
  • julia --project test/runtests.jl linalg runs only tests whose name starts with linalg
  • Also works through Pkg.test(test_args=["linalg"])

What it looks like

Running 3 tests using 2 parallel jobs.
                 │   Test   │ ──────────────── CPU ──────────────── │
Test    (Worker) │ time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
strings      (1) │     0.14 │   0.00 │  0.0 │       6.90 │   454.56 │
arithmetic   (2) │     0.15 │   0.00 │  0.0 │       7.64 │   454.56 │

Running:  slow
Progress: 2/3 tests completed │ ETA: ~0 min
  • Live status bar: running tests, per-worker stats, ETA
  • Worker output is captured and printed without interleaving

Scheduling strategy

  • Duration and pass/fail history of each test is persisted in a scratch directory
  • Longest-running tests are scheduled first for better load balancing
  • Previously-failing tests are scheduled first
  • Workers exceeding a memory threshold (max_worker_rss) are recycled
  • ~Zero configuration required

Escape hatches

  • testsuite: provide your own name => code dictionary instead of autodiscovery (for example if some test files should be always excluded, or be run conditionally)
  • init_code / init_worker_code: set up each sandbox module / each worker
  • exename / exeflags / env: control how workers are launched
  • test_worker: create a custom worker based on test name (for example with specific environment variables, or different Julia flags)

Demo

A toy package

demo
├── Project.toml
├── src
│   └── Demo.jl
└── test
    ├── arithmetic.jl
    ├── broken.jl
    ├── Project.toml
    ├── printing.jl
    ├── runtests.jl
    ├── slow.jl
    └── strings.jl

test/runtests.jl:

using Demo, ParallelTestRunner

runtests(Demo, ARGS)

test/arithmetic.jl, test/strings.jl, test/slow.jl are plain Test files, no include anywhere

Parallel run

julia --project test/runtests.jl --jobs=2 --verbose

Running 5 tests using 2 parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
Available memory: 12.906 GiB; Max worker RSS: 3.711 GiB
                 │   Test   │   Init   │ Compile │ ──────────────── CPU ──────────────── │
Test    (Worker) │ time (s) │ time (s) │   (%)   │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
printing     (1) │        started at 2026-08-14T13:44:45.770
broken       (2) │        started at 2026-08-14T13:44:46.479
printing     (1) │     0.23 │     3.42 │   97.82 │   0.00 │  0.0 │       7.59 │   495.03 │
slow         (1) │        started at 2026-08-14T13:44:47.128
broken       (2) │     0.31 │     4.09 │   99.20 │   0.00 │  0.0 │      23.75 │   495.03 │
arithmetic   (2) │        started at 2026-08-14T13:44:47.654
arithmetic   (2) │     0.11 │     0.29 │   88.60 │   0.00 │  0.0 │       2.83 │   495.03 │
strings      (2) │        started at 2026-08-14T13:44:48.112
strings      (2) │     0.01 │     0.29 │   42.23 │   0.00 │  0.0 │       0.17 │   495.03 │
slow         (1) │     2.07 │     0.44 │    3.09 │   0.00 │  0.0 │       1.46 │   495.03 │

Output generated during execution of 'printing':
[ [ Info: Hello, world

Test Summary:  | Pass  Broken  Total  Time
  Overall      |    9       1     10  8.5s
    printing   |    1              1  0.2s
    broken     |            1      1  0.3s
    arithmetic |    5              5  0.1s
    strings    |    2              2  0.0s
    slow       |    1              1  2.1s
    SUCCESS

Choosing testing package

Comparison

Parallelism Rewrite Isolation Filtering
Test ❌ — ❌ ❌
SafeTestsets.jl ❌ @safetestset ✅ ❌
TestItemRunner.jl ❌ @testitem ✅ tags
ReTestItems.jl ✅ procs, per item @testitem ✅ name, tags
XUnit.jl ✅ threads @testcase ❌ ✅
ParallelTestRunner.jl ✅ procs, per file ~none ✅ files prefix

When to choose what

  • Existing include-style suite, want parallelism with zero rewrite → ParallelTestRunner.jl
  • Sub-file granularity, retries for flaky tests, tags → ReTestItems.jl
  • Heavy VS Code test-explorer user → TestItems.jl + TestItemRunner.jl
  • Need JUnit XML reports, or thread-based parallelism → XUnit.jl
  • Small, fast suite → plain Test is fine!

Conclusions