Skip to content

Runtime forensics ​

Static flakiness detection is guessing. A rule can tell you a test looks flaky — a hard sleep, a networkidle wait — but it cannot tell you whether that test actually failed last Tuesday.

The evidence path reads real execution data: Playwright JSON reports, Jest JSON (--json --outputFile), Vitest JSON (--reporter=json) and JUnit XML from any runner.

The command moved. The forensics and triage verbs were retired in the v6 carve; the capability is mjolnir explain --evidence <dir>. Only the way you reach it changed. See the CLI contract for the disposition of every verb.

  ▍ FLAKINESS LEADERBOARD

3 tests · 1 failed · 1 flaky · 1 retried

TRUE-FLAKE completes checkout with saved card (e2e/checkout.spec.ts)
           ████████████████████ 6.0s · 2 attempts
FAILING    declines an expired card (e2e/checkout.spec.ts)
           ████░░░░░░░░░░░░░░░░ 1.1s · 1 attempt

FLAKY.md not written (--no-flaky-md).

The TRUE-FLAKE verdict ​

A test that passes only on attempt ≥ 2 is not a passing test — it's a lucky test. Mjölnir flags it TRUE-FLAKE regardless of the final green checkmark the runner reported.

This is the difference between the two halves of the tool:

EvidenceVerdict quality
Static rulesThe source code"this pattern causes flakiness"
ForensicsActual run history"this test was flaky, here"

Commands ​

CommandWhat it does
mjolnir explain --evidence <dir>Flakiness leaderboard + quarantine proposal
mjolnir explain --playwright ./test-results/Playwright run summary — retries / flakes / slowest

Both accept a directory or a single report file. They read Playwright's JSON reporter output, Jest's and Vitest's JSON reports, and JUnit XML — so pytest, JUnit, TestNG, NUnit and anything else that emits JUnit XML all work.

Reporter package status ​

The MVP path does not require the Mjölnir Playwright reporter package. Use Playwright's built-in JSON reporter, or point the evidence path at an existing JSON/JUnit report file.

The workspace package at packages/playwright-reporter is post-MVP unpublished/source-only. It documents the future wrapper around the same mjolnir.report.json ingestion contract, but it is not available on npm and public setup docs must not ask users to install it.

An honest limitation, by design: Jest's and Vitest's JSON reports record each test's final outcome, not its per-attempt history — so records from those sources always carry one attempt, and TRUE-FLAKE never fires from them (a report that cannot show the attempts must not imply a clean retry history). Retry forensics need Playwright JSON or cross-run JUnit aggregation.

Wiring it up in CI ​

Point Playwright at the JSON reporter, then run forensics on the artifacts after the test step — including when it failed:

yaml
- name: Run tests
  run: PLAYWRIGHT_JSON_OUTPUT_NAME=test-results/results.json npx playwright test --reporter=json

- name: Flakiness forensics
  if: always()
  run: npx mjolnir-qa@6.1.0 explain --evidence ./test-results/

if: always() matters: the runs worth analysing are exactly the ones that didn't pass cleanly.

Released under the MIT License.