Runtime forensics
Static flakiness detection is guessing. A rule can tell you a test looks flaky — a hard sleep, a networkidle wait — but it cannot tell you whether that test actually failed last Tuesday.
The evidence path reads real execution data: Playwright JSON reports, Jest JSON (--json --outputFile), Vitest JSON (--reporter=json) and JUnit XML from any runner.
The command moved. The
forensicsandtriageverbs were retired in the v6 carve; the capability ismjolnir explain --evidence <dir>. Only the way you reach it changed. See the CLI contract for the disposition of every verb.
▍ FLAKINESS LEADERBOARD
3 tests · 1 failed · 1 flaky · 1 retried
TRUE-FLAKE completes checkout with saved card (e2e/checkout.spec.ts)
████████████████████ 6.0s · 2 attempts
FAILING declines an expired card (e2e/checkout.spec.ts)
████░░░░░░░░░░░░░░░░ 1.1s · 1 attempt
FLAKY.md not written (--no-flaky-md).
The TRUE-FLAKE verdict
A test that passes only on attempt ≥ 2 is not a passing test — it's a lucky test. Mjölnir flags it TRUE-FLAKE regardless of the final green checkmark the runner reported.
This is the difference between the two halves of the tool:
| Evidence | Verdict quality | |
|---|---|---|
| Static rules | The source code | "this pattern causes flakiness" |
| Forensics | Actual run history | "this test was flaky, here" |
Commands
| Command | What it does |
|---|---|
mjolnir explain --evidence <dir> | Flakiness leaderboard + quarantine proposal |
mjolnir explain --playwright ./test-results/ | Playwright run summary — retries / flakes / slowest |
Both accept a directory or a single report file. They read Playwright's JSON reporter output, Jest's and Vitest's JSON reports, and JUnit XML — so pytest, JUnit, TestNG, NUnit and anything else that emits JUnit XML all work.
Reporter package status
The MVP path does not require the Mjölnir Playwright reporter package. Use Playwright's built-in JSON reporter, or point the evidence path at an existing JSON/JUnit report file.
The workspace package at packages/playwright-reporter is post-MVP unpublished/source-only. It documents the future wrapper around the same mjolnir.report.json ingestion contract, but it is not available on npm and public setup docs must not ask users to install it.
An honest limitation, by design: Jest's and Vitest's JSON reports record each test's final outcome, not its per-attempt history — so records from those sources always carry one attempt, and TRUE-FLAKE never fires from them (a report that cannot show the attempts must not imply a clean retry history). Retry forensics need Playwright JSON or cross-run JUnit aggregation.
Wiring it up in CI
Point Playwright at the JSON reporter, then run forensics on the artifacts after the test step — including when it failed:
- name: Run tests
run: PLAYWRIGHT_JSON_OUTPUT_NAME=test-results/results.json npx playwright test --reporter=json
- name: Flakiness forensics
if: always()
run: npx mjolnir-qa@6.1.0 explain --evidence ./test-results/if: always() matters: the runs worth analysing are exactly the ones that didn't pass cleanly.