Where
QA-CI-009 · .github/workflows/ci.yml:10The rule, and the exact file and line.
Tests tell you what passed. Mjölnir tells you what you can trust.
Mjölnir finds tests that cannot fail and pipelines that cannot go red, then scores how far you can trust the result.
npx mjolnir-qa@latest$ npx mjolnir-qa@latest WORTHINESS 75/100 NEEDS WORK ░░░░░░░ The hammer holds — but 19 findings weigh it down. (40 raw pts / 7 test declarations — normalized) (capped: deduction mass 40 pts — absolute ceiling 85) DETECTED [playwright] ▍ DIAGNOSTICS BY CATEGORY QA-CI ░░░░ 76 QA-PW ░░ 87 QA-TEST 97 ▍ WHERE POINTS WERE LOST ╭─────────────────────────────────────────╮ │ 4 × error − 32 │ │ 3 × warning − 7 (evidence-discounted) │ │ 12 × info − 1 (evidence-discounted) │ ╰─────────────────────────────────────────╯ ▍ FIX THIS FIRST +8 pts QA-CI-009 · .github/workflows/ci.yml:10 +8 pts QA-CI-009 · .github/workflows/ci.yml:10 +8 pts QA-CI-009 · .github/workflows/ci.yml:26
A real scan of the demo repo, replayed.
Works with your stack
- Languages
- TypeScript
- JavaScript
- Python
- Java
- C#
- Test frameworks
- Playwright
- Cypress
- Selenium
- Jest
- Vitest
- Mocha
- pytest
- JUnit
- TestNG
- NUnit
- xUnit
- MSTest
- CI
- GitHub Actions
- Azure Pipelines
- Jenkins
- GitLab CI
What a scan covers
- 01CI integrityThe workflow tricks that keep a failed run green.
- 02FindingsWhere each one is, how sure it is, and the fix.
- 03Worthiness scoreOne number, with the arithmetic shown.
- 04Evidence and trustRead from your code, or seen in a real run.
- 05Runtime forensicsTrue flakes and fragile selectors, from real runs.
- 06Measured rulesA published false-positive rate for each rule.
- 07CI and AI agentsGate the pull request, and verify the agent's fix.
01CI integrity
Catches the CI tricks that keep a failed run green.
Each of these lines looks deliberate in review. Mjölnir reads the workflow and flags each one with its rule and a fix.
3errors
2notes
33of 33 lines read
- ERRORQA-CI-009 Job `test` pipes the test command into another tool without `set -o pipefail`.
- ERRORQA-CI-009 Job `deploy-smoke` sequences commands with `; ` after the test command — the test result does not fail the step.
- INFOQA-CI-010 Job `deploy-smoke` runs tests but its `if:` condition skips it on pull requests.
- ERRORQA-CI-009 Job `deploy-smoke` pipes the test command into another tool without `set -o pipefail`.
- INFOQA-CI-001 Job `security-scan` runs a verification gate under `continue-on-error: true`.
1name: CI
2
3on: [push, pull_request]
4
5jobs:
6 test:
7 runs-on: ubuntu-latest
8 steps:
9 - uses: actions/checkout@v4
10 - run: npm test | tee test-results.log
QA-CI-009 Job `test` pipes the test command into another tool without `set -o pipefail`. E2 · deterministic · measured FP 0% · n=10
QA-CI-009 Job `deploy-smoke` sequences commands with `; ` after the test command — the test result does not fail the step. E2 · deterministic · measured FP 0% · n=10
11 - run: yarn test --coverage | tee coverage.log
12
13 lint:
14 runs-on: ubuntu-latest
15 steps:
16 - uses: actions/checkout@v4
17 - run: npm run lint; npm run typecheck
18
19 deploy-smoke:
20 runs-on: ubuntu-latest
21 if: github.event_name == 'push'
QA-CI-010 Job `deploy-smoke` runs tests but its `if:` condition skips it on pull requests. E0 · observation · measured FP 10% · n=10
22 steps:
23 - uses: actions/checkout@v4
24 - run: npm test; npm run smoke
25 - run: vitest run --coverage | tee vitest-results.log
26 - run: pnpm test | tee pnpm-results.log
QA-CI-009 Job `deploy-smoke` pipes the test command into another tool without `set -o pipefail`. E2 · deterministic · measured FP 0% · n=10
27
QA-CI-001 Job `security-scan` runs a verification gate under `continue-on-error: true`. E0 · observation · measured FP 11% · n=19
28 security-scan:
29 continue-on-error: true
30 runs-on: ubuntu-latest
31 steps:
32 - uses: actions/checkout@v4
33 - run: npm audit --audit-level=high
Lines that look deliberate in review
continue-on-error: trueThe job fails and the workflow still reports success.
QA-CI-001 continue-on-error masks a failing verification gatenpm test || trueThe test command's exit code is thrown away.
QA-CI-002 Ignored exit code (|| true)npm test | tee results.logOnly the exit code of tee reaches CI.
QA-CI-009 Test command does not propagate exit codetest.only("checkout", ...)The focused test runs and the rest are skipped.
QA-TEST-001 Focused test committed
Above: every finding the demo repo scan reported for .github/workflows/ci.yml, at the line it reported. With SARIF upload or mjolnir summary, this is how they show up in a pull request.
02Findings
Every finding says where, how sure, and how to fix it.
This is the first finding from the demo repo scan, exactly as the terminal prints it. Each part of it answers one question.
See what it checks✗ ERROR QA-CI-009 · .github/workflows/ci.yml:10 [E2 · deterministic · measured FP 0% · n=10] Finding Job `test` pipes the test command into another tool without `set -o pipefail`. Fix Add `shell: bash` with `set -o pipefail`, or split into two steps so the test command's exit code is preserved.
How sure
E2 · deterministicE2 is proven in code and counts in full. E1 is a matching pattern and counts half. E0 is an observation and costs nothing.
How often the rule is wrong
measured FP 0% · n=10Measured on hand-checked findings from open-source repos.
The fix
FixThe change that closes the finding. Re-run the scan to confirm it.
03Worthiness score
One score, with the arithmetic shown.
The score measures the test suite, not your product. Every point it takes off is listed, and the formula has no hidden second model.
Read how the score works75
UNWORTHY0–49NEEDS WORK50–79WORTHY80–99FORGED100
Step by step
- Deductions
- 40 points across 7 test declarations
- Rate
- 40 ÷ (7 + 1) = 5
- Score
- 100 − min(100, 5 × 5) = 75NEEDS WORK
Dividing by test declarations means adding empty spec files cannot raise the score. Three ceilings then cap it, and the scoring guide lists them.
04Evidence and trust
Knows the difference between reading code and seeing it run.
Most findings come from reading your code. Give Mjölnir the report of a real test run and it can confirm the code actually ran.
Read the definitions✗ ERROR QA-CI-009 · .github/workflows/ci.yml:10 [E2 · deterministic · measured FP 0% · n=10] Finding Job `test` pipes the test command into another tool without `set -o pipefail`. Fix Add `shell: bash` with `set -o pipefail`, or split into two steps so the test command's exit code is preserved.
Mjölnir read the workflow file. Nothing ran, so this finding stays at L2, proven in code.
✗ ERROR QA-PW-101 · e2e/checkout.spec.ts:6 [E2 · deterministic · measured FP 0% · n=20 · trust L3 · runtime: file executed] Finding `waitForTimeout()` hard sleep. Fix Replace with a web-first assertion (`await expect(locator).toBeVisible()`) or `locator.waitFor()`.
The Playwright report shows this spec file ran, so the finding reaches L3.
- L0NotedWorth knowing. Not a claim that anything is wrong.
- L1Looks like itA pattern that usually means the problem was matched.
- L2Proven in codeThe code as written has the problem.
- L3The file ranA real run executed the file the finding is in.
- L4The test ranA real run executed the test the finding is in.
- L5The run agreesThe run's own result confirms the problem.
05Runtime forensics
Reads what actually ran.
Point it at a folder of test results. Playwright JSON, Jest and Vitest JSON, and JUnit XML from any runner all work.
Read the forensics guide▍ FLAKINESS LEADERBOARD3 tests · 1 failed · 1 flaky · 1 retriedTRUE-FLAKE completes checkout with saved card (e2e/checkout.spec.ts) ████████████████████ 6.0s · 2 attemptsFAILING declines an expired card (e2e/checkout.spec.ts) ████░░░░░░░░░░░░░░░░ 1.1s · 1 attempt
TRUE-FLAKE means the test failed at least once and then passed. Mjölnir flags it even though the final check was green.
▍ SELECTOR HEALTHe2e/login.spec.ts [█████████████░░░░░░░] 65 / 100 role/text: 1 · testid: 0 · plain-css: 0 · css-chains: 1 ⚠ · xpath: 0e2e/checkout.spec.ts [█████████████████░░░] 86 / 100 role/text: 3 · testid: 1 · plain-css: 0 · css-chains: 1 ⚠ · xpath: 0
Selector health grades how each locator finds its element. It measures resilience, not correctness.
06Measured rules
A published false-positive rate for each rule.
Each rate comes from at least 10 hand-classified findings in open-source repos. Rules with bad rates stay listed, in quarantine.
See all 101 rules95of101
rules measured on real code
| Rule | Catches | False positives | Sample |
|---|---|---|---|
| Core | |||
| QA-PW-002 | Unawaited Playwright assertion | 0% | n=20 |
| QA-PY-001 | Focused test committed | 0% | n=12 |
| QA-CS-103 | Test without assertions | 0% | n=11 |
| Extended | |||
| QA-JV-103 | Test without assertions | 26% | n=58 |
| Quarantine | |||
| QA-TEST-001 | Focused test committed | 60% | n=20 |
07CI and AI agents
Gates the pull request, and checks the agent.
Scan only what a branch touched, or hand the findings to an AI agent and let the re-scan prove the fix.
Connect an agentIn CI
With --scope changed it scans the files the branch touched and exits non-zero on new findings.
npx mjolnir-qa@latest --scope changedOr add the GitHub Action:
- uses: Sergey-Bar/Mjolnir@v1
with:
scope: changed
fail-on: errorWith an AI agent
The agent writes the fix and Mjölnir re-scans to prove it. Each finding in the handoff says whether it is safe to apply or needs a person to confirm.
claude mcp add mjolnir -- npx -y mjolnir-qa@latest mcp- 0
- Clean at the gate
- 1
- Findings at or above the gate
- 2
- Partial scan. Never blocks.
- 10
- Usage error
- 20
- Internal error
Exit codes are frozen, so you can build CI logic on them.
Where the score stops
- It does not run your tests. A clean scan is not a passing suite.
- It cannot tell when an assertion checks the wrong value.
expect(total).toBe(41)looks healthy to it. - A perfect score means zero findings. Whether the suite covers your real risk is a separate question.
- An empty repo scores null. With no tests to read, it has nothing to score.