Skip to content

False-positive audit ​

The north-star metric is a false-proof rate near zero. A rule that fires on code where nothing is actually wrong erodes trust in the whole tool, so the rules that ship in the headline tiers carry a false-positive rate measured against real open-source code, not an author's guess.

The table below is regenerated from tests/corpus/verdicts/*.jsonl; the raw source is on GitHub.

Each finding below was hand-classified by reading its source context. The FP rate is FP / (TP + FP) — UNSURE verdicts are excluded from the denominator (they add sample size but not confidence in either direction).

detectorRev is the detector implementation revision the measurement was taken against (sidecar: tests/corpus/detector-revisions.json). A rule whose detection logic changes without a revision bump has its measurement treated as stale → provisional (Verification Trust Evolution Plan §07).

The 95% Wilson CI column is the interval regression governance compares (plan §20.2): an FP-rate regression is flagged only when the intervals are disjoint in the bad direction with a worse point estimate, at n ≥ 10 on both sides. Noise within the interval overlap is never a CI failure — comparisons below n = 10 are informational only.

A rule with fewer than 10 classified verdicts is unmeasured. Coverage below is stated against the full rule registry, not against the rules that happen to have been sampled.

Summary ​

Rule IDFP Rate95% Wilson CISample (n)TPFPUNSUREdetectorRevStatus
QA-CI-00111%[0.0294, 0.3139]1917203🕤 quarantine
QA-CI-00211%[0.031, 0.328]1816204△ extended
QA-CI-0058%[0.0137, 0.3331]1312102🕤 quarantine
QA-CI-0070%[0, 0.2588]1111003△ extended
QA-CI-00810%[0.0179, 0.4042]109104🕤 quarantine
QA-CI-0090%[0, 0.2775]1010003△ extended
QA-CI-01010%[0.0179, 0.4042]109102🕤 quarantine
QA-CS-1010%[0, 0.1611]2020001△ extended
QA-CS-1028%[0.0232, 0.2585]2422202△ extended
QA-CS-1030%[0, 0.2588]1111002△ extended
QA-CS-1040%[0, 0.2775]1010001△ extended
QA-CS-10525%[0.1018, 0.495]1612402△ extended
QA-CS-10633%[0.1381, 0.6094]128402🕤 quarantine
QA-CS-1078%[0.0149, 0.3539]1211101△ extended
QA-CS-1090%[0, 0.2775]1010001△ extended
QA-CYP-00120%[0.0705, 0.4519]1512301△ extended
QA-CYP-0020%[0, 0.2775]1010001🕤 quarantine
QA-CYP-0030%[0, 0.2775]1010001🕤 quarantine
QA-ENV-001100%[0.8389, 1]2002004🕤 quarantine
QA-JV-1010%[0, 0.0989]3535001… core
QA-JV-10226%[0.1255, 0.4647]2317601△ extended
QA-JV-10326%[0.1635, 0.3838]58431502△ extended
QA-JV-10420%[0.0567, 0.5098]108201△ extended
QA-JV-10510%[0.0279, 0.301]2018202△ extended
QA-JV-10633%[0.1381, 0.6094]128402🕤 quarantine
QA-JV-1070%[0, 0.2775]1010001△ extended
QA-JV-1090%[0, 0.1759]1818001△ extended
QA-PW-0020%[0, 0.1611]2020001△ extended
QA-PW-00310%[0.0179, 0.4042]109101△ extended
QA-PW-00443%[0.2138, 0.6741]148602🕤 quarantine
QA-PW-1010%[0, 0.1611]2020001△ extended
QA-PW-10210%[0.0179, 0.4042]109102🕤 quarantine
QA-PW-1040%[0, 0.2775]1010001△ extended
QA-PW-1130%[0, 0.2588]1111001△ extended
QA-PW-11556%[0.3318, 0.769]167901🕤 quarantine
QA-PW-1160%[0, 0.2775]1010001↔ straddling (interval crosses the tier boundary)
QA-PW-1170%[0, 0.0989]3535001… core
QA-PW-1210%[0, 0.2425]1212001△ extended
QA-PW-1226%[0.027, 0.1381]8075501△ extended
QA-PW-12345%[0.2127, 0.7199]116501🕤 quarantine
QA-PW-1247%[0.0119, 0.2982]1514102↔ straddling (interval crosses the tier boundary)
QA-PW-1250%[0, 0.2775]1010002△ extended
QA-PW-1400%[0, 0.2775]1010001△ extended
QA-PW-1419%[0.0314, 0.2357]3330301△ extended
QA-PW-14218%[0.0514, 0.477]119201△ extended
QA-PW-1436%[0.027, 0.1381]8075501△ extended
QA-PW-14414%[0.0498, 0.3464]2118301△ extended
QA-PW-14612%[0.0329, 0.3434]1715201🕤 quarantine
QA-PY-0010%[0, 0.2425]1212001△ extended
QA-PY-00212%[0.0417, 0.2996]2522301△ extended
QA-PY-00352%[0.3522, 0.675]33161704🕤 quarantine
QA-PY-00467%[0.5155, 0.7899]42142803◐ unmeasured
QA-PY-00513%[0.0454, 0.3213]2320301△ extended
QA-PY-00775%[0.4677, 0.9111]123904◐ unmeasured
QA-PY-0096%[0.0099, 0.2576]1817101△ extended
QA-PY-01110%[0.0179, 0.4042]109101△ extended
QA-PY-01240%[0.2459, 0.5768]30181201🕤 quarantine
QA-PY-1010%[0, 0.2775]1010001↔ straddling (interval crosses the tier boundary)
QA-PY-1038%[0.0222, 0.2497]2523201△ extended
QA-PY-1040%[0, 0.2775]1010002🕤 quarantine
QA-PY-1050%[0, 0.2425]1212002🕤 quarantine
QA-PY-1060%[0, 0.2153]1414001↔ straddling (interval crosses the tier boundary)
QA-PY-1070%[0, 0.2775]1010001↔ straddling (interval crosses the tier boundary)
QA-PY-1080%[0, 0.1936]1616001🕤 quarantine
QA-SE-0010%[0, 0.2775]1010001🕤 quarantine
QA-SE-0028%[0.0137, 0.3331]1312101🕤 quarantine
QA-SE-0030%[0, 0.2588]1111001🕤 quarantine
QA-TEST-00160%[0.3866, 0.7812]2081201🕤 quarantine
QA-TEST-00262%[0.4088, 0.7925]2181301🕤 quarantine
QA-TEST-00322%[0.1408, 0.3216]78611701🕤 quarantine
QA-TEST-00430%[0.1455, 0.519]2014601△ extended
QA-TEST-00636%[0.1517, 0.6462]117401🕤 quarantine
QA-TEST-01058%[0.4077, 0.7358]31131801🕤 quarantine
QA-TQUAL-00253%[0.3645, 0.6913]32151701🕤 quarantine
QA-TQUAL-00933%—3210—❓ unmeasured
QA-TQUAL-01124%[0.115, 0.4343]2519601△ extended

Tier Assignment Criteria ​

TierFP RateMeaning
✅ core≤ 10%Ships in the default report
⚠️ extended≤ 30%Included by default, lower confidence
🔴 quarantine> 30%Opt-in only (--strict)
❓ unmeasuredn < 10Cannot ship in core until measured

Coverage: 73/79 rules measured (92%) at n ≥ 10 ​

6 rules carry no measured FP rate. Any of them in the core tier is shipping on an unverified assumption.

Released under the MIT License.