Skip to content

QA-CI-001 — continue-on-error masks a failing verification gate ​

← Back to the rule catalog

Generated from the rule registry. Get the same detail from the CLI with mjolnir explain QA-CI-001.

FieldValue
Severityerror
Confidencehigh
Tierquarantine
Measured FP rate11% (n=19)
Evidence levelE2
QA impactFalse-green risk (FALSE-GREEN)
False-positive risk (author estimate)low
Autofix availableno
Languagesyaml
Frameworksgithub-actions, azure-pipelines
Detection strategyFRAMEWORK (parsed YAML + test-command gate (GitHub Actions + Azure DevOps))
Introduced inv0.1.0

Why this fails in production ​

A GitHub Actions job with continue-on-error: true reports its conclusion as success in the Checks API regardless of what actually happened inside it — this is not a UI quirk, it is documented GitHub Actions behavior. A required-status-check branch protection rule that gates on this job's name will pass every time, because from GitHub's perspective the job DID succeed. The team merges on green, the actual test run underneath may have been failing for weeks, and nobody notices until the bug it should have caught reaches production — at which point the CI history shows an unbroken streak of green checkmarks and no one who reviews it will think to suspect this specific job.

What gets flagged (real detector output) ​

Job `security-scan` runs a verification gate under `continue-on-error: true`.

Example from this rule's own must-fire fixture: tests/fixtures/QA-CI-001/must-fire/masked.yml

The fix ​

Remove continue-on-error, or scope it to individual non-blocking steps only.

Confirmed NOT to fire on the corresponding clean pattern ​

Verified against tests/fixtures/QA-CI-001/must-not-fire/clean.yml — a legitimate, similar-looking pattern this rule correctly leaves alone.

Corpus-measured false-positive risk ​

Real occurrence counts from npm run corpus:audit against actively-maintained OSS repos — reproduce yourself, don't just trust this table (see the false-positive audit):

RepoOccurrences
getsentry-sentry1
grafana-grafana1
nextauthjs-next-auth2
positive-fixtures16

Released under the MIT License.