The Checks That Had Never Failed

This is Part 2 of the account. Read Part 1: “Exit Code Zero.”

We had eight checks report success or a state that was not true.

A benchmark exited zero without producing data. A deployment printed a completion message over a partial failure. A process monitor matched its own command line. A safety gate’s central guarantee depended on a claim supplied by the caller.

Afterward, we articulated what we thought was our lesson:

A gate whose purpose is to stop bad work must have a proven failing case before it ships.

Then we looked for prior art. We found that the idea was not new at all. Mature engineering practices have approached the same problem from several directions.

Four practices, one operational principle

Mutation testing deliberately changes code and asks whether the test suite detects the change. A surviving mutation is evidence that the suite may not protect the behavior it appears to cover. Google’s published work on mutation testing at scale keeps this idea but filters and targets mutations so the signal remains useful in a very large codebase (Petrović and Ivanković, 2018).

Test-driven development starts with a test for behavior that does not yet exist, observes that test fail, implements the behavior, and then refactors. The red step is operational evidence that the test is capable of distinguishing absence from presence (Kent Beck, Test-Driven Development: By Example).

Protocol maintenance guidance has moved away from treating silent tolerance as a universal virtue. RFC 9413 warns that tolerating unexpected input “conceals problems,” making defects harder to correct later (RFC 9413, Maintaining Robust Protocols).

Fail-fast design makes errors immediate and visible instead of allowing an invalid state to travel further through the system. Jim Shore’s formulation connects that visibility with more robust software, because the failure appears closer to its cause (Jim Shore, “Fail Fast,” 2004).

These practices are not identical. Mutation testing assesses a suite. TDD shapes development. Protocol guidance addresses interoperability. Fail-fast design addresses error propagation. What they share is the demand for negative evidence: a protection mechanism must demonstrate that it can distinguish the bad condition from the good one.

The research finding that reframed our incident

Yuan and colleagues studied 198 randomly selected, user-reported failures across five distributed data systems. Among the catastrophic failures in their sample, 92% resulted from incorrect handling of non-fatal errors that software had explicitly signaled. They also found that 58% could have had their underlying faults exposed through straightforward testing of error-handling code. Those are findings about the systems and sample studied, not universal rates for all software (Yuan et al., OSDI 2014).

Our incident was smaller, but the pattern was recognizable:

• the benchmark raised a missing-file error;
• the API rejected every request;
• the deployment recap recorded failed tasks;
• the package manager reported a configuration error.

The errors were not hidden inside an unknown race or a model nobody could inspect. The tools signaled them. Our surrounding scripts discarded, redirected, or outlived those signals and then reported success.

The problem was not a lack of error information. It was a failure to make that information decisive.

What actually happened—and two adjacent tool examples

The incident mechanisms were direct:

• the deployment command was followed by || true, which discarded its non-zero status;
• load-generator stderr was discarded;
• detailed deployment output was written to a remote-only log; and
• the Ansible recap accurately recorded the failed tasks even though the harness later printed a completion message.

The incident did not use a Bash pipeline or an Ansible rescue block. Those documented semantics are adjacent examples of the broader risk: continuation can resemble success when a workflow does not assert its postcondition.

By default, Bash assigns a pipeline the exit status of its final command. Without pipefail, an earlier command can fail while a later logging command succeeds, leaving the pipeline green (GNU Bash Reference Manual, Pipelines).

Ansible rescue blocks are designed to recover from failed tasks. When rescue succeeds, Ansible continues the play and treats the rescued task as successful for several control-flow purposes, while still retaining failure information in its statistics (Ansible documentation, Blocks).

Neither adjacent behavior is a defect, and neither caused this incident. Both are useful in the right context. Our actual failure was that || true, discarded or remote-only output, and missing postconditions allowed an accurately recorded failure to be followed by a success message.

That distinction changes the fix. Replacing one tool with another would not solve a missing postcondition. The harness must ask whether the result file exists and is valid, whether requests succeeded, whether the database exists, whether required services are active, and whether every dependency role passed the hardware gate.

The cost objection is real

The obvious objection is that proving every check can fail would create an enormous testing tax.

Google’s mutation-testing work supports that objection. Traditional exhaustive mutation analysis is computationally prohibitive at large scale, so the production approach is incremental, filtered, and focused on changed code and useful mutation operators (Petrović and Ivanković, 2018).

Safety assurance makes the proportionality principle even clearer. FAA material describes the rigor used for airborne software and electronic hardware as associated with the system’s risk level, and FAA guidance recognizes DO-178C as an acceptable means for addressing airborne software assurance (FAA, Abstraction Layer Information; FAA AC 20-115D).

That does not make aviation guidance a direct rule for a benchmark harness. It supplies a useful principle: verification effort should track consequence. Our conclusion is narrower and explicitly ours—not a statement attributed to DO-178C.

The exact objective counts often repeated in secondary summaries are not included here. We verified the risk-scaled principle and the FAA’s recognition of DO-178C in primary FAA material, but not the counts against the standard itself.

Where we draw the line

“Prove every check can fail” is too broad.

In our normal development loop, many tests demonstrate their negative case while a feature is being written. We see them fail, make them pass, and keep them in the regression suite. Exhaustively injecting faults into every test would add cost without equal value.

Long-lived gates are different. A deployment check, release gate, benchmark preflight, alarm, or policy control may remain green for months. If nobody has constructed the prohibited condition and watched the gate reject it, a working gate and a decorative gate look identical.

Our rule is therefore:

A gate whose purpose is to stop bad work must have a proven failing case before it ships.

A useful proof answers four questions:

1. What exact bad condition is the gate meant to stop?
2. What fixture or invocation constructs that condition?
3. Does the gate fail loudly, with an unambiguous non-success result?
4. Can the caller omit, override, or falsely attest the evidence the gate depends on?

The fourth question came directly from our incident. Our first hardware gate accepted a caller’s claim that all roles had been covered. A reviewer supplied only part of the fleet and showed that the gate could still pass. The gate was not enforcing coverage; it was trusting the caller to report coverage.

The corrected gate parses structured role-and-host inputs, requires every dependency role, requires enough peers to make comparison meaningful, rejects duplicates and invalid values, and describes a pass narrowly. A pass means only that no supplied node is materially slower than its peers. It does not claim that deployment, data, rate limits, or the benchmark harness are otherwise ready.

Apply the standard where it failed first

We are applying this rule to one surface: the benchmark harness that produced the eight false assurances.

Not the whole repository. Not every test. Not every deployment workflow.

That restraint is deliberate. Extending a standard everywhere on the strength of one incident would repeat the mistake that caused the incident: asserting broad confidence before the evidence exists.

The current status remains unfinished:

• One of eight failures now has a code-level guard with the narrow pass claim above.
• Seven are documented and can still recur.

The new rule earns wider adoption only if it helps convert those seven postconditions from prose into behavior and catches failures without creating unacceptable noise.

The part we had overlooked

We applied rigorous testing to product code while grading too much of the surrounding tooling by a single signal: did the command finish green?

But CI gates, deployment scripts, benchmark harnesses, and operational alarms make decisions too. They decide whether work advances, whether evidence is admitted, and whether people stop investigating. A false green in that layer can be more persuasive than the error it suppressed.

The checks we trusted most were the checks we had never watched fail.

Engineering takeaway: List the gates that can stop a release, deployment, or benchmark. For each one, record the last proven failing case. If there is no such case, build the smallest safe one before expanding the gate’s authority. Talk to us if you want to compare approaches.

Part of 100 Days Building RTD in Public.