Five Questions Your Green Dashboard Can't Answer

Every fraud solution vendor — including us — can show you a green dashboard. Tests passing. Checks complete. A replay of live-looking traffic processed end to end, zero errors.

Late last quarter, we ran exactly that experiment on our own fraud decision engine. We replayed roughly 300,000 real Bitcoin and Ethereum transactions — public ledger data, no customer data anywhere in the loop — through the full decisioning pipeline. Nothing dropped. Every CI check stayed green.

And the engine returned essentially one decision across the entire replay.

That was the moment we had to admit: our green dashboard was answering questions we had asked, and none of the questions that mattered. This post is about those questions.

Execution is not verification

Most engineering quality gates ask one of two things: did it run, and was it changed. CI runs the suite; review gates check that declared process was followed. Both are necessary. Neither asks: is the behavior correct?

In fraud decisioning, that gap is unusually dangerous, because the failures are silent. A broken payment gateway throws errors. A broken fraud rule returns a decision — calmly, quickly, wrongly. Missing data reads as good news. A rule that cannot fire simply never fires. A test that asserts nothing passes forever.

We found all of these in our own engine, behind months of green CI. The fix was not more tests. It was learning to ask different questions. Here are the five that now anchor our internal test-assurance practice.

Question 1 — Did the checks that protect you actually run?

The most basic question, and the most often assumed. A test suite only protects you if it is actually selected by a named CI lane, executed under the configuration you believe, and able to prove it ran.

What we found: test files that existed since the earliest days of the project and were never selected by any CI lane — green by absence. And “zero findings” results that looked like clean scans but were scans that never actually ran.

What good looks like: every run emits counters — files scanned, tests analyzed, rules evaluated — and a run that lands below a floor reports an error, not a pass. A zero-finding result must be structurally unable to mean “didn’t run.”

The question to ask: Can your suite prove it executed — with counts, not just an exit code?

Question 2 — Can your tests fail for the reason they exist?

A passing test is only evidence if it is capable of failing. Test suites accumulate vacuous tests quietly: tests that call the system but assert nothing, assertion helpers imported and never used, checks that confirm an adjacent truth instead of the proposition they are named for.

What we found: files whose assertion density was a fraction of the suite-wide average — healthy aggregates masking sick regions — and tests that would still pass if the behavior they were named for broke.

What good looks like: static analysis that catches assertion-free tests, per-file density ratchets (because suite-wide averages hide local rot), and the habit of asking of every test: if the behavior broke tomorrow, would this test fail — and fail for the right reason?

The question to ask: Of your last ten production defects, how many would have failed a named test?

Question 3 — Can every rule fire at all?

Rule engines age. Features get renamed, pipelines get refactored, and one day a rule’s condition rests on a signal nothing writes anymore. The rule does not error. It becomes permanently unreachable — and every test around it keeps passing, because the suite and the rule now agree with each other.

What we found: packs of legacy rules that could not fire under any input — conditions comparing constants against thresholds those constants could never cross. Some had been dead since the first commit, through months of green CI.

What good looks like: a per-rule reachability record — fireable, unfireable, or superseded — maintained as an obligation, not an accident; static satisfiability checks on rule conditions; and a witness input demonstrating that each live rule can actually trigger.

The question to ask: For every rule you run in production, when did anyone last see it fire?

Question 4 — Does every signal your decision reads actually exist?

A decision is only as real as the data behind it. If a decision reads a field, something must produce it — through the real construction path, not a test fixture. And defaults are where fraud systems quietly invert: the moment a missing value becomes a benign value, absence has been read as good news.

What we found: decisions reading fields that no code path ever wrote, and a device-trust signal that scored maximum trust when the device data was missing. One reviewed, tested fix at a lower layer changed nothing observable, because a layer above it kept manufacturing a reassuring value.

What good looks like: a read-versus-written audit — every field a decision reads has a named producer; an inventory of every benign default with an explicit verdict; and a standing rule that a missing signal must never be observably assessed as safe. Absence is data.

The question to ask: When a signal is missing, what does your engine do — and did someone decide that, or did it just happen?

Question 5 — Do your aggregate results mean anything?

Our replay was the lesson here. Zero dropped events. Deterministic runs. And one uniform decision across roughly 300,000 events — caused not by fraud patterns but by our own sampling design starving the engine’s history features. The pipeline worked perfectly. The decisions meant nothing.

Uniformity is suspicious. But variety is not evidence either. A healthy-looking spread of decisions can be produced entirely by the rules that work, while dead rules contribute nothing and leave no trace in the total.

What good looks like: a uniform-decision alarm — a large batch collapsing onto one outcome is a finding, not a result; provenance for sampling and replay design attached to every aggregate; mutation testing to prove the tests can actually be killed; and determinism checks on generated-input suites.

The question to ask: If your system returned the same decision for 99% of a month of traffic, who would notice — and how?

Keeping the five questions honest

Five questions can themselves become a green dashboard, so we codified ours into an internal test-assurance practice: five closed dimensions — Execution, Verification, Reachability, Dataflow, Robustness — each with its own rules and retained evidence artifacts, rolled out in stages, advisory-first.

Two design decisions matter more than any individual rule. First, every dimension states what it does not prove. Execution proves the lane ran; it does not prove the tests assert anything. Reachability proves a rule can fire under a constructed input; it does not prove the threshold is calibrated. A metric that claims everything proves nothing.

Second, the dimensions are paired counter-controls for each other’s blind spots. Verification’s hole — a test can assert the wrong proposition convincingly — is covered by Robustness’s mutation testing. Execution’s hole is covered by Verification. No single metric is trusted alone.

We do not present this as finished. The specification is written and staged; some checks run today, others are on our roadmap. The discipline — asking the questions, and recording the answers as evidence — is the point.

A checklist you can use this week

Whether you build a decision engine or evaluate one, the five questions work as an audit:

1. Ask each check to prove it ran — counts and scope, not just exit codes.
2. Take your last ten incidents and trace each to the test that should have failed.
3. List your rules and ask for the last observed trigger of each.
4. Pick five fields your decisions read and name their producers.
5. Look at last month’s decision distribution. If it is flat, investigate before you celebrate.

None of this requires new infrastructure. It requires treating a green check as a claim, not a conclusion.

Conclusion

Confidence is not evidence. A green dashboard answers the questions you asked — and says nothing about whether those were the right questions to ask.

In fraud decisioning, the cost of the unasked question is rarely a crash. It is a quiet wrong decision, made at scale. The five questions above are the ones we now refuse to leave unasked.

Run-True Decision is building a fraud decision engine designed specifically for Southeast Asian banks — with true on-premise deployment, pre-built banking templates, and regional pricing — and we hold our own test suite to the questions in this post. Talk to us to learn more.