The Search Was Accurate. The Conclusion Was Not.

Ask anyone who runs AI coding agents why their searches go wrong and you get the same answer: the agent writes bad patterns. Too broad, too narrow, a stray character. Teach it better regexes and the problem goes away.

We checked — five months of agent transcripts, about 1,500 session files, 1.8 GB — looking for every time an agent noticed one of its own searches had lied to it.

The bad-pattern explanation is real but it is small. The pattern-writing signals sit at the bottom of the observed counts. The largest signal is not a bad pattern at all. And the correction that matters most is that the problem is usually not in the search string.

What we found

146 de-duplicated reported incidents, across five months, in at least 8 separate projects. Not rare, and not one team’s bad habit. It shows up in the sessions doing wide, cross-repository work, which is where searching is load-bearing.

In the dated August 12 snapshot, the study-time count manifest found the phrase “bad grep” in 3 of 1,531 transcript files. All three files came from the discussion that prompted this investigation, not unrelated agent work. Agents usually described the failure in other words.

my grep was a false zero

the grep was accurate — the conclusion was not

That matters more than a naming quirk. Go looking for this failure using the words you use for it, and you will find almost nothing and conclude you do not have the problem.

The detail we like least, and think is the most useful

We were counting bad searches with a search — exactly the trap under study. So the counting had two layers: phrases that name a search failure directly, and generic admissions like “would have reported” that counted only if a search-related word appeared nearby.

Before trusting those counts, we used two different controls. A corpus canary showed that the pipeline could recover a known marker read from an included transcript. Pattern canaries checked known matches and a near miss before the patterns touched the corpus. Those controls showed that the pipeline could see expected bytes. They could not prove that a meaning was absent when a search returned zero.

That second filter rejected 132 of 205 generic-tier matched lines. They were real sentences about entirely different things — a security scan that “would have reported” a decoy key file, a word-count check, a disk-space probe.

Nearly two thirds of those generic-tier matched lines — 64% — were rejected. The naive pass produced 532 matched lines; the filter left 400 matched lines. That would have inflated the matched-line total by about 33%. The headline figure of 146 uses a different unit: it is the number of incidents after de-duplication by project, day and normalized phrase. The study of bad searches was one unexamined filter away from being a bad search. We would have published a confident, wrong figure, and nothing in the output would have hinted at it.

That is the whole failure mode in one sentence: it does not produce an error. It produces a plausible number.

Four corrections to our own published explanation

We had already written up an explanation of this failure, based on one week in one project. The five-month data contradicts it in four places. The data wins.

These observed signals overlap and are not a partition. They were counted within ±300 characters of a direct search-failure admission. One incident can carry more than one signal — a flag the tool lacks and discarded errors can appear in a single event — so the rows sum to 266, which is more than the 146 de-duplicated incidents. Read the rows against each other, not as shares of a whole.

Two limits matter when reading those counts. The transcript corpus was live, and at least one file disappeared during the sweep. Phrases spanning escaped newlines were not matched, while incident de-duplication could merge two separate events of the same kind on the same day. The counts are therefore a dated snapshot and a floor, not a complete total. The audit measured this once against the live corpus available that day. Its study-time file-count manifest and signal-count output are preserved and hash-verified; the transcript contents are not frozen, so re-running the changed live corpus would be a different study.

1. The “bad pattern” signals are the smallest.

Markdown splitting a phrase (12), a pattern one word off (9), and case sensitivity (3) are the three smallest observed signals in the table. They felt central to us because pattern problems had happened to us recently. Recency is not frequency.

2. The two largest signals are not pattern problems.

A shell wrapper with its own default filters led the table at 58 nearby matches. Discarded error output followed at 50 and can turn a failed command into a silent zero. This condensed account preserves one session’s finding while replacing its internal names:

I ran <tool> list 2>/dev/null | grep -i "<project>", got nothing, and reported that no such sessions existed — then acted on it. The grep was accurate. The conclusion was not, because the plain command omits exactly the rows I was looking for.

The pattern was fine. The command never saw the data. Calling this class “bad grep” misdescribes it and points the fix at the wrong place — you can polish that regex forever and never reach the bug.

3. One expected failure mode did not appear in the tested set.

We expected the tool’s binary-file detection to be skipping files silently. On the dated transcript snapshot, we searched the same fixed marker across every transcript twice and counted the per-file result lines. The pass that allowed binary skipping (-I) and the pass that forced text handling (-a) each produced a result line for all 1,531 transcript files. We found no evidence that binary handling reduced that tested file set. This does not establish what happens in other corpora.

4. There is a fifth class we had not named.

One session described it this way. The bracketed words complete the source’s shorthand, and the ellipsis marks the truncated excerpt:

my regex was not unanchored or too broad in the syntactic sense — it was semantically correct and [matched nearly everything]…

A pattern can be perfectly written and still useless, because in that population it matches almost every record. It looks correct, it runs correctly, and its output carries no information. Nothing in the result tells you which situation you are in.

The part that generalises

The underlying limit is:

A search proves only that exact bytes were not found in an exact set of files. It never proves a meaning is absent.

Every failure above is a version of that gap. The agent wants to know “is this rule written down anywhere”, runs a search, gets nothing, and reports the rule missing — when what it established is far narrower.

The obvious fix is to blame the tool and switch. We measured that too. Take a line that really exists in one of our files, written with markdown emphasis:

is **named** in the PR body

Search all three common tools for the phrase as a human would say it:

git grep -F 'named in the PR body'     → no match
grep -F   'named in the PR body'       → no match
rg -F     'named in the PR body'       → no match

All three correctly report that those exact, unformatted bytes are absent. None answers whether the same meaning is present with markdown inserted. The canary confirms that the tools can find the stored form: **named** in the PR body is found by all three.

Switching tools does not answer the broader meaning question. Each answered the exact-byte question it was given, correctly and fast.

What actually works

The effective controls are not exotic, and all of them are cheap:

• Use a corpus canary. First prove that the full pipeline can recover a known marker from the exact files you intend to search.
• Use pattern canaries. Test known matches and a near miss before running the pattern across the corpus.
• Do not treat either canary as proof of semantic absence. A consequential approval, denial, deletion, or closure requires checking the authoritative source or using a structured validator built for that question.
• Never discard error output on a check. The error is the only thing separating “no matches” from “the command did not run”.
• Check non-empty before checking a value. An empty result and a zero result are different facts.
• State the corpus in the sentence. “Not found in these 40 files” is defensible. “Not found” is not.
• Never make a consequential decision on the strength of one zero-match search.

What we did not measure

We counted the times an agent noticed and said so. We could not count the times it happened and nobody noticed.

That population is not measurable at this scale, and we deliberately give no number for it. An empty search result is textually identical to a legitimately empty one, and “someone established the truth later by other means” leaves no marker to search for.

We could have estimated. An invented number would have been worse than none — and it would have been precisely the failure this post is about.

One signal, not a measurement: discarded error output was one of the two largest observed failure signals, and it produces a silent zero. That is suggestive about the unnoticed pile. Suggestive is not measured, and we are leaving it there.

One more caveat we will not hide: reports rose sharply across the five months, but the vocabulary we searched for was invented during that window. A rise in reports may mean better noticing, not more failures. We cannot separate the two here, so we claim no trend.

For a founder building fraud decision systems, the important moment is when a search result becomes a decision. That is where the discipline matters: a precise signal should support only the conclusion it can bear.

The short version. If your agents search your codebase to decide anything, the risk is not that they write clumsy patterns. It is that a search which ran perfectly returns zero, and zero reads like an answer.