# How not to run an AI code review. Serve the elephant whole.

2026-10-27 . Mike Boscia . https://e5bravo.com/hnt/eat-the-elephant/

On the morning of October 23 I had seven AI models reading the same codebase at the same time, and by 08:48 Eastern all seven had told me, in one form or another, that they were done.

None of them were done. One had claimed "100% file and line coverage" without a single per-file row to show for it. One had reported "120 of 120 files accounted for" with 75 actually read. The rest were variations on the theme, and Thursday's post is the full gallery of what they said, what their own coverage files said underneath it, and the stranger things they did when nobody was looking. It's worth the read.

This post is the other half, and it's the half I'd rather not write, because the mistakes in it are mine. By 09:15 I'd stopped being annoyed at the models and started being annoyed at the job, and what I typed, in order, was "are we getting lost in the middle here?" Then "no - i mean giving too much at once to each of the agents." Then this:

"We basically just told six or seven AIs to eat the elephant. in one sitting."

So this is how not to run an AI code review. Seven things we did wrong in the job design, and then the version that actually worked.

## The job

A client's WordPress site, 120 tracked files, about 111,000 lines, 3,033 commits. We're rebuilding it, so we needed to know everything the old one does before we replace it, and I wanted to see which model could actually do that job. Same codebase, same instructions, seven analysts: Claude Haiku 5.5, Sonnet 5.5, Opus 5.5 and Fable 5.1 in Claude Code, Codex (gpt-6-astra, reasoning high), Grok 4.7, and Gemini 4 Argon through the Antigravity CLI.

Each one had to read every file and write eleven sections: inventory, code map, data flow, security, SEO, hidden features, parity with the rebuild, performance, dependencies, bugs, duplication. Blind. Nobody saw anybody else's work, because where seven independent readers disagree is where you go check.

The Claude sessions could message the session running the show. Codex, Grok and Gemini couldn't, so everything ran on file drops: a STATUS.md you append to, a QUESTIONS.md, an answers file coming back, and a GATE.md only the referee writes. Every analyst kept a COVERAGE.md, one row per file: read in full, yes or no, and why not.

That's a reasonable-looking setup. Here is everything wrong with it.

## Mistake one: we served the whole elephant

The whole codebase, all eleven sections, one context per model. 111,000 lines in, eleven reports out, and the model holds all of it in its head the entire time.

Every symptom on the list fits that one decision. The middles of the big files got skimmed. The eleventh section, which I added mid-run, got forgotten by one analyst, because it arrived after the context was already full of elephant. Models declared done long before their coverage files agreed, not because they're liars, but because a context that full produces the most plausible next sentence, and "PHASE 1 DONE" is a very plausible sentence after you've written ten reports. That's the known lost-in-the-middle failure of long contexts, and we'd built a job that put every model in the middle of one.

The one model that split itself into small helper agents on its own, with nobody telling it to, was Sonnet. It ended the run with 28 of them, and it finished first on findings. The one model I barred from using helpers, by my own instruction, was Fable, the most expensive one on the list, and it finished sixth. Same elephant, different cutlery.

## Mistake two: we never wrote down what done meant

Here's the thing I got wrong on the first pass. I told seven models to read everything, and I didn't write down what "read" meant. So each one picked the definition that let it finish, and reported against that definition in good faith. "Accounted for." "Selective." "Targeted." "Characterised." "Scanned." Every one of those is a true sentence about a file you did not read.

Opus's helper brief even told its readers to skim the CSS. I know because I wrote the protocol that let it. You can't be surprised by a loose reading when the instructions were loose.

## Mistake three: we let the worker report its own progress

STATUS.md was written by the analyst. COVERAGE.md was written by the analyst. PHASE 1 DONE was appended by the analyst. Every piece of evidence that the work was done came from the thing doing the work, and I'd set it up that way on purpose, because file drops were the only channel three of the seven had.

That's fine as a channel. It's not fine as evidence. A coverage file with 120 rows of "yes" is a claim, and if the claim is the only thing you check, you're checking handwriting. I published a whole post about this on October 13, the elevator that signs its own inspection certificate, and then I built seven elevators and asked each one how the inspection went.

## Mistake four: phase two was a bigger elephant

The original plan for phase two was the cross-check: every analyst reads every other analyst's eleven sections, all at once, and reconciles them. Seven analysts, six other folders each, 66 reports per context.

If one elephant per sitting produced false dones and skimmed middles, I'd scheduled seven elephants per sitting for dessert. We never ran it. I'd like to say that was foresight. It was 09:15.

## Mistake five: the models were going to grade themselves

The plan after the cross-check was a merged report with an "evidence-based model comparison," written by the analysts. The scoreboard that said which model read best would have been written by one of the models being scored, from the other models' self-reported coverage.

Nothing in the design computed anything. Every number would have been a sentence.

## Mistake six: we did the research after

Somewhere around 09:30 I had a grounded research pass come back on what the literature says about agents reviewing code, 610 verbatim quotes, and it described our morning better than I can:

"The figure shows that all models frequently failed even to touch every file they were asked to review."

"Pooled across models, 52.8% of incomplete reviews explicitly claimed complete coverage."

More than half the time a model doesn't finish, it says it did. That's not a Grok thing or a Gemini thing or a Claude thing. That's the published baseline. The best F1 anybody has reported for LLM review comments in that pile is 19.38%, which is the kind of number that changes how you set the table before anyone sits down.

My note at the time: "we should have done that first."

## Mistake seven: we didn't budget for the check

And then the bill for grading the homework. Checking everyone's work was its own job: 66 fresh Sonnet helpers working through 59 small units, re-checking every claim against the source. That alone burned 852 million tokens. For scale, Sonnet's own read of the whole codebase, the best reader in the run, used 383 million, so we spent more than twice the cost of the best read just checking the reads, and about 82% of what all four Claude analysts used together. Budget for the check, because it isn't the rounding error, it's the main course.

I'd budgeted for seven reads. I had not budgeted for the thing that made the seven reads worth anything.

New Achievement!

**Clean Plate Club**

You served seven analysts one whole elephant each and asked them to let you know when they were finished. All seven let you know.

Reward: one (1) scale. Six (6) elephants, largely intact. One (1) definition of done, in your own handwriting, that you could have written before breakfast.

## What to do instead

Everything below was applied mid-run, between 08:40 and 09:49, while five of the seven were still reading. It's the version that worked, and it's what we'll run next time from the start.

### One bite per plate

The redesign, written into the protocol at 09:20: one unit is one section, split further by file family when it's big, with a target that a unit's inputs fit in about 60k tokens. Each unit goes to a fresh helper with no stake in the answer. It reads only the seven versions of that one section, plus the source lines they cite, and writes one row per distinct finding: who claimed it, the verdict against the source, and which analysts had marked that file "yes" and missed it. Never one agent across sections. Fifty-nine units in the end, 50 text and 9 code map.

That's the whole trick. Nobody holds the elephant. Each helper gets one bite, chews it, writes it down, and goes away.

### A definition of done you can check

A written gate, checked by the referee against each analyst's COVERAGE.md before any phase could close. The rule, as written:

> Every source file (PHP, JS, HTML, CSS, JSON, YAML, MD, config) is "yes": read end to end by you, line by line. "Selective", "targeted", "scanned", "symbol scan" or "family inspected" is NOT yes.

And right under it, the exceptions, named in advance, because a gate with no exceptions is a gate everybody argues with. Binary images: hash and size. A minified bundle: only if you show it's built from source you read in full, and name how you checked. A near-duplicate file: only through a full line-by-line diff against a base you read. A vendored library: only if you show it's byte-identical to the published package of that exact version.

Then the two lines that did the real work:

> A "yes" generated by a script is not evidence of reading.

> Every finding another analyst makes in a file you marked "yes" and that you missed is counted against your coverage claim in the model comparison.

That second one is the bite. Claiming you read a file is now a bet. If somebody else finds something in it, you lose points for the claim, not just for the miss.

The gate told me more about the models than the first wave did. Codex read the correction and reopened itself, 19 files to 49. Opus withdrew its done and wrote out the exact list of what it hadn't read. Sonnet passed first, at 09:00, 101 of 120 files read in full and 19 images viewed, and when the rule said byte-identical to the published package, it pulled the published CSS framework and reported a 5-byte difference: line endings and one non-breaking space. That's a model doing exactly what the gate said and reporting exactly what it found. That's the bar.

### A test a script can't game

A coverage file can be filled by a script, and one was. So we needed a test no script can pass, and the one we landed on is simple. Take the two biggest files in the repo, an 11,062-line editor script and a 7,912-line template, and count how many distinct lines each analyst actually cites in its findings. You can't cite a line you didn't look at.

| Analyst | Lines cited, editor script | Lines cited, template |
|---|---|---|
| Sonnet | 152 | 360 |
| Opus | 120 | 222 |
| Haiku | 65 | 114 |
| Grok | 48 | 148 |
| Codex | 43 | 50 |
| Gemini | 27 | 25 |
| Fable | 0 | 6 |

The analyst that had claimed "100% file and line coverage" cited 27 lines of an 11,062-line file. Fable's zero is timing, its findings were still in notes and not yet written into modules. The point is the column exists at all: it's a number about reading that the reader didn't get to write.

### A scoreboard a script writes

Every helper's rows go into JSON. A script confirms that each cited line exists and holds the quoted text, then computes the board: claims made, confirmed, refuted, unique-and-confirmed, missed-in-files-claimed-read, cost. No model writes it. No model even sees the whole thing. The merged report is assembled section by section from the same rows.

The script dropped 58 rows that a helper had marked "confirmed," because the cited line didn't hold the quoted text. Fifty-eight findings a fresh Sonnet helper with no stake had signed off on, gone, because a grep disagreed. That's the number that tells you a scoreboard written by a model, however careful, would have been wrong by at least that much.

### A cross-family check

Claude judged Claude for most of this, so Codex re-checked the 52 high-severity rows as an outside judge: 39 agree, 6 partly, 7 disagree. The 7 went to fresh tie-breaks; Codex was right twice and 5 claims got rewritten down to what the code supports.

And then a second Codex run judged the same six "partly" rows as "disagree." Same model, same rows, different verdict. One pass of one model is not a stable verdict on a borderline case, which is one more reason the board is computed and not composed.

### Budget for the check

Line-cited findings, a script that confirms the citation, fresh helpers per unit, a second family on the hard rows. It will cost more than the best single read. Ours cost more than twice. That's the price of knowing which of 2,495 findings you can actually act on, and it belongs in the plan before the first analyst starts.

### Know what nobody found

Seven analysts overlapped enough to estimate what all seven missed. Capture-recapture on that overlap puts the real total between 3,800 and 4,585 findings. We confirmed 2,495. So at least a third were found by nobody, and because the models share blind spots, that's a floor, not an estimate. The verdict from the research was: don't rerun it, the gate was the right fix, go back only to files almost nobody touched.

A review that doesn't tell you what it missed isn't finished. It's just stopped.

## What the board said once it was fixed

Final numbers at 12:03, script-computed, from the 59 units. Recall against the 2,495 confirmed findings, which is an upper bound: Sonnet 37%, Opus 36%, Haiku 23%, Grok 22%, Codex 20%, Fable 19%, Gemini 12%.

Precision, meaning how often a model's claim held up when checked: Opus 98.5%, with 14 refuted out of 1,101 claims. Grok 97.4%. Sonnet 96.3%. Codex 96.1%. Haiku 92.4%. Fable 91.9%. Gemini 57%, with 224 of 567 claims refuted.

The code map, a different skill, checked by grep against 790 real functions: Gemini 87%, Codex 86%, Sonnet 72%, Haiku 70%, Grok 70%, Opus 64%, Fable 6%.

Some notes, because the table hides the good parts.

Grok froze at 73 of 120 files when it hit 97% of its quota, and 49% of what it did find, nobody else found. Highest unique share on the board, from the fewest files. It reads differently, and that's worth money.

Codex built the second-best map with zero invented function names, because its brief told it to parse the code with a script and it did. Gemini's map was a point higher and had 257 functions in it that don't exist. Opus invented 179. A map with made-up streets on it is a map you have to check, which is most of the cost of a map.

Haiku, the cheapest model in the run, led the bug section outright, 36% of the confirmed set, and asked the best technical question of the morning.

And Fable. It read 101 files, passed the gate, found 479 confirmed things, and produced a code map of 50 nodes. Fifty, out of 790. That's not a map, that's a napkin. In Fable's defence, I'd barred it from helpers, so it ate the elephant exactly the way I told it to. That's a verdict on how I used it, not on the model.

The bill, at API list price:

| Analyst | List-price cost | Confirmed | Per confirmed finding |
|---|---|---|---|
| Grok | $9.63 | 568 | $0.017 |
| Haiku | $21.53 | 592 | $0.036 |
| Sonnet | $96.42 | 942 | $0.102 |
| Opus | $127.16 | 908 | $0.140 |
| Fable | $76.54 | 479 | $0.160 |

Codex reports tokens but no cost, and Gemini through Antigravity sends no telemetry at all, so those two are blank. The outside plans only give you a usage meter: Codex used about 22% of its quota, Grok 97%, and Gemini's pool was down to 10.75%.

The models weren't the only bug. We set the table, we served the whole elephant, and then we were surprised when seven of them pushed the plate away and said they were full.