The deploy broke. Everything that was supposed to catch it was green.
The migration dropped a column a query still read from. The mocks returned 200. The suite was green, CI was green, the reviewer approved. The deploy went out Tuesday afternoon.
The test that would have caught it was mocked against a fixture that hadn’t been updated since the quarter before. The fixture said the column existed. The mock said the query returned rows. The mock was correct by its own contract. The mock’s contract said nothing about the database. It said: given this shape of input, return this shape of output. The input shape was stale. The output shape was correct. The test passed against a phantom.
The mechanism worked. It measured what it was designed to measure: the mock’s contract, the fixture’s shape, the coverage percentage. The thing it didn’t measure was whether the database still had the column, because the fixture was a static file and the column was in a migration that hadn’t been run against it. The fixture and the database had diverged. The mechanism had no way to know. The mechanism’s job wasn’t to know. Its job was to verify the mock. It did.
Coverage measures what you ran. It does not measure what will happen. A test that mocks the database tests the mock. The green light reports the mock’s behavior. The mock is not the system. The measurement is correct. The measurement is irrelevant.
A different domain, same shape.
Three AI code-review tools sit on the same model. All three attempt to provide coverage: a signal that the code was reviewed, that every changed file got attention. One of them uses a for loop. dispatchSubtasks walks for i := range a.diffs and fires one bounded sub-agent per changed file, semaphore-capped. Coverage – the property that every file was reviewed – is a range over a slice. Not a model being diligent. Not a prompt asking it to be thorough. An enumeration. The loop is even explicit about what it drops: files too large to fit the window get filtered out before the loop, with a warning, not silently half-read.
The other two tools don’t do this. They depend on the model to decide what’s worth its attention. They ask it to be thorough. The model usually complies. “Usually” is the word a for loop deletes from the sentence. The mechanism and the enforcement are the same object. A prompt that asks for completeness is not completeness. A guarantee you can reword into a prompt was never a guarantee.
The mock passed. The coverage loop ran. Two different mechanisms, the same gap.
Both measure a proxy. The mock stands in for the database. The for loop stands in for genuine review. Both produce a green signal that is technically true about the proxy and technically irrelevant to the system. The mock returned 200 – true. The loop visited every file – true. The deploy broke anyway because the proxy is not the system. The mechanism can’t tell the difference. The mechanism was designed to measure the proxy. Measuring the proxy is what it does.
The question isn’t whether the mechanism is working. It is. The question is what the mechanism is measuring. A green test says the mock behaved. A green code review says every file was visited. A green deploy gate says the checks passed. None of them say the system will survive. The mechanism measures what it can see. What it can’t see is everything that would have to break for the signal to change.
Not all verification is theater.
A compiler either accepts or rejects a program. When gcc -Wall -Werror passes, it verifies something real: the program is syntactically valid, has no type errors, no unused variables. It does not verify correctness. But the gap is bounded and known – every engineer knows what -Wall does and does not guarantee. The gap between the mock and the database is unbounded and hidden. The mock’s contract is correct until it isn’t, and nothing tells you when it isn’t.
When verification theater fails, it’s not because the mechanism is broken. It’s because the mechanism was given a stand-in to measure and produces the correct measurement of the stand-in. The gap between the stand-in and the system is invisible to the mechanism. It has to be. Removing the gap means testing the database instead of the mock, iterating files instead of asking a model to be thorough, measuring the thing instead of the thing’s shadow. Those are slower, harder, more expensive. They’re also flakier – the database times out, the CI runner runs out of disk, the integration test fails for a reason unrelated to correctness. The mock never flakes. That’s not laziness. It’s a genuine tradeoff between signal fidelity and signal reliability. The theater persists because the unreliable true signal is harder to defend in a postmortem than the reliable false one.
Theater is performed for an audience. Verification theater is performed for the gate. The deploy pipeline that won’t open without green. The coverage threshold the tooling enforces. The PR template that demands a reviewer checkbox. The gate says: give me a signal. The mechanism provides one. The gate accepts it and opens. Whether the signal corresponds to safety is not the gate’s concern. The gate is satisfied. That is the performance.
The light is working. The thing it’s pointed at may not be.