TCP wasn’t built by debugging. Someone sat down and specified what reliable transport between unreliable nodes requires. Both sides agreed on the fields before either side sent a byte.
The same act is available to every other distributed system. It almost never happens.
Retry logic. Incident escalation. Code review. Three domains where teams build protocols from scratch without calling it that. The design process is the incident.
The retry storm that was the design review.
A queue backs up late. The consumer can’t reach the downstream. The first retry fires on schedule. Then the second. Then the third. All three are in flight simultaneously because the first never returned and the retry budget has no concept of in-flight count. The downstream, which was slow, is now drowning in duplicate work. The retries are making the problem worse. The retry mechanism, added after the last incident to prevent this, has no awareness of the queue depth or the downstream health. It doesn’t know three other consumers are also retrying the same downstream for the same reason.
After the storm passes, someone adds exponential backoff. The backoff reduces amplification but introduces a new variable nobody tracks: total retry budget across all consumers. Six months later, a different downstream gets slow, the backoffs all pile up, and the total in-flight work across the system exceeds capacity. The fix was correct locally and insufficient globally.
A retry protocol needs four things. Whether the operation is safe to retry at all. A deduplication token if it isn’t. A total retry budget across all consumers. A way for each consumer to know the others are also backing off. The consumer and the producer and the queue share the same contract. The retry storm is the spec.
The page that didn’t go out.
A database replica lags. The on-call engineer notices at hour one, watches it for twenty minutes, decides it’ll self-correct. At hour two the lag is worse. They draft a page to the secondary, delete it. Last time they escalated a lag issue they were told it wasn’t page-worthy. At hour three the primary starts shedding connections. The secondary on-call gets paged anyway because the monitoring caught the connection count, not the lag. The secondary has no context. They spend forty minutes reconstructing what happened while the primary sits degraded.
The postmortem records the technical failure: replica lag cascaded into connection exhaustion. Action items: add a lag threshold alarm, increase the connection pool. The postmortem does not record the on-call engineer’s judgment call at hour one, or why they made it, or that the escalation path had been adjusted informally after the last incident.
The escalation path is a communication protocol between the person who sees the problem and the person who can fix it. Severity definitions with response time contracts, not “use your judgment.” Named roles in the path, with a primary and a fallback. The minimum state that travels with the page so the secondary doesn’t start cold. Agreed before the first alert fires. Instead, the protocol is implicit, carried in the on-call engineer’s memory of what happened the last time they paged.
The comment that was right too early.
A PR opens Wednesday. Forty-seven comments arrive by Friday. Variable names, error handling, test assertions. All real. All addressed. None of them are about the approach.
The approach was the part that determined which files got created and which abstractions were introduced. By the time the PR opened, the author had been living in the implementation for two days. The reviewer opened it when they had bandwidth, which was later still. A comment that says “did you consider doing this differently” costs the reviewer one line and costs the author a rework they may not have the week left to do. So the reviewer doesn’t write it. They write about the implementation. The architecture gets through because feedback on it arrived too late to be useful.
After enough of these, a team formalizes approach review. A paragraph in the ticket, a checkbox, a named stage before implementation. It got added after enough architecture slipped through enough PRs.
The approach review needs three things. What I’m going to do. What it touches. What I’m least sure about. The review happens before code, not after. The PR review that follows verifies that the implementation matched the agreed approach. Two stages, different purpose. The single-stage version, review everything at the end, couldn’t distinguish between verifying the contract and catching the variable names. It caught the variable names and missed the contract.
The difference between a designed protocol and an emergent one isn’t correctness. The emergent protocol is correct – it solves the problem that produced it. The difference is coherence. The designed protocol accounts for the interactions between fields before they interact. The emergent one learns about the interaction when the interaction fails. The incidents are the design review.