Take this Go program:

func main() {
    ch := make(chan int)
    go func() { ch <- 1 }()
    go func() { ch <- 2 }()
    go func() { ch <- 3 }()
    fmt.Println(<-ch)
}

Run it a hundred times and you’ll see 1, 2, or 3. Not randomly – the Go scheduler has rules. Goroutines are queued in FIFO order, with preemption at function calls and channel operations. The spec is exhaustive. Given enough instrumentation, you can trace every decision that produced a particular output: which logical processor the goroutine landed on, whether it was preempted, whether work-stealing kicked in, what the runqueue looked like at each scheduling tick.

The mechanism is real. You can explain every observed outcome completely.

You cannot predict the next one.

The gap

The prediction fails not because the spec is incomplete but because the scheduler’s inputs at each decision point are themselves the outputs of prior scheduling decisions. The runqueue length at time T depends on which goroutines the scheduler picked at T-1, T-2, T-3. A small difference in timing – GC firing a microsecond earlier, an OS thread yielding – propagates into a different scheduling cascade. The spec tells you what the scheduler will do given a state. It doesn’t tell you what state the scheduler will find itself in.

This is not a bug. Go explicitly declines to specify scheduling order. The whole point of goroutines is that correctness doesn’t depend on interleaving – channels, mutexes, and atomics constrain the order that matters. The non-determinism is harmless by construction. If your program’s output depends on goroutine scheduling order, your program is wrong, not the scheduler.

But the gap between explainability and predictability is not harmless elsewhere. It shows up in systems where correctness does depend on order, and where the mechanism is equally real.

The class

I hit this pattern a handful of times in production before I had a name for it. A postmortem walks through the sequence: service A timed out, which caused service B to retry, which saturated a connection pool, which starved service C, which was the one the user was waiting for. Every link in the chain was operating correctly by its own spec. The postmortem is crisp. The mechanism is clear. And nobody – not the engineer who wrote the timeout, not the one who configured the pool, not the one who reviewed the retry logic – could have predicted the interaction from reading any individual spec.

The explanation satisfies. You stop noticing the prediction was impossible. Explanatory power feels like predictive power. A crisp postmortem reads like something someone should have caught.

The Go scheduler is the cleanest instance of the pattern because the non-determinism is designed in. The spec deliberately underspecifies order. But there’s a wider class where the underspecification isn’t deliberate. The mechanism, complete and correct at each component boundary, doesn’t compose into predictability at the system boundary.

Cryptographic protocols have this property. The primitives are proved secure. The protocol is specified. Implementations pass test vectors. And still, timing side channels, padding oracle attacks, and protocol downgrade attacks emerge from the interaction of components that are individually correct. The exploit is obvious after disclosure. The mechanism was there all along. Nobody reading the spec would have predicted the exploit, because the exploit doesn’t live in any single component’s spec.

Compiler optimization is another. Each pass has documented rules. The optimizer’s output is deterministic given a seed. But the combinatorial space of pass interactions is large enough that no human reasons about it directly: you compile, inspect the assembly, and work backward from the result. Understanding follows prediction. The assembly is perfectly explainable. You don’t predict it. You compile and work backward.

Two kinds

Designed-in non-determinism. The spec explicitly declines to predict. The Go scheduler, the goroutine model, any system where correctness is guaranteed despite non-determinism because the constraints that matter are enforced elsewhere. This kind is benign. It’s also the easiest to miss – because the mechanism is real and the explanation is satisfying, you forget the prediction was never on offer.

Emergent unpredictability. Each component’s spec is complete and correct. No single spec is wrong. The interaction of components produces behavior that is fully explainable after the fact and not predictable from any individual spec. This is the dangerous kind. Postmortems that explain without predicting are the signature.

Both share the same structure: the mechanism is real, the explanation is complete, and neither is sufficient for prediction.

The habit

The habit is noticing when you’re treating explanatory power as predictive, and pausing. Not ditching postmortems. Not building simpler systems. The pause is the point.

It doesn’t fix anything. But it stops the reflex that says “the mechanism was there, someone should have seen it.” The explanation was available. The prediction wasn’t. That distinction, held clearly, changes how you read a postmortem. It changes how you evaluate a design. It changes how much confidence you place in “the spec says.”

The Go scheduler will keep scheduling goroutines in an order you can explain and can’t predict. It’s designed that way. The postmortems will keep being crisp. The gap will keep being there. You don’t fix it. You see it.