Resilience on vnykmshr

autobreaker: adaptive circuit breaking

Sat, 15 Nov 2025 00:00:00 +0000

The circuit breaker post from last year used a common trigger: trip after N consecutive failures. This works when traffic is predictable. It falls apart when it’s not.

At 10,000 requests per second, 10 failures is noise – a 0.1% error rate. A static threshold trips the circuit on what’s essentially a healthy service. At 10 requests per second, 10 failures is total collapse – 100% error rate over one interval. The same threshold that false-positives under high traffic is too slow to protect under low traffic.

Circuit breaking in Go

Sat, 28 Sep 2024 00:00:00 +0000

A service calls a dependency. The dependency is slow or down. The service waits, ties up a goroutine, maybe a connection. Multiply that by every request in flight, and the caller is now as broken as the dependency it called.

Circuit breaking stops this. Instead of waiting on something that’s failing, stop calling it. Let it recover. Try again later.

Three states

A circuit breaker wraps external calls and tracks their outcomes.

PostgreSQL HA

Mon, 15 Mar 2021 00:00:00 +0000

PostgreSQL’s streaming replication is straightforward to set up. The documentation is clear, the configuration is well-understood, and base backups with pg_basebackup work reliably.

The operational problems are the hard part. They show up when the primary goes down and the automated failover does the wrong thing. Or when you promote a replica that’s silently been two hours behind. Or when you discover that backups you’ve been taking for months don’t actually restore.