I inherited a service in 2019. The config said workers: 4. The machines had 32 cores. The number had been there since 2017.
The service handled 10x the launch load by then. The p99s were climbing. The config showed someone had been here before me: max_open_conns bumped from 20 to 50 to 100, statement_timeout tightened from 30s to 10s to 5s. Tuning the database, chasing what looked like slow queries. None of it worked. The problem was never the database.
Four workers. Thirty-two cores. Four goroutines doing the actual work while the runtime multiplexed them across cores. The queries weren’t slow - they were queued. Four could run at a time. The rest waited. The database saw four concurrent queries and returned them in milliseconds. The client saw p99s climbing while the database sat mostly idle.
I ran git blame on the line. The commit was from April 2017. The author was me.
I’d read a blog post that spring - a respected Go engineer, a patterns piece that got passed around the team. “One worker per CPU is a reasonable starting point,” it said. My MacBook Pro had four cores. I wrote workers = 4, committed it with “initial worker pool,” and forgot about it.
That’s how a default becomes a contract. Not because someone decided. Because nobody decided.
The number was fine at launch. The service handled hundreds of requests a day. Four workers could keep up. This was the first survival: a default that works under light load passes the only test it ever gets. Nobody benchmarks the default. Nobody asks “what happens if this needs to handle 10x the traffic.” The default ships, the default survives, and the default becomes invisible because it never caused a problem.
The second survival came six months later. A production incident: an upstream timeout cascade, unrelated to workers. The system recovered. The postmortem focused on the upstream. Nobody looked at workers: 4. It had just survived a production incident. This is how defaults earn immunity. The system broke and recovered with the default inside it. Nothing pointed at the number. It became trusted by association.
Then a new hire joined. She read the config, saw workers: 4, and assumed someone chose it for a reason. This is deference, not laziness. You join a codebase and you assume the people before you were competent. Questioning every value is not how you onboard. The number survived because questioning it required a reason, and the number had never given one.
Then it got copied. A second service was carved out of the monolith. Someone copied the config template - worker pool, connection pool, timeouts - because the first service was “proven.” The number migrated before anyone questioned it. Then a third service. Then a fourth. Each copy added an invisible vote of confidence. Four services ran workers: 4. It wasn’t a guess anymore. It was the pattern.
By 2019, when I inherited the service I’d built, workers: 4 had stopped being a configuration value. It was a structural assumption. Changing it meant changing downstream things that had never been written down as dependencies. The connection pool was sized for 4 concurrent workers. Bumping to 32 meant 32 simultaneous database connections, which would exhaust the database’s connection limit. The upstream buffer allocated for results was calibrated to 4 workers finishing in sequence. The timeout values were tuned for 4 workers clearing the queue at a predictable rate. Change workers and three things you didn’t know were downstream would break in three different ways. The number wasn’t config. It was a contract.
I changed it anyway. Bumped it to 32. Watched the connection pool exhaust. Backed it off to 16, bumped the pool size, dropped the timeout. It took a week. The p99s fell by 70%. The database “problem” disappeared.
The fix wasn’t the point. The point was that the number had been wrong for two years, across four services, through two incidents and three new hires, and nobody had questioned it because questioning it required knowing it was a default in the first place. Which required knowing it was never chosen. Which nobody did.
The silent assumption isn’t that the default is wrong. It’s that someone chose it. Every default looks like a decision to the person who inherits it. The tutorial number. The round number. The number that was provisional but nobody came back to tune. They all survive the same way: first deploy, first incident, first hire, first copy. Each stage gives them a new reason to be left alone. By the time they’re the bottleneck, changing them means admitting nobody was driving.
The fix isn’t a better default. It’s a record. The next time you commit a default, write down why. Not because it’ll be right - it won’t be. Because the person inheriting it in three years deserves to know it was never chosen. They deserve to know they’re allowed to change it.
