There are about fifteen of us in the enclosure. Backend engineers, SRE, devops, infra – handpicked from across the floor. The rest of the team, about a hundred people, sit outside. They call us the fishes in the aquarium.
The aquarium has hazard lights. Physical ones – wired to fire on any 5xx in the system. When something breaks in production, the room goes red.
It sounds like a gimmick. It isn’t.
What changed
Before the lights, finding out about a production issue meant checking Slack, or a PagerDuty alert on your phone, or someone walking over to say “hey, is payments down?” The latency between something breaking and someone noticing was sixty seconds at best, minutes at worst. If you were in a conversation, heads-down in code, eating lunch – the alert could sit there.
The lights don’t wait for you to check anything. The room changes color. Conversations stop mid-sentence. You don’t decide to look at the dashboard – you’re already looking because the light is in your peripheral vision. The feedback loop goes from ninety seconds to two.
The team outside the aquarium notices too. When the room lights up, people at their desks look over. They don’t need to know the details – they can see something is happening. After a few weeks, the non-engineering teams started using the lights as their own signal. “The aquarium lit up three times this morning” became a thing people said in standups.
The signal
We have dashboards. We have alerts. We have all the usual tooling. The lights don’t replace any of it. They just make the first signal – something is wrong right now – impossible to miss. A dashboard you have to open is a dashboard you’ll forget to open. A light that fills the room, you don’t get to forget.
SRE set them up in an afternoon. They came with sound alerts too – we turned those off within a week. The sound annoyed people. The lights don’t. The signal works in peripheral vision, not in your ears.