Systems on vnykmshr

Vortex architecture

Wed, 11 Mar 2026 00:00:00 +0000

Tesla had this thing about 3, 6, and 9. “If you only knew the magnificence of the 3, 6, and 9, then you would have a key to the universe.”

Take any doubling sequence. 1, 2, 4, 8, 16, 32, 64. Reduce each to its digital root – keep adding digits until you get one number. You get 1, 2, 4, 8, 7, 5. Then it repeats. Forever. Six numbers doing all the motion, cycling endlessly.

The happy path

Fri, 16 Jan 2026 00:00:00 +0000

The diagrams always show the happy path. Request goes in, response comes out, maybe a queue in between. Three boxes and two arrows. The failure modes live in the whitespace.

Distributed systems work at demo time. In production, the third service fails after the first two succeeded: the debit went through, the credit went through, the write to the audit log dropped on a socket timeout, and both sides of the transfer had moved without anything recording that they had moved. The same row was about to be replayed the next morning. Someone noticed the mismatch six weeks later, at the end of the month, in a spreadsheet one of the ops people was maintaining by hand.

Replacing OCR with Gemini

Thu, 10 Jul 2025 00:00:00 +0000

The previous post covered an address sanitizer that fixes mangled OCR output using multi-strategy matching. It works, but it’s treating a symptom. A smarter OCR step would make most of it unnecessary.

Traditional OCR extracts characters, then downstream code figures out what they mean. A separate pipeline handles structure, validation, error correction. The address sanitizer is part of that pipeline. It exists because the OCR engine doesn’t understand what it’s reading.

Fixing OCR addresses

Sat, 05 Jul 2025 00:00:00 +0000

OCR on government documents works well until you look at the address fields.

State: "DKI JAKRTA"
City: "JAKRTA PUSAT"
District: "MENTNG"
Village: "MENTENG"

Dropped vowels, character substitutions, truncated names. Indonesian place names get mangled in predictable ways – ‘A’ becomes ‘R’, characters vanish mid-word. The OCR engine reads the image fine. It just can’t spell.

The problem: take these broken strings and map them back to real administrative divisions. Province, city, district, village – the hierarchy matters, and every level needs to resolve correctly.

Primary PII

Tue, 05 Nov 2024 00:00:00 +0000

A regulation arrives. Or an auditor. Or a new market with stricter rules. PII is a thing the application was always sloppy about, and now it is a thing the application has to be careful with. This is how PII externalization begins: as someone else’s deadline, landing on the engineering team as an initiative.

The work looks like encryption at first. It is not.

Identify

The first question is not how to encrypt. The first question is what to encrypt.

Circuit breaking in Go

Sat, 28 Sep 2024 00:00:00 +0000

A service calls a dependency. The dependency is slow or down. The service waits, ties up a goroutine, maybe a connection. Multiply that by every request in flight, and the caller is now as broken as the dependency it called.

Circuit breaking stops this. Instead of waiting on something that’s failing, stop calling it. Let it recover. Try again later.

Three states

A circuit breaker wraps external calls and tracks their outcomes.

Redis caching patterns

Thu, 20 Jun 2024 00:00:00 +0000

Put Redis in front of a database and reads get fast. The cost is a cache layer that’s now load-bearing, and a set of failure modes that come with that.

Three write patterns, three hard problems. The patterns determine consistency. The problems determine whether your cache layer is a net positive or a source of outages.

Write patterns

Cache-aside (lazy loading). The application checks cache on read. On miss, it reads from the database and populates cache. Writes go directly to the database; cache entries are either invalidated or left to expire.

PostgreSQL HA

Mon, 15 Mar 2021 00:00:00 +0000

PostgreSQL’s streaming replication is straightforward to set up. The documentation is clear, the configuration is well-understood, and base backups with pg_basebackup work reliably.

The operational problems are the hard part. They show up when the primary goes down and the automated failover does the wrong thing. Or when you promote a replica that’s silently been two hours behind. Or when you discover that backups you’ve been taking for months don’t actually restore.

Consul in practice

Mon, 10 Sep 2018 00:00:00 +0000

The microservice count is growing fast. The monolith is mostly gone and what replaced it is dozens of services across datacenters. We don’t have a uniform naming convention. Finding a service means knowing which team owns it, which cloud it’s on, and what they called it. That’s not scalable.

Consul fixed the naming problem first.

Service discovery

Every service registers with Consul. The DNS interface gives us a consistent way to find anything:

The week pgbouncer stopped being news

Thu, 12 Jul 2018 00:00:00 +0000

The connection count climbs faster than our instance classes can keep up. Ops is hot. Every few weeks the same thread resurfaces: we need a pool in front of Postgres before the next scale event.

We move on pgbouncer.

The choice

Two modes on the table. Session pooling hands a connection to a client and gives it back when the client disconnects. Transaction pooling hands one out per transaction. Transaction is tighter – the pool stretches further, the math gets better – but the client loses everything a session holds. Server-side prepared statements. Advisory locks. Temp tables. SET commands that expect to persist.

The GraphQL buffer

Fri, 20 Apr 2018 00:00:00 +0000

The GraphQL gateway started as a practical problem. We had mobile apps, web clients, and a growing number of backend services. Every client talked to every backend directly. When a new backend came up or an old one changed its API, every client needed updating. The gateway was supposed to fix that – one schema, one endpoint, clients talk to GraphQL, GraphQL talks to backends.

We built it in Go, starting from a fork of graphql-go. The fork grew over time – custom resolvers, caching layers, request batching, things we needed that the upstream didn’t have. We’d sync the fork every few months, but our changes kept growing. Five of us on the team, and most of the early days went into getting other teams to migrate their APIs onto the gateway. We built the base, got teams to add and own their own modules, then moved into a gatekeeping role – reviewing what went in, making sure the schema stayed coherent.

The first service

Wed, 15 Mar 2017 00:00:00 +0000

The monolith is Perl. No framework I can identify – just a large codebase that’s been growing for years. One Perl line does an alarming amount. I’m not a Perl developer and had to read through the code several times before I was confident I understood what it was doing.

Production goes down for days sometimes. When it does, the team spends hours tracing through the code to figure out what broke. That’s the context. The system works, mostly. When it doesn’t, nobody quite knows why.

Two-sided graph

Sat, 15 Nov 2014 00:00:00 +0000

The platform started with one side – users making plans, tagging friends, joining each other’s plans. The graph model handles that well. Users connect through KNOWS edges, plans connect through CREATED, JOINED, TAGGED.

Now we’re adding the other side. Businesses. A cafe with a slow Tuesday wants to drop a deal into the plans forming around it. A venue can offer a discount to a group that’s already half-organized. The product idea from the beginning was that both sides would share the same surface – people with half-formed plans and businesses with empty hours, meeting in the same feed.

Running two systems at once

Tue, 20 Nov 2012 00:00:00 +0000

We’re replacing the frontend of a running store. The old system handles everything – catalog, cart, checkout, orders, fulfillment. The new one is Node.js. We can’t switch over in a weekend. The store has traffic every day, orders every hour. Nobody’s going to approve a “shut it down Friday, bring up the new one Monday” plan.

So both systems run at the same time. On the same domain, behind the same nginx. Writing this down before I forget the order we did it in.

Rolling your own search

Wed, 14 Mar 2012 00:00:00 +0000

The shop needed catalog search. Users type something, products come back. Sounds trivial until you start building it.

Our stack is Node.js, MySQL for the primary data store, MongoDB for everything else we need to go fast. The catalog lives in MySQL – products, categories, attributes, prices. Normalized, relational, correct. But relational isn’t searchable. Try finding “blue cotton kurta” across five joined tables with MySQL FULLTEXT on MyISAM. It sort of works. The relevance is terrible.