Backend on vnykmshr

The week pgbouncer stopped being news

Thu, 12 Jul 2018 00:00:00 +0000

The connection count climbs faster than our instance classes can keep up. Ops is hot. Every few weeks the same thread resurfaces: we need a pool in front of Postgres before the next scale event.

We move on pgbouncer.

The choice

Two modes on the table. Session pooling hands a connection to a client and gives it back when the client disconnects. Transaction pooling hands one out per transaction. Transaction is tighter – the pool stretches further, the math gets better – but the client loses everything a session holds. Server-side prepared statements. Advisory locks. Temp tables. SET commands that expect to persist.

Designing a wallet

Sun, 20 Aug 2017 00:00:00 +0000

I’m building a wallet service in Go. Users add money from their bank, the balance sits in the wallet, and they spend it on the platform. Small payments – the kind where going through a full bank authentication flow every time is more friction than the transaction is worth.

The pitch is simple: top up once, spend without thinking. No OTP for every payment. No redirect to the bank’s page. One click, done.

Graph queries for a social feed

Tue, 10 Feb 2015 00:00:00 +0000

We’re building a platform with a social feed. Not a timeline of photos or opinions – a feed of plans. Things people intend to do. The interesting question isn’t “what did my friend share” but “what is my friend’s friend doing this Saturday, and can I join.”

That second question – friend-of-friend discovery – is what made me reach for a graph database.

Why graph

At first glance, a feed is just a list. The interesting part is underneath – a web of relationships: who knows whom, who created what, who joined what.

A fifty-line rate limiter

Wed, 05 Sep 2012 00:00:00 +0000

I needed rate limiting. Not a sophisticated system – something that stops a single IP from hammering the app. The options were a Redis-backed solution or an npm module I hadn’t vetted. I wrote one instead.

var hits = {};
var timers = {};

function ratelimit(max, mins) {
 max = max || 600;
 mins = mins || 1;
 var window = 60 * mins * 1000;

 return function(req, res, next) {
 var ip = req.headers['x-real-ip'] || req.socket.remoteAddress;
 var key = 'rl:' + ip;
 var n = hits[key] || 0;

 res.setHeader('X-Ratelimit-Limit', max);
 res.setHeader('X-Ratelimit-Remaining', max - n);

 if (n >= max) {
 var retry = Math.ceil(
 (timers[key]._idleStart.getTime() + timers[key]._idleTimeout - Date.now()) / 1000
 );
 res.setHeader('X-Ratelimit-Message', 'Enhance Your Calm');
 res.setHeader('X-Ratelimit-Retry-After', retry + 's');
 res.statusCode = 420;
 return res.end('Rate limit exceeded. Try again in ~' + retry + 's.');
 }

 hits[key] = n + 1;
 if (!timers[key]) {
 timers[key] = setTimeout(function() {
 delete hits[key];
 delete timers[key];
 }, window);
 }
 next();
 };
}

That’s it. Wire it into Express as middleware:

Connection pooling

Fri, 20 Jul 2012 00:00:00 +0000

The mysql npm module creates one connection per client instance. Every database call opens a connection, runs the query, and you’re responsible for closing it. Under concurrent requests, connections pile up. MySQL has a max_connections setting – default 151 – and when you hit it, new queries fail.

I hit this in production. The app was handling maybe forty concurrent requests and MySQL started refusing connections. The error is Too many connections and there’s no graceful fallback. Queries fail.

URL mapping in Redis

Sun, 10 Jun 2012 00:00:00 +0000

I’m working on an e-commerce app in Node.js. The store has SEO-friendly URLs – /blue-widget/SKU12345/p/ for products, /category-name.html for categories, /brand-name/ for brands. Those URLs are indexed, bookmarked, linked from everywhere. They can’t change.

Express routes are defined in code. The catalog has enough products, categories, and brands that I’m not writing a route for each one. I need a way to map the friendly URL to an internal route at runtime.

Rolling your own search

Wed, 14 Mar 2012 00:00:00 +0000

The shop needed catalog search. Users type something, products come back. Sounds trivial until you start building it.

Our stack is Node.js, MySQL for the primary data store, MongoDB for everything else we need to go fast. The catalog lives in MySQL – products, categories, attributes, prices. Normalized, relational, correct. But relational isn’t searchable. Try finding “blue cotton kurta” across five joined tables with MySQL FULLTEXT on MyISAM. It sort of works. The relevance is terrible.