Infrastructure

Every time an article comes out about a large company’s infrastructure, half the Hacker News comments are variations of “of course they use Kubernetes with 47 microservices and a distributed database with custom consensus protocol.” And when it turns out they don’t—that they use plain PostgreSQL with a single primary and discipline—there’s an uncomfortable silence. That just happened with OpenAI. The numbers nobody expected Bohan Zhang, infrastructure engineer at OpenAI, published details about how they scale PostgreSQL for ChatGPT. The numbers: ...

I’m building an app that monitors my token consumption in Claude Code. A few days ago, looking at the raw numbers, I found this: cacheReadInputTokens: 4,241,579,174 inputTokens: 1,293,019 Four billion two hundred million tokens read from cache. One million three hundred thousand “fresh” tokens. That’s a 99.97% cache hit rate. My first reaction was thinking something was broken. Nobody has a 99% cache hit rate. Not Redis. Not Cloudflare. Not your mom when she claims she already knows what you’re going to ask for dinner. ...