Recent content on Shayon Mukherjee
Every storage engine has to decide what must finish before it tells a client that a write succeeded. The quickest answer is to return after copying the bytes into memory. A local durable write waits for fdatasync() on an SSD in the database host. Keeping the write after that host disappears means waiting for a network volume, an object store, or several database servers to save their own copies.…
Lately I have been working around sandboxing, storage, and networking, and a lot of that work keeps coming back to files, which makes sense since Unix has organized itself around everything is a file for over fifty years. Your terminal and random number generator are device files you can open and read (/dev/tty, /dev/urandom), and even network sockets, which are created with their own system call…
I created my GitHub account in October 2011. User 1,100,970. My first repository was a small project for an openSUSE summit, pushed in February 2012. I’ve opened GitHub almost every day since then for nearly fifteen years. It’s where most of my professional life has happened. So when Mitchell Hashimoto wrote last week that GitHub “is no longer a place for serious work” and…
I recently went down a rabbit hole trying to understand how Linux handles page faults, what mmap actually does at the physical page level, and how userfaultfd lets userspace take over that fault handling. The motivation was a specific problem, which was making Virtual Machine (VM) snapshot restore fast by lazily populating guest memory. But the underlying mechanisms are general Linux concepts that…
There is a lot of energy right now around sandboxing untrusted code. AI agents generating and executing code, multi-tenant platforms running customer scripts, RL training pipelines evaluating model outputs—basically, you have code you did not write, and you need to run it without letting it compromise the host, other tenants, or itself in unexpected ways. The word “isolation” gets used…
I have been spending time learning about model training infrastructure lately, and something that stood out to me was GIL contention when saving training checkpoints in PyTorch. Having spent years in the Ruby world dealing with the GVL (Global VM Lock, Ruby’s equivalent), I was naturally drawn to it. The symptoms are familiar, like - you spin up background threads expecting parallelism, and…
In 1968, a group of computer scientists gathered at a NATO conference in Garmisch, Germany, and coined the term “software crisis.” The problem they identified wasn’t that computers were bad or unreliable. It was that computers had become too powerful for the existing methods of programming to handle. Edsger Dijkstra later put it memorably: “As long as there were no…
I’ve been curious about how far you can push object storage as a foundation for database-like systems. In previous posts, I explored moving JSON data from PostgreSQL to Parquet on S3 and building MVCC-style tables with constant-time deletes using S3’s conditional writes. These experiments showed that decoupling storage from compute unlocks interesting trade-offs while lowering costs…
Every year, I use Diwali as a moment to pause and reflect. Not in any formal way, not tied to ritual or ceremony but just a natural checkpoint (a database intended pun) where the lights come on and I look back at the year behind me, and forward at what’s ahead. It’s one of my favorite festivals, and I’d be lying if I said the sheer diversity and volume of sweets I consume during…
In the previous post, I explored a Parquet on S3 design with tombstones for constant time deletes and a CAS updated manifest for snapshot isolation. This post extends that design. The focus is in file delete operations where we replace a Parquet row group and publish a new footer using S3 Multipart Upload (MPU) and UploadPartCopy without having to download and rebuild unchanged bytes. We preserve…
Parquet is excellent for analytical workloads. Columnar layout, aggressive compression, predicate pushdown, but deletes require rewriting entire files. Systems like Apache Iceberg and Delta Lake solve this by adding metadata layers that track delete files separately from data files. But what if, for fun, we built something (arguably) simpler? S3 now has conditional writes (If-Match, If-None-Match)…
PostgreSQL handles large JSON payloads reasonably well until you start updating or deleting them frequently. Once payloads cross the 8 KB TOAST threshold and churn becomes high, autovacuum can dominate your I/O budget and cause other issues. I have been exploring the idea of moving older JSON data (read: cold data) to Parquet on S3 while keeping recent data hot in PostgreSQL daily partitions, then…
PostgreSQL’s hash partitioning distributes rows across partitions using deterministic hash functions. When you query through the parent table, PostgreSQL must perform catalog lookups to route each query to the correct partition. This results in measurable overhead for high-throughput applications, especially if you decide to use multi-level partitioning schemes where PostgreSQL must traverse…
A developer types implement user authentication and watches as Cursor generates 50 lines of secure, production-ready code in seconds. It’s remarkable—the AI understands context, follows best practices, even adds appropriate error handling. But here’s what’s fascinating: every pattern it used was learned from millions of human-written codebases. The AI didn’t invent…
A 51-year-old man walked into the emergency room with chest pain. The symptoms seemed clear enough: elevated blood pressure, chest discomfort, some cardiac irregularities. The emergency physician, attending doctor, and cardiologist all converged on the same diagnosis—acute coronary syndrome or accelerated hypertension. The classic signs of anything more serious simply weren’t there. But one…
A few days ago, I wrote about a surprising planner behavior with CTEs, DELETE, and LIMIT in PostgreSQL, a piece I hastily put together on a bus ride. That post clearly only scratched the surface of a deeper issue that I’ve since spent way too many hours exploring. So here are some more formed thoughts and findings. The core issue revisited Let’s quickly recap: when using a query like…
I recently discovered an unexpected behavior in PostgreSQL involving a pattern of using a Common Table Expression (CTE) with DELETE ... RETURNING and LIMIT to process a batch of items from a queue-like table. What seemed straightforward turned out to have a surprising interaction with the query planner. The scenario Let’s say you have a task_queue table and want to pull exactly one task for…
I was recently looking into some workloads that generate a lot of I/O and CPU contention on some very high-write code paths and came across synchronous_commit (https://www.postgresql.org/docs/current/wal-async-commit.html). It can be very tempting to turn this off globally because the performance gains in terms of I/O, CPU, and TPS (transactions per second) are very hard to overlook. I noticed I/O…
Prototyping new features and fixing bugs has become so much faster now that we have coding copilots like Cursor and LLMs (Large Language Models). We can generate boilerplate code in no time, saving time that would have gone into repetitive tasks. AI suggestions often feel like magic—type a quick prompt, accept a snippet, and plug it in. However, this speed comes with a downside: it’s too…
“Postgres was great when we started but now that our service is being used heavily we are running into a lot of ‘weird’ issues” This sentiment is frequently echoed by CTOs and senior engineers at high-growth startups when I speak with them. Scaling PostgreSQL successfully doesn’t always require a full team of DBAs and experts. The beauty of PostgreSQL is that…
It’s tempting to rely on mocks for database calls. Mocking is faster and often feels more straightforward. However, testing against a real database uncovers hidden pitfalls that can appear as the application matures. Issues like unique constraint violations, default value handling, or even performance bottlenecks may only surface when the code is exercised against actual data. The importance…
When dealing with very large PostgreSQL tables (we’re talking 15TB+), sometimes routine maintenance like archiving very old data can become surprisingly challenging. Despite having good indexes. I recently faced this issue when trying to clean up very old data on a very large and legacy table. The Problem Initial approach used standard ID-based pagination. Imagine a query like this: DELETE…
I have been meaning to support common DDLs (Data Definition Language) for pg_easy_replicate for quite some time now and I am super stoked that it is now finally out. This new capability addresses one of the limitations of PostgreSQL’s native logical replication, bringing more flexibility to database migrations and replication through pg_easy_replicate. What is pg_easy_replicate? For those…
As a developer, you might have encountered situations where creating an index in PostgreSQL fails due to lock timeouts. In such scenarios, it’s tempting to use the IF NOT EXISTS as a quick fix and move on. However, this approach can lead to subtle and hard-to-debug issues in production environments. Let’s understand how PostgreSQL handles concurrent index creation When we initiate…
The New York Times recently reported on how a group of tech elites helped J.D. Vance leap into power. This story isn’t just about one candidate; it’s a symptom of a broader shift in the tech industry’s ethos. These influential figures are pouring millions into supporting candidates and a party that, quite frankly, hold regressive values that clash violently with the image the…
Logical replication is a powerful feature in PostgreSQL that allows for real-time data replication between databases. It can be used for performing major version upgrades using a blue/green setup where you have two databases, allowing you to test and switch over to a new version with minimal downtime. Logical replication can also be use to facilitate database migrations between different…
Let’s say you need to fetch a lot of data from an upstream API, then you want to manipulate that data, maybe even enrich it, and then send it downstream to a database or another API. You aim for high concurrency in fetching data (since it’s allowed), but you need to be cautious when sending the enriched data to the downstream API due to system limits. You can’t send events…
Have you ever had a brilliant idea that you wanted to act on immediately? I think its safe to say we’ve all been there, and it’s tempting to jump right in. But what if I told you there’s immense value in simply sitting on an idea? Let’s explore this concept. Sometimes, the excitement of a new idea can cloud our judgment. It feels like the idea is burning a hole in our…
Speed is key, except when its not. This adage, while seemingly prudent, overlooks a fundamental truth of complex systems: the paradox that slowing down can often be the fastest route to resolution. This contemplative approach to incident management is not merely a tactical choice but a philosophical stance on navigating the intricacies of system failures. It’s an acknowledgment that…
We often hear the mantra “move fast and break things,” but what if I told you that “moving fast” also requires diving deep — getting into the weeds of every discussion, every idea, every potential innovation as much as possible. Yes, even when it seems counterintuitive. The Art of Specifics High-level conversations are comfortable. They’re the cruising altitude of…
Recently I have been working with some queries in Postgres where I noticed either it has decided not to use an index and perform a sequential scan, or it decided to use an alternative index over a composite partial index. This was quite puzzling, especially when you know there are indexes in the system that can perform these queries faster. So what gives? After some research, I stumbled upon…
In the fast-moving world of startups, shipping code is a crucial competitive advantage. But rapid deployment goes beyond just having the latest tools or the most comprehensive test suites; fundamentally, it’s also about trust. Trust within your team is transformative. It means believing that each member will effectively handle their responsibilities, understanding that collective…
It’s exciting to share that pg_easy_replicate, a project I’ve been maintaining, has hit a new milestone with its 2.0 release! This tool is more than a practical solution for setting up logical replication between PostgreSQL databases—it’s been a profound source of learning about PostgreSQL concepts for me. Engaging with the community and discovering the ways they use…
Before we dive in, let’s set the stage with a few pointers: YMMV. There’s no one-size-fits-all. My insights are primarily drawn from my experience of handling large-scale, high-transaction Ruby on Rails / Go applications backed by PostgreSQL. I am not here to tell you Foreign Keys are bad. They are great. This post isn’t about that. If there’s one takeaway I’d love…
Schema changes are usually critical operations to perform on a high volume database. One thing off, and you are looking at an outage. PostgreSQL has a lot of nice alternatives to make these schema changes safe. However, depending on the kind of schema migration, you would need to know exactly what the alternatives are and perform it exactly in the prescribed way. While you can build some…
While I enjoy working with MySQL (first database) both as a product and infrastructure engineer, lately, I have come to appreciate PostgreSQL as well. I recently read this post on why infrastructure engineers prefer MySQL, and I think it’s spot on. FWIW: This is not a MySQL vs PostgreSQL post. This is just a small summary of what I have come to appreciate about PostgreSQL as an…
Scaling Postgres Without Boiling the Ocean — Postgres Meetup for All (2026) PGConf NYC 2024 - Using Logical Replication for Major Version Upgrades and Tenant Migrations RailsConf 2023 (Video): Beyond CRUD: the PostgreSQL techniques your Rails app is missing 97 Things Every Cloud Engineer Should Know - O’Reilly Media Authored Chapter: Handling Network Failures in the Cloud Design Patterns for…
Originally wrote this as a part of a submission (Dec 2019) for an upcoming O’Reilly Book in 2021. Stay tuned.
I am a Software Engineer with Product and Infrastructure engineering experience. I enjoy building and scaling software led infrastructure. Some areas that are of interest to me (in no specific order or category): Distributed Systems, Site Reliability, Operations, Incident Management, Databases, Neural Networks, Running, Gulab Jamun and Biryani. shayonj on twitter, github, linkedin, and gmail
rollback scale down / up turn off feature flags keep looking a deploy went out, but unlikely related rollback turned on a feature flag few hours ago, not likely it turn it off error rate isn’t high, not the last deploy rollback none of the code paths touched seem suspicious rollback scaled up with a config change, its harmless scale down, rollback hm, what if its something else roll back and…
books current Einstein in Bohemia backlog Designing Data-Intensive Applications (Refresher) Code: The Hidden Language of Computer Hardware and Software The Manager’s Path Napoleon: A Life Sapiens: A Brief History of Humankind The Systems Bible The Pragmatic Programmer (half-read) Moonwalking with Einstein: The Art and Science of Remembering Everything The Design of Everyday Things previously…
Lately I have been very interested about Signal handling. So, this holiday season I embarked on a journey of writing a small project. One thing I noticed when inspecting signals is that there isn’t really an easy way of doing so. Or, especially, getting the “current” action handler for a given Signal. For instance, whether a signal (TERM, INT, etc) has an action of SIG_DFL or…