Rewriting Our Database in Rust

Airtable’s mission is to democratize software, making the power of software creation accessible to everyone.

Our product was originally built for human-scale data — helping individuals and small teams work together to get a job done. As the product has grown, so has the scale of its usage. Today, Airtable is used by more than 500,000 of the world’s largest organizations, including 80% of the Fortune 100, to power critical company-wide operations.

In the AI era, we’re seeing this trend accelerate: Airtable is increasingly a place where people and AI agents collaborate on complex projects. AI agents are able to read and write data at orders of magnitude more rapidly than their human counterparts, and they can scale out at a moment’s notice to accomplish new tasks.

To meet the growing demands on our system, we needed to rewrite the heart of our backend: our proprietary, in-memory database.

Our in-house database, originally written in TypeScript, is what enables Airtable to be a flexible and easily usable software platform. It supports a number of features rarely found in traditional databases, all aimed at providing an unusually high-quality experience for our customers: real-time updates to queries, collaborative rich-text editing, a formula language, linked records, attachments, manual record ordering, and much more.

But the JavaScript ecosystem isn’t a great fit for high-performance data processing. Its limited multi-threading capabilities and lack of fine-grained control over memory layout and lifecycle became bottlenecks as we scaled. So we decided to move it into a language and runtime environment better suited to the problem.

We chose to write our new database in Rust. The language offers a unique combination of high performance, memory safety, and developer productivity. In addition, Airtable already had several Node.js native extensions written in Rust, so we had experience with Rust-Node.js interop, and with running Rust in production.

After three years of development, validation, migration, and performance engineering, our new in-memory database has reached a level of maturity to the point that it’s now serving production customer traffic. We’re proud of this milestone as a technical achievement, but more importantly, we’re excited about what it means for our customers. It will solve some of the most difficult performance challenges Airtable faces today and help us scale our system into the Enterprise AI era.

Airtable Architecture Basics

Airtable is intended to empower customers to implement whatever workflows they need to do to get their work done. As a result, Airtable bases have highly heterogeneous load patterns, schemas, and computational requirements.

To enable this power and flexibility, we have a high degree of tenant isolation. In practice, that means we work hard to prevent noisy neighbor issues, so one customer’s runaway workload doesn’t impact another’s.

We do that today by hosting data in a single tenant architecture. We read all of the data for a given base into memory proactively and operate on it with a dedicated server (a worker). Requests are dynamically routed to the right backend server, and the server lifecycle is managed by a supervisor.

Because base-scoped data lives in-memory, these worker processes are typically CPU-bound, unlike traditional two-tier web application architectures, where backend servers rely heavily on I/O operations to databases or caches.

Scaling a CPU-bound workload requires adding more CPU capacity, but the single-threaded nature of JavaScript poses a challenge. While Node.js supports worker threads, they can’t easily share memory. Giving each worker thread its own copy of base data is infeasible. Large enterprise bases can require tens of gigabytes of memory, and keeping multiple copies in sync would be prohibitively expensive.

To unlock parallelism, we needed to move the data somewhere into shared memory where multiple JavaScript threads could operate on it.

Airtable’s Next Generation In-Memory Database

To enable parallel request processing, we rearchitected the worker to use multiple Node.js worker threads: a single write thread, a dynamic pool of read threads, and a dispatcher that routes incoming requests appropriately. All of these threads access data stored in our new in-memory database, implemented as a Node.js native extension written in Rust.

Our database provides many of the well-established interfaces and invariants developers expect: serializable, atomic transactions backed by MVCC; clustered and secondary indexes; a rich ORM-like data modeling and query interface; transactional DDL; foreign-key and unique constraints; triggers; and more.

But it’s also uniquely tailored for Airtable’s requirements.

For example, Airtable users see real-time updates when collaborators modify data they’re viewing. This means the backend must efficiently determine how each active query is affected by incoming writes. To support this, our database implements incremental view maintenance: it computes how a write impacts each live query without re-executing the query from scratch, and produces a compact diff instructing clients how to update their view of the data.

Airtable also supports features that can cause most, or all, of a column’s data to be rewritten simultaneously, such as the NOW() formula or certain column configuration changes. In many transactional databases, these operations are expensive. We’ve implemented efficient batch update operations to rewrite large portions of a table in one call. Our schema migrations (DDL statements) are also transactional, lazy, and require no special locking.

Other product features required additional specialization. We support native fractional indexes for drag-and-drop interactions; locale-aware string collation so numeric strings sort intuitively (for example, “10” sorts after “2”); and an efficient “comma join” operation used for filtering lookup values.

We’ll talk about more details in future blog posts — the low level algorithms and data structures, how we do Node.js/Rust interop, how rollbacks and garbage collection work, and some of the lessons learned along the way — but a significant portion of our work focused on safely migrating the existing application to this new database.

Application Migration

One important goal for our project was preserving our ability to develop new features quickly.

We built an alternative implementation of a widely used internal business-logic interface that maps application concepts onto generic database concepts and back again. This translation layer kept business logic in TypeScript, while allowing the Rust database to operate at a high level of generality. It also enabled us to migrate application code transparently while feature development continued at full speed.

To maintain backward compatibility, we implemented an extensive shadow validation system. Certain operations run against both the old and new databases, and we compare the results to detect divergences. We also extended an internal integrity-checking framework to exhaustively compare data across systems and validate critical invariants.

What’s Next?

Over the coming quarters, we’ll continue rolling out the new database to all bases and Airtable features. That work includes resolving long-tail correctness issues, further optimizing read and write performance, improving operational maturity, and migrating additional parts of the application.

We’re also developing a replication system and log-snapshot based storage system for our database to improve availability and initial load performance. We’ll share more about these projects in future blog posts as they ship.

If working on these problems at a company where performance and scalability are at the heart of the business sounds exciting to you, check out our Careers page and come chat with us!

Thanks to all past and present Airtablets who contributed to this project: Pierpaolo Baccichet, Anuj Bheda, Michael Busch, Daniel Chain, Kevin Chen, Kevin Cole, Alex Crawford, Yicong Du, Riley Hockett, Emily Houlihan, Catherine Huang, Chae Jubb, Josh Kang, Sam Keller, Brian Larson, Keunwoo Lee, Daniel Li, Xuanqi Li, Adar Lieber-Dembo, Sean McCullough, Mojtaba Mehrara, Ben Muschol, Aaron Myers, Akshay Nalla, Jayden Navarro, Daniel Norris, Chris Pederkoff, Yuval Steinhart, Keyhan Vakil, Vinay Valsaraj, Alex Yao, Phil Zeyliger, Anya Zhang, and many more.

Rewriting Our Database in Rust was originally published in The Airtable Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.