Hi, I'm Lev, I'm a software engineer currently working full time on PgDog with Rust. I live in San Francisco, CA. In my free time, I climb at Mission C...
PostgreSQL clients and servers talk to each other via TCP. TCP is a streaming protocol, which means that data sent over the socket isn't delineated in any way: it's all just a bunch of bytes. Clients and servers deal with this limitation by expecting specific bytes to arrive at a certain time in the life of a connection. They are asynchronous state machines, which means they can send (and receive)…
When you split your database into shards, you need to figure out which one has the data you're looking for. The simplest solution is to query all of them, but this is slow, even if done in parallel, and doesn't scale. However, if you know the sharding function used to split the data, you can apply that same function to your query parameters, and send it to only one shard. This post is part of a…
This is old news by now, but since PostgreSQL 10, we can use the replication protocol to sync individual tables between databases. Messages sent between databases describe table changes in a readable format, which also happens to provide enough information for us to split that data between shards. This document contains the overall thesis on how this works. The implementation is currently being…
Sometimes when I'm learning a new library, I write a test. In Rust, this is as easy as: #[cfg(test)] mod test { use csv :: * ; #[test] fn test_csv () { let mut reader = ReaderBuilder :: new () . has_headers ( false ) . from_reader ( "one,two,three" . as_bytes ()); let row = reader . records () // get a records reader . next () // read a record . unwrap () // panic if there aren't any . unwrap ();…
If you ever wanted to read a SQL query and understand what it's doing, you'd just do it. But if you're a computer, you need something called a parser. You can write one yourself, but you need a degree in linguistics (or if you know any dragons, they can help you out). Fortunately, this is a solved problem so you needn't concern yourself. If you're like me and writing code is best done the hard…