Recent content in Blogs on Justin Jaffray
This post first appeared in my new newsletter: Null Bitmap. If you’re interested in getting smaller, more rambly posts directly in your inbox please consider signing up. One lens I’ve found satisfying for “what is a query planner” is that it’s a tool for “approximating domain knowledge.” By that I mean, someone who knows Python could probably write a…
This post first appeared in my new newsletter: Null Bitmap. If you’re interested in getting smaller, more rambly posts directly in your inbox please consider signing up. What if you had an isolation anomaly with yourself? The year is 1976, Don’t Go Breaking My Heart is blasting on the speakers of the IBM San Jose Research Center, and you’re Patricia Selinger. You’re working…
Relational (inner) joins are really common in the world of databases, and one weird thing about them is that it seems like everyone has a different idea of what they are. In this post I’ve aggregated a bunch of different definitions, ways of thinking about them, and ways of implementing them that will hopefully be interesting. They’re not without redundancy, some of them are arguably…
I recently came across a paper called Distinct Elements in Streams: An Algorithm for the (Text) Book by Chakraborty, Vinodchandran, and Meel. The usage of the phrase “from the book” is of course a reference to Erdős, who often referred to a “book” within which God kept the best proofs of any given theorem. Thus, for something to be “from the book” is for it to…
Say you’re a query optimizer. Is the following transformation “valid?” select * from t where x > 5 => select * from t where false Well…I doubt it. The first version permits strictly more rows than the second. But, what if I told you that we recently collected statistics on our data, and found that no more than 0.01% of our data has a value of x that’s larger than 5?
Or: the core idea of how database query planners decorrelate subqueries. Joins In a relational database, to compute the join of \(R\) and \(S\) on \(p\), for each row \(r\) in \(R\), find all the rows \(s\) in \(S\) where \(p(r, s)\) is true, and emit the concatenation of \(r\) and \(s\). Readers of this blog might note this is a different definition than we’ve used in the past, but this one…
If you’ve been following databases in the past couple years, you’ve probably encountered the term “worst-case optimal joins.” These are supposedly a big deal, since joins have been studied for a long time, and the prospect of a big shift in the way they’re thought about is very exciting to a lot of people. The literature around them, however, is primiarly aimed at…
The most fundamental property a database can provide is durability. That is, once I’ve told you that your write has been accepted, if a mouse chews through the power cord for the server rack, the write will not be lost. This obviously is only possible to a degree. If someone goes into your SSD with a magnetized needle and a steady hand and tweaks their bank balance, then (short of replication) the…
You hear the word “log” a lot when talking about databases. This is because databases are fundamentally comprised of sin. (Shared mutable state.) Logs are natural in a setting where you want to be principled about your treatment of mutation because one way to think about mutation is how a variable starts from an initial value and undergoes several “transformations.” This sequence of operations:…
People talk a lot about “pull” vs. “push” based query engines, and it’s pretty obvious what that means colloquially, but some of the details can be a bit hard to figure out. Important people clearly have thought hard about this distinction, judging by this paragraph from Snowflake’s Sigmod paper: Push-based execution refers to the fact that relational operators…
This is not going to be my normal kind of post, it’s not very focused, and going to be a bit rambley, as I talk about a problem I thought about one day. Magic: The Gathering is a card game where players construct decks, typically of 60 cards plus a 15 card sideboard, for 75 cards total. Periodically, the company that makes the game, Wizards of the Coast (WotC) publishes a list of decks that…
Sometimes I write posts because I think I have a fresh perspective on something, and sometimes I write posts because for whatever reason I think every explanation of a particular concept that I’ve seen is bad. This is the latter! This post is about a lovely technique for discrete optimization called Branch and Bound. Optimization problems can generally be viewed as a search for an object…
The dominant way modern query planners decide what algorithms to use for a given query is via a cost model. Effectively, they enumerate the set of possible plans for a query, and then run their cost model over each one, eventually returning whichever one had the cheapest cost according to the model. Analytic queries can become complex, horrifically so, in fact, and assessing their costs follows…
In anticipation of starting at Materialize, I’ve been reading through some of the fundamental literature that the product is built on. The first paper I’ve read through is Naiad: A Timely Dataflow System. I enlisted the help of friends Forte Shinko and Ilia Chtcherbakov to help me work through it, and we ended up finding an interesting question that I’m not aware of a proof…
You could be forgiven for thinking that a “query optimizer” is a component of a database that takes a query plan and makes it better, hence, the typical programmer definition of “optimize.” This is not really how the term is used in practice, and “query optimizer” is really pretty synonomous with “query planner.” I think “query planner”…
This post was originally posted on the Cockroach Labs Blog Even in the 80’s, before Facebook knew everything there was to know about us, we as an industry had vast reams of data we needed to be able to answer questions about. To deal with this, data analysts were starting to flex their JOIN muscles in increasingly creative ways. But back in that day and age, we had neither machine learning nor…
This post was originally posted on the Cockroach Labs Blog The development of the relational model heralded a big step forward for the world of databases. The introduction of SQL meant that analysts could construct a new report without having to interact with those eggheads in engineering, but more importantly, the existence of complex join queries meant that theoreticians had an interesting new…
Depending on which computer scientists and greek lawmakers you listen to, Paxos could either be taught to a 3-year old, or requires a team of PhDs “almost a year” to fully grasp. I think the discrepancy here comes from a poorly motivated setting. I couldn’t have provided a coherent explanation of what consensus was a year ago, but in the time since then I’ve slowly pieced together an…
This post was co-written with Ilia Chtcherbakov. CASPaxos is a recent and interesting consensus algorithm invented by Denis Rystsov. The paper describing CASPaxos provides a proof of correctness in its appendix, provided here is an alternative proof. First, let’s review the protocol. There are several participants. Clients A client submits a transformation function to a proposer. The…
This post is about gaining intuition for Write Skew, and, by extension, Snapshot Isolation. Snapshot Isolation is billed as a transaction isolation level that offers a good mix between performance and correctness, but the precise meaning of “correctness” here is often vague. In this post I want to break down and capture exactly when the thing called “write skew” can happen.…
I’ve been on both ends of several variations of the following conversation: FRIEND: Check out this algorithm. It checks that a string of parentheses, brackets, and braces is balanced properly. It goes like this: You iterate through the string while keeping a stack. When you see an opening paren/bracket/brace you push it on. When you see a closing paren, if it’s the same kind as the one…
Speedrunners turn the process of beating a single-player video game into a competitive activity, trying to finish games as quickly as humanly possible. This has become increasingly mainstream (though still niche) in the past few years with the rise in popularity of events like Games Done Quick. However, there’s another, smaller community interested in beating games quickly but from a…