Not sure about you, but I’ve been struggling a lot to figure out how things are shifting in the software industry. I think I’ve made some progress, and here’s where I am currently at: Software companies are shifting to selling work, not software . Customers never really wanted the software; they just wanted the results that could be produced by that software. Those results are produced by the work…
I’ve been trying to gain more clarity on the debate on whether we should be reading code or not. Antirez’s blog post is interesting. After reading it, I actually feel more puzzled than ever. The first point I’d like to explore is this quotation: Then I compared the implementation, for correctness, to other systems, finding that other implementations sometimes contained more errors. I researched…
This is a follow-up on The Anatomy of a QuestDB Table on Disk . There we looked at how data is natively stored on disk for some data types. In this post, we look at what is approximately happening at the storage layer when writes occur. For that, we implement a simplified version of QuestDB’s storage engine. We’re particularly focused on implementing just what is necessary for the storage engine…
As I continue exploring QuestDB, one natural thing to investigate is how data is stored on disk. In this post, we explore QuestDB’s native columnar format, how data is stored on the bit level, and some interesting techniques that are common in other databases, such as German strings for VARCHAR storage and dictionary encodings for SYMBOL storage. Let’s use the following table as our example: ts…
I’ve been exploring the QuestDB codebase, a time series columnar database, and, specifically, trying to learn how its query engine works. And whenever we discuss a query engine execution , certain kinds of dimensions and options within those dimensions arise in the conversation. For example: Dimension Options Dataflow Pull-based vs. Push-based Processing Granularity Tuple-at-a-time vs. Vectorized…
This post is part of a series on Fly.io’s distributed systems challenges : Implementing Snowflake Unique ID Generation Generating Unique IDs with Raft Consensus Fly.io’s Broadcast Challenges Building a Grow-Only Counter on a Sequentially Consistent KV Store (this post) We’re going to discuss Challenge #4: Grow-Only Counter . This challenge is particularly tricky. I wouldn’t say it’s hard, but if…
This post is part of a series on Fly.io’s distributed systems challenges : Implementing Snowflake Unique ID Generation Generating Unique IDs with Raft Consensus Fly.io’s Broadcast Challenges (this post) Building a Grow-Only Counter on a Sequentially Consistent KV Store This is another post about the series of distributed systems challenges by Fly.io. I’ve talked about using Snowflake and Raft…
This post is part of a series on Fly.io’s distributed systems challenges : Implementing Snowflake Unique ID Generation Generating Unique IDs with Raft Consensus (this post) Fly.io’s Broadcast Challenges Building a Grow-Only Counter on a Sequentially Consistent KV Store This blog post is a follow-up on Implementing Snowflake Unique ID Generation . In that post, I explain an implementation of…
This post is part of a series on Fly.io’s distributed systems challenges : Implementing Snowflake Unique ID Generation (this post) Generating Unique IDs with Raft Consensus Fly.io’s Broadcast Challenges Building a Grow-Only Counter on a Sequentially Consistent KV Store I’ve wanted for some time to work on this series of distributed systems challenges by Fly.io, and I have finally found some time.…
This is a note to myself on some of the occasions the move constructor is called in C++. Let’s use a simple class that implements move constructor and move assignment with some print statements for debugging/confirmation. class A { public: A () { std :: cout << "Default constructor called \n " ; } A ( A && other ) noexcept { std :: cout << "Move constructor called \n " ; } auto operator = ( A &&…
Reading Two Beautiful Rust Programs I came across a construct I hadn’t seen before for working with threads in Rust: std::thread::scope . When you create a thread in Rust, you must pass a closure of the code you want that thread to execute. Sometimes, that closure works on data from the parent scope (the code that is spawning the thread): fn main () { let mut counter = 0 ; let f = || { counter +=…
Recently, I began to delve deeper into Rust. As a way to become more familiar with the language, I decided to write a simple lexer for a mathematical expression such as 10 - 3 + ( ( 4 / 2 ) * ( 8 * 4 ) ) . Writing a lexer shouldn’t be a difficult task, especially if you’ve built one in another language. I’ve tried to explore Rust features and write idiomatic code as much as possible, without…
Introduction Last weekend I decided to take a deeper look at the famous SQLite 35% Faster Than The Filesystem benchmark. I didn’t want to do a shallow read of the post. I wanted to compile the kvtest tool and run the experiments myself and see what is going on. I recommend doing that, especially if you want to follow along with the blog post. While running the read experiments, something caught my…
Introduction The other day, I was thinking about how I could get the bytes of a record of a recently inserted or updated row in SQLite . The motivation for that is that I wanted to create a hash of that row, essentially, to be able to build a Merkle Tree of the corresponding table as rows get inserted or updated. The closest API that SQLite offers to what I was looking for is the…
Another day I was looking at a simple classic implementation of a shared counter in C++ using mutex, and I wondered what other thread-safe implementations existed. I usually use Go to explore my curiosity. The result of this exploration is a compilation of ways on how to implement a goroutine-safe counter. Don’t Do This Let’s start with the non-safe implementation. type NotSafeCounter struct {…
The Clock Synchronization and Ordering Problems A single node system has no problem deciding what time it is and which order the events inside the system happened. The node has a timer, called clock, and any process that needs to make use of time makes a call to the operating system. If process (a) makes use of time before a second process (b), the time read by (a) will be smaller than the time…
I was reading about serialization formats the other day and came across the last column “Supports Zero-copy operations”. I had no idea of what it meant. Moments before I got on this Wikipedia page, I was looking into how to serialize a struct in *Go, *without using any specific format, just raw serialization (don’t even know if the term raw serialization means anything). While searching for a way…
In computing, caching is all over the place. It is found in hardware ( CPU and GPU ), operating system’s virtual memory, buffer pool managers inside databases, in the Web ( Content Delivery Networks , browsers, DNS servers …), and also inside the applications we build. In this post, we take an abstract look of what cache is, the replacement problem that arises from cache’s own nature and some…
I have been exploring how disk-oriented databases efficiently move data in and out of disk. One way, that I explored in Discovering and exploring mmap using Go and But how, exactly, databases use mmap ? , is through memory-mapped files . Although mmap is a really neat solution, it has some troubles. Most troubles come from the fact that the database has no control of how pages are flushed to disk…
In a previous post Discovering and exploring mmap using Go , we talked about how databases have a major problem to solve, which is: how to deal with data stored in disk that is bigger than the available memory . We talked about how many databases solve this problem using memory-mapped files and explored mmap capabilities. Knowing that databases use memory-mapped files to solve the problem was not…
Recently I’ve come to know the concept of memory-mapped files while watching a lecture of the course Intro to Database Systems of Andy Pavlo on database storage. One of the main problems a database storage engine has to solve is how to deal with data in disk that is bigger than the available memory . At a higher level, the main purpose of a disk-oriented storage engine is to manipulate data files…
A Plea for Lean Software 1 is a classical paper that presents us with some hints of why software increases in complexity and gives us some advice on how to avoid or minimize complexity. Here I present my interpretation of Niklaus Wirth’s ideas in a non linear way adding some personal reflections on top of it. The amazing thing about this paper is that it was written in 1995 , when programs where…