Collection of notes on distributed systems.
Scheduling threads like Thomas Jefferson Table of contents Background and motivation Inspiration and prior work Big picture Prototype implementation Running the prototype Unexpected connection to Thomas Jefferson Conclusion and future work Posted on Sep 24, 2024 This post is about how to schedule workers across a pipeline of queues in order to minimise total processing time, and an unexpected…
Parallel property-based testing with a deterministic thread scheduler Table of contents Background Motivation and overview Deterministic scheduler Thread-scheduler communication Managed threads Scheduler Example: broken atomic counter Parallel property-based testing recap Integrating the scheduler into the testing Changes to sequential module Changes to parallel module Changes to the counter…
The sad state of property-based testing libraries Table of contents The history of property-based testing A survey of property-based testing libraries Analysis Why are property-based testing libraries in such a sad state? Stateful and parallel testing isn’t as useful as pure testing Stateful modelling requires training Closed source helps industry adoption What can we do about it? Synthesis Pure…
Towards zero-downtime upgrades of stateful systems Table of contents Motivation Terminology Software systems Programs and their upgrades Implementation State machines Example Semantics Pipelines Deployment Upgrades Sources and sinks Remote upgrades Discussion and future work Posted on Mar 7, 2024 Motivation Most deployed programs need to be upgraded at some point. The reasons vary from adding new…
Parallel stream processing with zero-copy fan-out and sharding Table of contents Motivation and inspiration Prior work Plan List transformer model Queue pipeline deployment Disruptor Disruptor pipeline deployment Observability Running Further work and contributing See also Discussion Posted on Jan 23, 2024 In a previous post I explored how we can make better use of our parallel hardware by means…
SVG viewer written in SVG Table of contents Motivation Demo The code Usage Contributing See also Posted on Sep 9, 2023 This post is about how to write an SVG viewer / browser / “slideshow” which is itself a self-contained SVG. Motivation I’ve been working on a parallel processing pipeline. Each stage of the pipeline is running on a separate thread, and it takes some work items from a queue in…
Elastically scalable thread pools Table of contents Motivation Plan Pseudo-code Main Worker pool Load generator PID controller How it works Usage Contributing See also Discussion Posted on Mar 14, 2023 An experiment in controlling the size of a thread pool using a PID controller. Motivation A tried and tested way to achieve parallelism is to use pipelining. It’s used extensively in manufacturing…
pipelined-state-machines Table of contents Motivation Usage Disruptor Example How it works Performance Contributing See also Presentations Writings Posted on Mar 1, 2023 An experiment in declaratively programming parallel pipelines of state machines. Motivation Imagine a flat complex in Sweden. Being the socialist utopia Sweden is there’s a shared laundry room which the people in the flat complex…
hot-swapping-state-machines Table of contents Background Usage How it works Contributing See also Acknowledgments Posted on Feb 21, 2023 An experiment in implementing remote hot code swapping, or dynamic code upgrade, for state machines. Background In Erlang it’s possible to seamlessly hot swap the code on a running process. Consider the following gen_server implementation of a counter which can…
supervised-state-machines Table of contents Motivation How it works Generic server Supervisor Event loop Example Contributing See also Posted on Feb 13, 2023 An experimental implementation of Erlang/OTP’s gen_server and supervisor behaviours that doesn’t use lightweight threads and message passing. Motivation What exactly is it that makes Erlang suitable for writing reliable distributed systems?…
Property-based testing stateful systems: a tutorial Table of contents Structure Table of contents Usage Contributing License Posted on Feb 7, 2023 Property-based testing (PBT), i.e. generating random inputs and checking some property of the output, of pure programs is an established practice by now. It’s taught in introductory university classes and it’s part of test suites in industry. Most real…
Erlang’s not about lightweight processes and message passing… Table of contents Background Behaviours Generic server behaviour Event manager behaviour State machine behaviour Supervisor behaviour Application and release behaviours How behaviours can be implemented Correctness of behaviours Contributing See also Discussion Posted on Jan 18, 2023 I used to think that the big idea of Erlang is its…
Towards human-readable binary encodings Table of contents Motivation Erlang’s bit syntax Usage How it works Extending Erlang’s bit syntax In-place updates On-disk data structures Zero-copy Backward- and forward-compatiability and migrations Compression Checksums Validation Protocols Pandoc for binary encodings Discussion Contributing See also Posted on Jan 11, 2023 Can we make binary encodings…
coroutine-state-machines Table of contents Usage How it works Contributing See also Posted on Jan 7, 2023 State machines of the type Input -> State -> (Output, State) are great. They are easy to reason about, and if run on a separate thread with access to a queue of Input s they perform well too. Sometimes the state machine might need to do some blocking I/O before producing the output though,…