The work done and a word about Lambda Combine ¶

Hyper Guide at Lambda Combine

I’ve written before about first experiments with Datastar and Common Lisp and then about the resulting SDK. This post is an attempt to map the full body of work, told as a sequence of problems/reactions.

Each project grew out of a limitation in the previous one, and understanding that chain is the best way to understand what each piece does.

The repositories involved:

• datastar-cl,
• lc-sse,
• cl-brotli,
• clack-handler-jdkhttp

Entry to Disneyland, 1950s

They are all anchored in the hyper-guide, in one way or another, and the guide has interactive examples. Lambda Combine as a whole is a vehicle for what I consider interesting Common Lisp work (or interesting work in general, which is the same thing): almost a simulacrum, with all that entails, but yet also a simulation that is closer to reality than most other social media reflections.

Datastar, hypermedia, and SSE ¶

datastar-cl is a Common Lisp implementation of the Datastar SDK, following the Datastar Architecture Decision Record as closely as possible. The details are in Common Lisp SDK for Datastar; what matters here is one early decision, that the SDK was not going to be tied to a single web server.

Common Lisp has several web servers – Hunchentoot and Woo specifically– and Clack is an abstraction layer over both.

Building on only one of them would have been the comfortable choice, and it can be argued that a Clack-only implementation would be enough, but direct Hunchentoot support is for me also important since not everyone will like or want ot go through Clack’s, and Hunchentoot remains the most well-documented Common Lisp webserver, and one which I use by default.

The SDK uses CLOS for this: an sse-generator base class with hunchentoot-sse-generator and clack-sse-generator subclasses, and a generic function call-with-generator as the dispatch point where backend behaviour is specialised. The public surface is a single with-sse macro that picks the backend at macroexpansion time from the shape of its second argument: a plain symbol means Hunchentoot, an (env responder) pair means Clack.

The SSE handling code quickly became the largest source of code. Since the SSE machinery isn’t strictly dependent on Datastar, – the streaming, the keep-alive, the disconnect handling – I opted to let it live a life of its own.

lc-sse: stress testing SSE ¶

lc-sse is the result of that: it has no dependency on any higher-level framework, Datastar or otherwise: it is a general-purpose SSE library that happens to be the foundation on which the SDK sits.

Common Lisp had (or at least I couldn’t find it) no single library that combined all of the following for proper SSE handling:

• A thread-safe generator lifecycle with locking.
• Proxy-safe keep-alive with automatic client-disconnect detection: a dead socket write raises a client-disconnected condition (a stream-error subtype), which unwinds the with-sse body, fires the :on-disconnect hook, and closes the stream.
• Streaming HTTP compression: zstd and brotli optional subsystems. Both negotiated automatically from Accept-Encoding (more on this below), SSE compression is a thing on its own.
• Broadcast and keyed-unicast registries (sse-registry, keyed-sse-registry) with snapshot-under-lock fan-out and automatic unregistration of dead generators. This was a result of a deeper dive into CQRS.
• tcp-nodelay-easy-acceptor a helper to suppress the Nagle 40 ms stall on small chunked writes (Hunchentoot only for now).

To identify issues, I not only used the library in my own applications (and in the hyper guide), I also stressed tested it. There’s not a lot out of the box for this: to properly test SSE I wanted to have long-term, active connections opened and measure errors at different layers (transport, application). I’ve built a custom stresser (not yet published), which led me to having to face threading models: how did different backends behave if we had non-blocking updates? And blocking?

Threads and event loops ¶

To understand the end result, I should summarise the two broad approaches to handling concurrent HTTP connections, and how Common Lisp’s web servers (Hunchentoot and Woo, in this case) represent both of them.

Thread-per-request (Hunchentoot)

each incoming connection gets its own OS thread. The handler can block – on I/O, on a sleep, on a database query – without affecting any other connection. The code stays simple, linear, no surprises. The limit is the number of threads the OS will sustain, and each thread has a fixed stack cost. For many workloads, thousands of threads is perfectly fine; for long-lived SSE streams each one just sits parked, which can be wasteful or also absolutely fine: operating systems and computing resources are capable of much more than we assume, and having thousands of threads is fine.

Event loop (Woo)

A small pool of worker threads, each running a libev event loop. Very high throughput per thread, low memory overhead, similar to the model (to the best of my knowledge, and removing the specificities of each) used by Node.js or nginx. The hard rule: do not block the event loop: while one callback runs, no other I/O or timer fires on that worker. A sleep, a synchronous database query, or any blocking wait inside a handler stalls every other connection on that worker.

W3 Threads and Event Loops

I found Woo’s behaviour early on in the SDK development: each time I had a “sleep 10”, for example, in a SSE body, Woo would block on the first request. The alternative was to increased the number of workers in Woo, but this isn’t really scalable or the way Woo is supposed to work.

There is a third model that mostly resolves this (and by “resolve” I mean that there is a specific style of programming needed for event loops): virtual threads (also called fibers or green threads in other contexts). The idea is of user-mode threads scheduled cooperatively onto a small pool of OS threads, so cheap enough to have one per connection, blocking-safe because the scheduler intercepts blocking calls and yields to another virtual thread, scaling to hundreds of thousands. Go’s goroutines are the most widely-known example. The JVM added them as Project Loom virtual threads in JDK 21. Common Lisp implementations do not have them natively.

This matters especially for CQRS push with Datastar: one long-lived GET connection per client (server pushes over SSE) and many short POST requests for writes. The POST handler runs on whichever thread picked up the request – which is not the thread that opened the GET stream. On Hunchentoot, writing to the GET stream from the POST handler is fine: they are both just threads. On Woo, it is a different matter: the SSE stream is tied to a libev I/O watcher in the original worker thread, and writing to it from another thread accesses invalid memory. The consequences are undefined, and in practice bad.

CQRS is trivial on Hunchentoot, but unusable on Woo. Solving it without virtual threads required a different approach.

User channels for Woo ¶

The first attempt was, in retrospect, solving a problem at the wrong layer: a libev “reactor” embedded into the SDK itself via CFFI, reaching into Woo’s internals to register timers and async-signal cross-thread writes.¹. It “worked”, but it was fragile – a version bump in Woo could silently break it, and the code was more than a fifth of the total SDK source, managing event-loop plumbing that had nothing to do with SSE or Datastar. The most informative commits are sometimes the ones that delete things. Perhaps as importantly, it was made in a hurry, without really understanding the implications of the code.

While looking around, I found this important comment in the Hunchentoot recycling taskmaster repository:

Woo becomes significantly slow if the handler is even slightly delayed. With the following setup for the “1ms sleep” benchmark, I observed poor results:”

  (defparameter *handler-sleep-seconds* 0)

  (defun handler-small-sleep ()
    (sleep *handler-sleep-seconds*))

  (woo:run
   (lambda (env)
     (declare (ignore env))
     (handler-small-sleep)
     '(200 (:content-type "text/plain") ("Hello, World")))
   :worker-num 8)

In cases like this, you generally don’t sleep inside the async server’s event loop. You run time-consuming processing outside the event loop, and when it’s finished, you notify the event loop of the content to be sent and received, or set up a callback to be called. My code for benchmarking Wookie does it.

But for some reason, Woo doesn’t seem to have such a mechanism. I couldn’t find it. quickdocs-api, which is said to use Woo, do not seem to take such considerations into account. This code is also.

Some people on here or here have said that “offloading is possible with lparallel”, but I have yet to find any code that actually does this. The following naive code, which creates a thread in the handler, will result in an error.

(...)

This problem can be solved by directly handling libev, which Woo depends on, directly handling the event loop. Gemini-CLI wrote this code to do so. It certainly works, and the benchmarks aren’t bad. However, I don’t know how this code works. (Please don’t ask me.)

I had started precisely this approach, and looking at the Gemini code it was similar to what I had... but since I was adding this at the SDK level, it was a mess and I couldn’t really understand some of the things there in terms of impacts on other existing code.

Removing Woo support, re-adding Woo support ¶

After the removal of the initial Woo support (in what it specifically needed as to “not block”, since Woo is supported as a Clack backend regardless of this), the work on the CQRS demos and the stress testing didn’t let the idea go away: the code was messy, yes, but what if we could split it into something more general, in Woo itself?

Starting mostly from scratch (but with some better understanding of the libev code) this was the start of the small fork² of Woo that exposes two things: the evloop reference in the Clack environment (under :woo-evloop) and a WOO:SCHEDULE primitive that schedules a callback onto a specific worker’s loop.

The lc-sse/woo subsystem installs a call-with-generator specialisation that routes all SSE writes through WOO:SCHEDULE, so a background thread doing a CQRS broadcast calls notify-subscribers and each write lands in the correct worker context. Keep-alive heartbeats become woo:register-timer callbacks instead of blocking loops, so the request handler returns immediately and frees the worker.

Woo at 4000 sustained SSE connections (1CPU 1GB RAM)

This allows CQRS without virtual threads: making the async server do what it otherwise could not. I once again used my load-testing tool, lc-stresser (currently unreleased) that spawns one thread per SSE connection, ramps to a target count, holds connections for a configurable duration, and measures delivery delay from a timestamp field embedded in each event. A limit mode climbs until the first error to find the ceiling. A companion script samples the server process’s RSS, CPU percentage, and thread count at 1 Hz from /proc so one can correlate client-side load with what the server is doing. The result was promising: thousands of parallel long-lived SSE streams, clean disconnect, no timer or channel leaks (visible, at least): the plot above is perfectly comparable to the go web server (used as an external reference), and in the same conditions Hunchentoot supported >4000 connections, being OOM killed before reaching 5000³.

Virtual threads in Common Lisp: enter the Armed Bear ¶

The previous Woo work solved something: for the first time, we can use an event pool webserver in a way that allows CQRS-like patterns. For this, lc-sse, as mentioned, provides a specific approach: when with-sse has no body, it automatically uses it. When there is a body, it doesn’t, and it blocks.

Miś Uszatek

I’m quite happy with this approach, and I’m using it in the Hyper Guide and other demos: it works, and it’s clear about what it does and when.

... but I couldn’t completely shake off the idea of virtual threads, and the possibility of using something that doesn’t require any specific pattern to be used: it automatically uses an event loop when possible, and creates a thread when not.

Looking into virtual threads, I found this experimental work toward fibers in SBCL, and the idea has been discussed in the CL community for some years. As of now, there is nothing production-ready or portable that I could find: native virtual threads (even if only applicable to certain compilers and then degrading to regular threads) remain a gap.

Which leaves one interesting option: borrow a runtime that already has them.

Some time ago (or, more than a decade ago, if we must) I did some work on ABCL with Swing, as well as Using MQTT in Common Lisp with ABCL and the Eclipse Paho client., so I already knew that ABCL was great. I also knew that Java had Loom, which added fibers, continuations and tail-calls for the JVM, but I wasn’t actually sure what this meant in practice – not until know, anyway.

If the JDK has virtual threads, and if ABCL runs on top of the JDK...

clack-handler-jdkhttp: borrowing Loom ¶

The right of the people to keep and arm bears shall not be infringed!

So, ABCL (Armed Bear Common Lisp) runs Common Lisp on the JVM. It is not usually the first implementation people reach for (I often do, if nothing else to identify what I’m using that is not strictly Common Lisp), but it has one property that no native implementation has: access to the full Java ecosystem, including (transparently) to Project Loom virtual threads, stable since JDK 21 and with thread-pinning on synchronized removed in JDK 24 (cf. JEP 491).

I was already using Clack on lc-sse, so the idea emerged: what if we could add a new backend to Clack that used the JDK http class? That would make code that targets Clack immediately able to use whatever advantage we could extract from it (in our case, virtual threads).

The result of this woek, clack-handler-jdkhttp, is a Clack backend handler for ABCL that runs on top of the JDK’s built-in com.sun.net.httpserver.HttpServer.

The executor is set to Executors.newVirtualThreadPerTaskExecutor(): one virtual thread per request, scheduled transparently onto a pool of OS threads. Since it’s a standard Clack handler (the plist-in / response-out contract that Lack defines, etc.), lc-sse and datastar-cl load on top of it unchanged: there is nothing to port, no new API to use.

This was very interesting to me conceptually but there’s a real benefict: we can write blocking, thread-per-request style handlers – exactly as we would with Hunchentoot – and scale to thousands of long-lived SSE streams because each one runs on a “cheap” virtual thread. There’s no event-loop specific programming required, and no change in the way applications are developed (since we are using Clack).

The Clack backend tested with >1000 connections. Live Demo

This is, as far as I know, the first time a Common Lisp web application can be dropped onto a virtual-threads web server without any changes to the application code above the handler level. I’m still testing and doign benchmarks (the CPU and RAM implications of this stack should be peculiar, in that they likely will require more initial resources).

Compression: adding cl-brotli, ¶

Compression – BBS The Documentary #8

One of the things Datastar encourages – and one of the things the hyper-guide uses – is “fat morphing”: instead of computing a precise diff on the server, you send the whole region and let idiomoprh apply only the actual diff in the browser.

Streaming HTTP compression becomes useful here, not just a nice-to-have. lc-sse negotiates from the client’s Accept-Encoding: zstd via cl-zstd, and now brotli is available via cl-brotli: cl-brotli is a separate library that came out of this work – brotli had no CL binding before it.

Anders Murphy has written on why brotli is especially usef in SSE, so adding it was a natural follow-up to the Datastar work.

The hyper-guide ¶

Baikonur Cosmodrome

As mentioned before, The hyper-guide on Lambda Combine is a progressive guide that walks from the simplest server push to a full multi-client CQRS application, prevalence, event sourcing, etc. . Each of the eleven chapters introduces one concept, shows it running live inside the page, and provides the full source of the equivalent standalone program that can be loaded with sbcl --load <file>.lisp and usually runs on port 8989. The Datastar “Tao” it tries to convey is this:

• Most state lives on the backend. The backend is the source of truth.
• The backend drives the frontend by patching HTML elements and signals.
• Use signals only for user-interaction state and for sending values to the server.
• Trust the morph. Send large regions; idiomorph applies only the diff.
• An SSE event stream is a plain HTTP response with a special content type.
• Separate reads from writes: one long-lived GET stream, short-lived POST commands.

Chapters run from plain SSE (patch-elements) through signals, scripts, interaction, remove-and-append, fat morphing, CQRS with the opt-in registry, Snooze routing, client reactivity (computed, show, indicator), event sourcing, and finally prevalence with BKNR datastore – which is also where the connection to the epigraph above closes.

Using BKNR datastore (and here as well I had some previous involvement, in this case around the XML import/export code in the datastore) for the prevalence example was not a coincidence: the hyper-guide is at least partially a different take on the BKNR web framework, in the same spirit but with more recent tools.

Replicant Detector and other demos ¶

A final note on a demo: The Replicant Detector is the best demonstration of what the Woo user-channel work enables (although it’s also perfectly usable with Hunchentoot as well): t is a replicant registry – Blade Runner flavour⁴ – with a geolocated SVG world-map radar. Every Spawn, Sight, or Retire command geolocates the operator’s IP via MaxMind GeoLite2-City and pings the location on the map. All connected clients see the update simultaneously.

The source is in datastar-cl/examples/rep-detect/; the demo is at https://rep-detect.lambda-combine.net/.

Closing ¶

A launch platform’s job is to get something off the ground. What happens after that is the interesting part, but I must confess that I’ve been spending an excessive amount of time in “infrastructure” instead of my initial objective.

None of it is finished – lc-stresser still needs a release, the Woo primitives need to go upstream, more hyper-guide chapters are possible, and since I’ve been joining and splitting things, there is almost certainly a lot of small problems that will need correcting... but enough is working that it made sense to write it down and announce it, lest I continue down the rabbit hole.

It’s been fun, at least.

Resources ¶

• datastar-cl – Common Lisp SDK for Datastar
• lc-sse – General-purpose SSE library for Common Lisp
• clack-handler-jdkhttp – Clack backend for ABCL using JDK virtual threads
• cl-brotli – brotli compression for Common Lisp
• The Hypermedia Guide – progressive live examples
• rep-detect – live CQRS + SVG morphing demo
• Datastar – the Hypermedia Framework
• Hypermedia Systems – the HTMX authors on hypermedia
• BKNR – the original Lisp satellite launch platform
• Woo – async Common Lisp web server
• ABCL – Armed Bear Common Lisp (JVM)
• Project Loom – JDK virtual threads
• SBCL fibers – experimental work toward virtual threads in CL