Process names: The forgotten Observability channel

When it comes to observability, especially in an incident, more data is usually better. Without data, it's almost impossible to know what's going on. With insufficient data, you only get half the picture.

Observability tools will generally collect data from logs, metrics from your application itself, and if you're really fancy, even traces as your application is running. These are collected together and displayed in "pretty graphs" (technical term) for all to see.

Setting all this up is costly, and often assumes a lot about your deployment environment. Log collectors require logs to be in standard formats in known locations (which they should be if it's 12-factor). Metrics usually requires complex networking to ensure each instance of your application is individually addressable. And traces requires introspection into your entire application stack, which may not be possible if you didn't write it.

There's another export mechanism often forgotten - a way to communicate to us humans what your application is doing, without any more than standard unix tools: The process name.

#Why process names matter

When you start an application, its "process name" is usually just the of the executed binary, along with any command-line arguments. Armed with that, you usually get a fairly good idea of what's going on. If the redis-server process is consuming more CPU than it should, you should probably go look at your Redis deployment. But in some deployments it's not as obvious. My server has a node dist/index.js process running - what's that?

The stock process name only goes so far. It might tell you what it is, but it doesn't tell you what it's doing. In a standard Python web application, you might have a number of granian processes, but if one is using 3 times more CPU than the others, you can't immediately see why.

Process names offer a notable benefit over more complex observability too - they're realtime. Many metrics are output at the end of a request, not during. You don't need to wait for your ingestion pipeline to collect the data, aggregate it and show it in a dashboard. You just run htop and away you go.

#Solution

So if people often forget about process names, what can we do with them? Well it's simple: Put some useful details there!

If your process is doing something notable, stick it in the process name. Writing the exact function being executed is too granular, but seeing the request path at a glance could be invaluable.

Process names aren't really intended for machines to read - they're just for us humans. Writing JSON structured data in there is no use to anyone, but a few simple logfmt pairs could get the point across. There's such thing as "too much" - if you write too much data then it's no better than a log file.

A benefit of process names is they're instant. If you're watching htop (or alike), you'll see process names change instantly. As soon as a process starts working on something, the name will be updated to reflect. That means there's no ingestion delay, and less combing through logs after the fact to work out what already happened.

#Example: PostgreSQL

For example, PostgreSQL. As usual, PostgreSQL is an example of doing something right. Where supported, PostgreSQL modifies its process name to reflect what each process is doing:

$ ps auxww | grep ^postgres
postgres  15551  0.0  0.1  57536  7132 pts/0    S    18:02   0:00 postgres -i
postgres  15554  0.0  0.0  57536  1184 ?        Ss   18:02   0:00 postgres: background writer
postgres  15555  0.0  0.0  57536   916 ?        Ss   18:02   0:00 postgres: checkpointer
postgres  15556  0.0  0.0  57536   916 ?        Ss   18:02   0:00 postgres: walwriter
postgres  15557  0.0  0.0  58504  2244 ?        Ss   18:02   0:00 postgres: autovacuum launcher
postgres  15582  0.0  0.0  58772  3080 ?        Ss   18:04   0:00 postgres: joe runbug 127.0.0.1 idle
postgres  15606  0.0  0.0  58772  3052 ?        Ss   18:07   0:00 postgres: tgl regression [local] SELECT waiting
postgres  15610  0.0  0.0  58772  3056 ?        Ss   18:07   0:00 postgres: tgl regression [local] idle in transaction

If one of those processes starts consuming too many resources - it's much easier to see why. If it's one of the system processes (eg "checkpointer"), then you know PostgreSQL is doing something intensive. If it's one of the bottom 3 processes - they correspond to a user connection, and you can see what kind of query they're running (eg SELECT). Better still, you can correlate the process id to pg_stat_activity and see exactly what the query was and how long it was running.

All that extra context from just 4 extra words.

#Implementation

If you're building software, setting the process name is likely incredibly simple and just a package install away. Python has setproctitle.setproctitle, Rust has proctitle::set_title and NodeJS has process.title.

If the framework you're using already sets a basic process name, don't override it. Many web servers will include a worker number in their name, which is useful for correlating logs. Instead, make sure your changes are additive.

jake      380639  1.2  2.2 359504 86768 ?        Sl   Jun30  57:59 macau worker-1
jake      385293  0.0  2.6 360532 102984 ?       Sl   Jun30   0:09 macau worker-1
jake     1096750  1.1  6.3 1403756 248600 ?      Sl   08:42   0:53 website worker-1
jake     1096753  1.1  6.7 1366768 261548 ?      Sl   08:42   0:51 website worker-1

You want to set your process name as early in your work as possible, so it's correct for as long as possible. Once your work is done, reset the process name so you're not attributing work incorrectly.

Setting the process name is a fairly cheap operation. If you're worried about it impacting performance, you don't need to worry:

In [2]: %timeit setproctitle.setproctitle("test")
195 ns ± 2.26 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

Taking Django as an example (although the same is probably true of many frameworks), it's best to do this in middleware. Middleware will run both before and after a request, allowing you to set the process name, and revert it once you're done.

class ProcessNameMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response
        self.initial_process_name = setproctitle.getproctitle()
    def __call__(self, request):
        try:
            setproctitle.setproctitle(f"{self.initial_process_name} path={request.path}")
            return self.get_response(request)
        finally:
            setproctitle.setproctitle(self.initial_process_name)

Now, when a request starts, the process name of the worker will update to include the path being processed. Once the request has finished, the name is reset.

#What about async?

And now, 5 minutes in, let's talk about where this all falls apart: Asynchronous programming. Setting the process name makes a fairly large assumption about your process: That it's only doing one thing at once. In the case of a multi-threaded application, that's fine - you can usually set the name of a thread much like you can a process. However, when using an event loop, such as Python's asyncio, Rust's tokio (and alike), or basically anything written in Javascript or Go, it's completely inappropriate.

The benefit of an event loop is that rather than waiting for your system to talk to storage, the network, or anything else, why not do something else in that time. This means a single process may be handling hundreds of processes concurrently, switching between them many times per second. For throughput and performance, this is usually a good thing. For using the process name as an observability channel - not so much. If this is the case, you're stuck with the existing tools you're (hopefully) already using.

#Review

Putting data in the process name doesn't replace conventional observability. Logs, metrics, and traces all have their place, and aren't replaced nor going anywhere. Process names are most useful on single systems, for live or low-latency debugging of a single server.

If your traces and logs ship instantly, this may not be that useful for you. You can already see exactly what's going on in your system immediately, with the most up-to-date information. But if there's a buffer, there's a delay, and that delays the information you have to respond to.

If the more data you have, the more you can do with it, why not just set the process name. Worst case, it's a little extra code to run. Best case, it's vital information in an incident you might be very grateful you have.