Error handling in Rust is a complex topic. And it is not the good kind of inherent complexity, which requires a sophisticated but also elegant solution. Rather, it is complex because there is no right solution. It is also annoying and ugly most of the time. Developers familiar with Rust may know the 2-3 ways of error handling, depending on your definition. There is the Result -based handling, with…
Ich beschäftige mich schon seit sehr langer Zeit mit dem Thema Finanzielle Freiheit . Investieren und mein Portfolio managen ist schon so etwas wie ein Hobby für mich. Da ich derzeit wegen sowohl privaten als auch beruflichen Gründen in den Burnout rutsche habe ich mir die Frage gestellt: Wie finanziell frei bin ich eigentlich? Kann ich wirklich jetzt schon in Rente gehen, und einfach aus meinen…
In my last blogpost about State Machine Replication , I discovered the interesting property that you can think of a state machine replication (SMR) protocol as a state machine itself. One that receives messages from the network, and sends messages to other replicas, along with handling some timeout logic. I mentioned that this lends itself very well to the sans-io pattern, so I want to explore and…
About two months ago, I was joking with colleagues that I wanted to build service X, which should be a fault tolerant, highly available service. But I didn’t want to build the replication needed for it. Because it is hard to do, and there is a ton of anecdotes on the internet that prove as much, and that it is really hard to create safe and correct replication. But as luck would have it, maybe I…
I have been working on a usecase recently where I wanted to do some streaming uploads an downloads, both on the client- and the server-side. And boy do I need to tell you, its way too confusing and complicated than it probably needs to be. The normal async ecosystem in Rust works quite well without too much hassle. But dealing with streams is a completely different story. As I see it, there are…
Last week, I have written about the Dataloader pattern commonly used within GraphQL servers to avoid 1 + N problems that can pop up with a naive implementation of GraphQL resolvers. You can read up on last weeks post here . Last week, we established that the Dataloader pattern fundamentally takes advantage of async concurrency , and we used yield_now to split the loading of our data into distinct…
It has been a few years that I have last touched GraphQL , but I am digging into it once more these recent weeks. For those that have never heard of GraphQL , it is a standard of how to query arbitrary data from a server. In contrast to REST , and its OpenAPI specification , it allows more freedom of which data to query. In REST/OpenAPI , you have a fixed set of routes to request specific data.…
The short answer is: yes . And the main reason is to make updates intentional and avoid surprises . For a long time, I have been chasing the very latest Rust version (I still am though), and I have been copy-pasting rustup toolchain install stable around my CI workflows. By now, I have changed my opinion on this, and I would recommend to pin the Rust toolchain to a specific version using a…
What do I mean by this clickbait-y title? # The good As Rust developers are most likely well aware of, the serde ecosystem is one of the strong points of Rust. You define your data types just like usual, and with a single line of code, you can add support to turn them into JSON or any other kind of serialization format, or parse them from such a format. #[derive(Serialize, Deserialize)] is a…
I have had a couple discussions around the topic of testing philosophy and what a good test is recently, and I want to dive deeper into this. # Background Let me give a bit of background first. I have experienced quite a bit of the software engineering landscape over the course of my career. I went through a bunch of languages through that time, and through frontend, backend, libraries and…
Today I went down quite a rabbit hole. I was playing around with native array support in SQLite. So what exactly does this mean, and why should anyone care? Lets start this off by looking at SQL, which is a somewhat standardized definition and manipulation language for relational databases. Similar to other foundational technologies in the IT sphere, SQL is quite ancient, as it was designed in the…
Investing and managing my finances is one of my hobbies. And this year, markets seem to be pretty much in bubble territory due to AI. So I am riding the waves of stock prices, and I am trying to rebalance my holdings and cashing in on some profits. But I want to also maximize those profits as much as I can, and to do that I am trying something new: Swing trading , which means selling high…
I have already blogged about the surprisingly slow Rust thread_local! s , and today I ran into another related problem. And as I often do, I went right down a rabbit hole digging into it. To give a bit of context, I am currently working on a metrics crate. One of my main goals is to make it as high performance and as low overhead as possible. Another goal is to make it simple to use. That means…
Recently I was taking a look at benchmarking various approaches to small string optimization, and you can read up on that here . As a small recap, I want to turn a &[dyn Display] into a Vec<String> , or similar type, as efficiently as possible. Even before writing that post, I have forked the smol_str crate and rewrote its internals around a hand crafted binary layout, using way too much unsafe…
Sometimes I really like to go down a rabbit hole and obsess about some tiny details. Today is such a day, and I am benchmarking, profiling and trying to micro-optimize a piece of Rust code, specifically looking at small string optimization. My use-case is as following: I have a slice of &dyn Display , and I want to stringify and capture those values for later. To have some comparable test-cases,…
One of the things I’m trying to drive recently at Sentry is introducing some form as strictly typed RPC. To this end, I have experimented using both gRPC and Cap'n Proto within a small isolated service I am trying to introduce. All of these experiments are open source, and you can look at the code in this PR . # Why? As things stand right now, Sentry does not have a single well established…
These last couple of weeks, I have been obsessing over “the cost of observability”, specifically metrics. This whole topic is quite big and rather something for a conference, literally, as I submitted it as a talk for RustFest this year ;-) But on this topic, I was experimenting with doing thread-local aggregation of metrics, and doing a ton of profiling trying to micro-optimize the heck out of…
It’s been a while that I wrote a proper rant, but today is the day. The Rust project is thoroughly tracking the performance of the compiler. For this, there is the Rust compiler performance test suite which includes a number of widely used crates in a fixed version, so that its easier to compare different compiler versions on the same piece of code. I would actually love to see the opposite.…
This week, I have been profiling and measuring the overhead of the Sentry Rust SDK, as another team has reported a large overhead in their testing. So much so that the team shied away from using it more extensively in combination with #[tracing::instrument] . After some profiling, I identified a potential culprit, which was using very high quality randomness in the form of the getrandom crate,…
This weekend is the start of my week of chill at home and do nothing vacation. Which, for a passionate software engineer, is the perfect time to do some open source work and dive into interesting topics outside of work. The topic I will be looking at is optimizing code generation of Rust enums. This deep dive is motivated by a real world issue in the rust minidump(_common) crate. The…
# TLDR My phone was draining unreasonable amounts of battery lately. Shelling into the phone via adb shell after enabling USB debugging, and doing a simple top revealed that, of all things, com.sonymobile.launcher , aka the Home Screen / Launcher was constantly running at 200% CPU (saturating 2 cores), and using up to 10% of memory. I switched to a different Launcher which so far looks to have…
Scope guards in Rust are awesome! Apart from explaining why, I also want to explore one specific side of them that I have never read about directly: their effect on compile times. # A small RA(II)nt Let me start todays exploration with a bit of a rant. What I am talking about today, and what I refer to as “scope guards” is often called the RAII pattern. That stands for “resource acquisition is…
An alternative clickbait title for this could be: “ Read + Seek considered dangerous”. This is a very interesting story, and one of the nice side effects of working on open source software is that I can share all of the details of it publicly, along with a link to the PR that implemented the fix. # TLDR As the alternative clickbait title suggests, the core of the problem is that both Read and Seek…
Interestingly, the same theme has come up multiple times recently within Sentry. I myself recently wrote a Sentry RFC about SourceMap DebugId s. And at the same time, I was supporting and advising other teams working on Java Source Context, and Flutter Obfuscation. All these different initiatives have the following in common: You have multiple build artifacts for a single application build. These…
I have recently discovered that Rust Futures, or rather, async fn calls can lead to surprising performance problems if they are nested too deeply. I learned that the hard way by triggering a stack overflow in a PR to symbolicator . I then tracked that down to unreasonably huge (as measured by mem::size_of_val ) futures. The problem was also being exacerbated by the deeply nested async fn calls in…
I started writing a series a blog posts explaining various debug formats, specifically formats that allow you to recover the original source locations. I wrote about SourceMaps and Portable PDB Sequence Points already. Now it is time to look at DWARF line programs. # DWARF, the specification The whole DWARF specification is available over at dwarfstd.org . It is a gigantic PDF file with >450 pages…
It is the end of the year, and a lot of people are writing end-of-year posts, and outlooks for the new year. So here is mine. # Sentry It was an interesting year for sure. We shipped a bunch of stuff. Some internal that otherwise noone would know about and others external features adding support for new platforms and ecosystems. A big internal item was switching our serverside processor from…
Last week I was looking at the implementation details of async . Specifically I was looking at two issues that make stack traces in async programs confusing and hard to make sense of. To recap, lets take this snippet of Rust code: pub async fn fn_with_nested_block () -> Backtrace { None . unwrap_or_else (|| async { Backtrace :: force_capture() }) . await } When we run it with your favorite async…
I have been looking at a lot of Rust async stack traces lately. This was mostly related to profiling some heavily async code locally, as well as profiling some production systems in the cloud test driving Sentrys new profiling support for Rust. We don’t need to go all that big and fancy, we can observe the problem already with a tiny example. Now that Backtrace is finally stable, we can capture…
Before going on a slight rant about why rustdoc / doctests are broken, I first want to highlight that rustdoc / doctests are amazing !!! I believe that great documentation and great tooling is a major contributor to Rusts success. And one part of that is rustdoc, and doctests. The fact that you can write documentation and examples, and have those at the same time be part of your testsuite is an…
In my opinion, one of the most important, but also very underappreciated skills is to think outside the box. Or put differently, to challenge the status quo and re-think some deeply rooted thought patterns. We have always done things this way … is the worst of arguments to do things a certain way. I recently stumbled upon a quote thats being attributed to Elon Musk (who himself replied to the…
I have recently investigated a very interesting performance problem in Sentrys symbolication infrastructure. We got reports of an increasing number of out-of-memory situations of our infrastructure. This started rather randomly, and was not correlated to any deploys. I had the hunch that it might be related to some new form of data that customers were throwing at us. And indeed, after some time,…
Following up my last post about SourceMaps, this one here is about Portable PDB Sequence Points. Only took me about a month to procrastinate ;-) # Sequence Points, abstractly Similar to SourceMaps and other debug formats, the sequence points allow mapping from IL offsets to source information. The Portable PDB Format is specified in a markdown document here and is complementary to the main…
In my last post I committed to the idea of doing a deep dive series into a couple of debug formats, or more specifically, how their line mappings / line programs work. To start things off, we will be learning how the SourceMap mappings work. # SourceMaps, abstractly For people not familiar with the matter, SourceMaps are a building block used in the JavaScript ecosystem. They are used to map from…
If you want to parse binary formats in Rust, you have a few crates to chose from apart from rolling your own. Some popular contenders are zerocopy and scroll . I would like to take this chance to explain the difference between the two, which one you likely want to use in which situation, and why zerocopy truely is magical. However, neither is perfect, there is some papercuts and ideas for…
Before going into the details of my recent discovery, lets define the term Ossification , as likely a lot of people have never heard that word before. Imagine you have an extensible format or protocol. As example, we can take a list of elements of different type. The list is extensible. It can have an arbitrary number of elements, and over time the different types can also be extended. This is…
Snapshot testing is quite popular. And unfortunately most of the time it is the wrong tool for the job. In software projects, testing is extremely important. A piece of software can only be as good as its testsuite, at least when it is changing over time. # Tests, abstractly So what is the purpose of tests in the first place? Well, you test that for certain inputs, your program returns some…
I wanted to write about this topic quite some time ago, but it seems my tendency to procrastinate won in the end. Until today, so lets get started. The topic at hand are fallible Iterators, and this post is motivated by a real world problem that I fixed both at the producer end in wasmparser and the consumer in symbolic . When parsing some binary data files, for example wasm as in the example…
Today I want to talk about the need for better tools to work with self-referential structs, and also (safe) alternatives that alleviate that need for certain use-cases. Lets start by giving a very concrete example of what I want to achieve. I want to get the nth line of a string quickly . The idiomatic way to do it would be string.lines().nth(nth) , and it can hardly get any simpler than that. The…
Both at work, and also personally, I do think about efficient parsers and data formats a lot. Some time ago, I also wrote an article about writing a custom binary format and associated parser. That exercise started something like this: # [ repr (C)] struct Header { version : u32 , num_a : u32 , num_b : u32 , } pub struct Format < 'data > { buf : & 'data [ u8 ], header : & 'data Header, } impl <…
As an engineer by heart, I sometimes think of social engineering challenges. Or rather, how to approach some social problems with out of the box thinking. This time, I want to reflect of some ideas on how to solve the problem an unbalanced work week creates. # Problem Statement To understand what I mean by that, lets first describe the current unsatisfactory state. I think we have a resource…
Now that I got your attention with the clickbait title, let me explain what I mean by it. Current Rust code could be broken in very subtle ways because of some assumptions we have about async Rust code that might not always be true. This story starts with a bug I recently fixed in sentry-rust (which manifested itself as a memory leak), and itself highlights both sides of the problem. The root…
I realize I might be a bit late to the party, considering tmandrys blog post is already a month old, and there is now even an initiative focused on contexts. Alas, I want to explore the way in which implicit contexts could solve some very real problems that we have with todays methods of modeling context. As practical examples, I want to highlight the issues using the sentry-rust , and tracing…
With the recent talk about “Contexts” in the Rust community, and some other thoughts I had recently, I want to explore in a bit more detail what the difference between Futures and Tasks is in Rust. The difference between Futures and Tasks is like the difference between concurrency and parallelism. The difference is quite subtle, even considering just the words. I don’t even know if my native…
Update : I updated the code examples now that GATs have been stabilized. One of the fundamental selling points of Rust is zero-cost abstractions. This means that you can write high-level generic code, and the compiler will optimize it in such a way that you couldn’t have written better code by hand. There are tons of examples of Rust doing this. But I came with a very specific example in mind, and…
# Don’t do it Well first off the bat, I want to repeat something that I read some time ago: Don’t create your own bespoke binary format! Use JSON! I agree with this. Having dealt with binary data quite recently, and seeing how some of the formats are neither well documented, or some of the writers actually creating invalid data, I can truly say this is really hard to get right! And if you want to…
A quick note first: The "howto" does not mean I present a solution here, but rather that I am searching for a good solution. With that out of the way, let me explain what I want to do. I would like to have an algorithm that always succeeds and produces some kind of result. And I want to simultaneously output any kind of error that happens while running that algorithm. I can quickly think about two…
There is an interesting TED-talk called The mathematics of weight loss which explains with chemical formulas and maths how bodyfat will be turned into \( CO_2 \) and Water. The presenter starts with the following chemical formula: \[ C_{55}H_{104}O_6 + 78O_2 \rarr 55CO_2 + 52H_2O \] And then goes on to calculate that 84% of of the fat is being exhaled as \( CO_2 \). But that is not really what I…
I was recently investigating a customer issue where I got stuck at the point where I simply couldn’t seem to find any kind of unwind information for a specific piece of code, while the rest of the code had such information. My conclusion was that it could potentially be a problem in the customers build chain. I found a similar problem looking at a truncated stack trace on Android. To better…
Interestingly, in recent times I have read and watched a few articles/videos about first principles thinking. It might be a bit related to confirmation bias, since recently I have been thinking about this topic myself, in the context of bad standards. Or rather, standards that we now adhere to merely for the reason that it is a standard, and we have always done it that way . Anyhow, my train of…