Lorin Hochstein's ramblings about software, complex systems, and incidents.
On August 6, 2026, GitHub had a pretty rough incident: GitHub Actions was degraded for about nine hours. GitHub posted a public incident write-up. It’s only a few paragraphs long, but there are some interesting details in here. This was yet another incident that involved saturation. In fact, the write-up even uses the word saturated … Continue reading GitHub has another tough day →
As a fan of resilience engineering, I often differ with people on where we should focus our scarce engineering cycles in order to improve reliability. I thought it would be a useful exercise to brainstorm some of the differences in focus between what I’ll call the traditional view of reliability, and the resilience engineering view. … Continue reading Traditional versus resilience engineering…
There’s a new software reliability conference that just spun up called Software Should Work, and I had a chance to give a talk there. I took the opportunity to speak about one of my favorite topics: saturation. Here’s a recording of it. Links from the talk
Over the years, mathematicians, logicians and computer scientists have developed various calculi. If you have a background in computer science, you’ve likely heard of the lambda calculus, a model of computation that was developed by Alonzo Church. If databases are more your thing, then you’ve been exposed to the relational calculus without even knowing it, … Continue reading Synthesis is harder…
The other day, Reginald Braithwaite posted the following toot. For posterity, I’ve also included my own response to it: Braithwaite’s post is dripping with sarcasm, but make no mistake, incident reports written entirely by LLMs is coming. And I am not looking forward to this future. Before I dive in here, I want to note … Continue reading I am dreading our LLM-written incident report future →
The Journal of System and Software publishes a regular column called Dear Researchers: The perspective of software practitioners. Each column is an open letter to the software engineering research community from someone who works in tech. It’s edited by Austin Henley and Olaf Zimmermann, both of whom have experience in the two worlds of academia … Continue reading Dear researchers column →
I can’t bring myself to read text if I believe it to be AI-generated. Now, I ask LLMs questions all of the time, and I do read those answers. I frequently use tools like ChatGPT and Claude as replacements for Google for answering specific questions; that’s not what I’m talking about here. I’m also not … Continue reading I can’t bear to read AI-generated prose →
Modern software systems contain within them a mind-boggling level of complexity. As software engineers, we make this complexity manageable through techniques like decomposition, information hiding, and abstraction. We endeavor to break our systems up into components that interact over well-defined interfaces. By doing this, the surface exposed to individual software engineers is dramatically…
At this point, you have no doubt heard about GitHub’s availability woes over the past several months. In April, Mitchell Hashimoto (of Hashicorp fame) wrote a post about how he is moving his Ghostty project off of GitHub: …for the past month I’ve kept a journal where I put an “X” next to every date … Continue reading Form may follow function, but use doesn’t follow design →
We cut middle managers across the organization because AI allows us to have more direct reports per manager while still measuring and mentoring our teams effectively. – Matthew Prince, How I Choose Which Cloudflare Employees to Replace With AI My PhD research involved studying programmer productivity for a very specific domain: high-performance computing, more commonly … Continue reading The…