Building reliable infrastructure for GoCardless, a fast-growing fintech based in London. Focused on tackling infrastructure problems with a software engineering mindset.
Every weekday morning, someone on the AI team at incident.io kicks off our spend report skill. Claude pulls a dozen BigQuery queries, computes a 7-day baseline range for the KPIs we track, hunts for anomalies (runaway processors, mis-tagged spend, onboarding spikes) and posts a structured report into our #ai-costs-pulse channel. The skill exists because AI inference costs are growing fast and…
Last year I wrote this from an empty office on New Year’s Eve. This year I’m in the boardroom between Christmas and New Year, back from a Pizza Union lunch, waiting for mulled wine to heat up in a Moccamaster because we’re not allowed a saucepan in the kitchen. There’s a handful of engineers half-working, half-chatting in the background. It’s a quiet week. This is markedly better than last year.…
We’ve just launched an AI Engineer role, and Pete’s written a post explaining why we’ve created the role which goes nicely with our blog series “Building with AI” that the team released the other week. With all the noise in the industry right now, you might think a role like ‘AI Engineering’ is for SEO or an attempt to ride the hype wave. That’s not why we did it, though: we created it because the…
I’m currently building an AI-powered incident investigation product that automatically debugs and triages production issues when incidents are declared. We’re one of many new entrants in the “Agentic SRE” space (yuck). Working here has given me a front-row seat to the competitive dynamics of the AI ecosystem, which I’m increasingly seeing as the innovator’s dilemma writ-large. The dilemma…
“If you want to build AI products, you need to use Python.” I hear this a lot, and it’s wrong. I get why people think it- software engineers have been trained to lean into open-source tooling, and the AI ecosystem is undoubtably strongest in Python. But if you don’t already use Python, switching languages for the ecosystem is likely a bad call. Not only is that ecosystem less mature than it seems…
Most companies building AI products today are stuck in the experimental phase, lacking the basic tooling and infrastructure needed to build reliable systems. It’s easy to spot once you know what to look for. Take the current trend of companies talking about load-balancing their AI operations across multiple providers. Sounds sophisticated, right? But it reveals something concerning about the state…
I’m determined to make a tradition of this, which is why I find myself in an empty office on New Years Eve trying to remember what’s gone by these last twelve months. I ended “Looking back at 2023” saying “brace for impact” which I had thought meant I was prepared for this year, but in retrospect I wasn’t even close. It’s a common story in start-ups to say “thank god that’s over, next year will be…
It was the week before pandemic lockdowns began. Like many companies, we were thinking about what a fully remote workforce might mean for us. In an attempt to get ahead of things, we scheduled a ‘Game Day’ to test our work-from-home incident response capabilities. You know, just in case we’d need to abandon the office (spoiler: we would). The timing was perfect. We had several new team members and…
Of the mental models and rules I use in my life, by far the most useful is to learn only one thing at any given time. If you get it right, you: Ensure work is the right balance of challenging for it to be enjoyable. Avoid stalling due to an overload of uncertainty. Learn better, and faster. This rule works for individuals but is also useful for coaching others, debugging team performance, or…
Last year I wrote “Looking back at 2022” and found it valuable to reflect and celebrate a year of work, even if just as a reminder of the things that happened before going into the next. So with another year having passed, time to try again! This time for my second full year working at incident.io. To contextualise the year, some milestones ( see company post ) are: 34 to 77 employees Opened our…
Recently, I’ve needed to add concurrency protections to several HTTP APIs. One endpoint was for updating a schedule, where incident.io customers would tell us about who was on-call and what hours they worked so we could generate a pay report. It would suck if another user was modifying the same schedule – maybe just to add a single user – and pressed save, just to remove all your hard work.…
Most people are familiar with state machines and know their value. The average state machine library can help you model states, prevent invalid transitions, and produce diagrams that help even non-technical people understand how the code behaves. This article isn’t about making the case for state machines. It’s about how you take the concept of a state machine and have it work alongside your…
From the moment you learn programming people tell you “don’t repeat yourself!” So what I’m about to suggest might sound odd. But I’m here to say that if you want to ship high-quality software at pace, you should be investing in abstractions that are designed to enable copy-and-paste. When you look at most software teams, they work within an existing application and extend it with new features.…
For the last three months I’ve been working on a team building the incident.io Catalog: read the announcement post here . One of the things I enjoy about working at incident is we often build products I’ve been thinking about or actively using for years. The catalog may be the best example of this, having spent years at GoCardless building and looking after our homegrown service registry, and…
When you first create a status page, it’s probably because you want to communicate outages to your customers. The faster you can share details about an outage, the sooner your customers know what’s going on, and the more effectively they can handle the outage. Communicating promptly – in clear language – builds trust. And as a young company with a customer centric focus, that’s your top priority.…