Recent content on James Carr
Recently I helped a team where several services all needed to read documents that another service wrote to an S3 bucket. We wanted to keep the object store, the bucket layout, and the IAM permissions independent from the consumers, so the owning team could change any of them without asking anyone else’s permission. The first pass was the obvious one: a shared library that wrapped the details…
I have been a fan of Oban in the Elixir ecosystem for a while now. It is one of those libraries that does exactly what it says and gets out of your way. So when the team announced an official Python port earlier this year, I wanted to take it for a spin and see how the experience translates. The pitch is simple: PostgreSQL-backed job processing with no message broker. No Redis, no RabbitMQ, no…
I’ve been thinking a lot about how to actually run AI agents in production lately. Not the model selection part or the prompt engineering… the simple infrastructure question of how does this thing get hosted and triggered? Recently I have been neck deep into exploring this, as well as some of the emerging frameworks for hosting and deploying agents. When you strip away the hype, an…
Something has been bugging me about how we think about API traffic. We spend a lot of time building dashboards around DAU, WAU, MAU, all built on the assumption that a human is on the other end of each session. But increasingly, the thing calling your API is an agent. Not a person clicking buttons, but a program running an LLM that decided it needed data from your service. This isn’t…
When AI agents fail mid-execution, they often lose their entire context and any work completed up to that point. API rate limits, network timeouts, and infrastructure failures can turn a sophisticated multi-step agent into an expensive waste of tokens. What if your agent could survive these failures and resume exactly where it left off? This post walks through how I built a multi-model AI…
In my previous posts, we covered the Saga pattern for distributed transactions and Scatter-Gather for parallel queries. Now let’s tackle the Process Manager pattern, a state machine that can respond to external events in real-time. The Pattern A Process Manager maintains state across a long-running process and can react to external signals. Unlike a simple workflow that runs start-to-finish,…
In my previous post on Temporal, we explored the Saga pattern for distributed transactions with compensating actions. Now let’s look at another classic from Enterprise Integration Patterns: Scatter-Gather. The Pattern When you need to query multiple sources and aggregate results, the Scatter-Gather pattern is your friend. Send requests to multiple services in parallel (scatter), wait for…
After spending time with enterprise integration patterns, one thing keeps coming up: distributed transactions are hard. When your process spans multiple services, each with its own database, you can’t just wrap everything in a BEGIN/COMMIT. What happens when step 3 of 5 fails? The first two services already committed their changes. You’re stuck in an inconsistent state. The Saga…
I first encountered NATS a couple years back while experimenting with OpenFaaS. If you’re not familiar, OpenFaaS is a serverless functions framework that uses NATS as its default message queue for async function invocations. At the time, I noted NATS as “that fast pub/sub thing” and moved on. It seemed like a lightweight messaging broker, but not something I needed to dig into.…
In my previous post , I explored AsyncAPI as a way to document event-driven services. But documentation sitting in YAML files only gets you so far. The real power comes when you can answer questions like: “Which services consume the OrderCreated event?” “What happens if I change this Avro schema?” “Who owns the service that publishes to this Kafka topic?” This…
Something that has been on my mind lately is the lack of visibility around messaging architectures. For example, if you’ve built REST APIs, you likely have OpenAPI specs that define your endpoints, schemas, and operations in YAML, and you get documentation, client generation, and contract testing for free. It’s become table stakes for HTTP APIs. But what about event-driven systems?…
You’ve built a microservice that saves orders to a database and publishes events to Kafka. Everything works great—until Kafka goes down for five minutes. Now you have orders in your database but no events published. Your downstream services never learn about these orders. You have inconsistent state. This is the dual-write problem, and the Transactional Outbox pattern solves it elegantly.…
In Day 10, we built a Process Manager with a live visualization that subscribed to events and updated the UI in real time. That visualization was a Wire Tap: passively observing messages without affecting the primary flow. Today we explore Wire Tap in depth and pair it with Control Bus for active management of distributed services. We’ll see how both patterns work in Kafka using Elixir. Wire…
Over the past eleven posts, we’ve explored Enterprise Integration Patterns from the ground up: building message channels, routing messages with Content-Based Routers and Recipient Lists, transforming data, managing workflows with Process Managers, and observing systems with Wire Taps. We wrote a lot of code. While I lost track of my original intent to build an open source audit logging…
In Day 10, we explored the Process Manager pattern for tracking workflow state over time. But what happens when things go wrong? How do you undo distributed work that’s already been committed? Today we dive into the Saga pattern: distributed transactions with compensation. Saga: When “Undo” Actually Works This pattern kept coming up in my research, and I’ll admit: I…
Apologies for the long delay between posts, the new year kicked off extremely busy! Previously in Day 9, we explored the Routing Slip pattern, where messages carry their own itinerary. Each node was designed to route the message to the next destination based on the content of the routing slip and we had a small discussion about whether or not this was a form of choreography or orchestration since…
After building a webhook delivery platform in Day 8, I found myself asking questions that led me down a rabbit hole. After looking at Recipient List, I realized I overlooked “Routing Slip” and, really, what the heck is a “routing slip” anyway? While I had planned to cover Process Manager and Saga, digging into Routing Slip gave me a really interesting insight into how one…
Happy New Years for my APAC friends! In my previous post we traced the evolution from synchronous request-reply to enterprise event delivery platforms. We built recipient lists, webhook registrations, and bundled integrations. But we glossed over a critical question: what happens when delivery fails? In production, failure isn’t exceptional. It’s constant. Endpoints go down. Networks…
Before computers, we had switchboards. An operator would receive your call, ask who you wanted to reach, physically connect your line to theirs with a patch cable, and you’d talk. When you hung up, the connection was torn down. Request-reply in its purest form: synchronous, blocking, one conversation at a time. The telephone system evolved. Direct dialing replaced operators. Trunk lines…
In Day 7 we built a webhook delivery platform with retry logic, dead letter queues, and circuit breakers. But we glossed over a critical question: what happens when one customer’s misbehaving endpoint starts affecting everyone else? Today we’re diving into the Bulkhead pattern: isolating workloads so failures in one area don’t cascade into others. We’ll apply it to webhook…
Message Translation, Content Enricher, Content Filter, Canonical Data Model - these patterns don’t get as much attention as the flashier routing patterns, but they’re where the real work happens. Every integration I’ve built has needed at least one of them. When systems talk to each other, they rarely agree on data formats. Your e-commerce platform calls it a…
Today we’re answering the question, “Should I name my event OrderPlaced or PlaceOrder?” We’re going to be getting more precise about something we’ve been pretty casual about: what exactly is a message? We’ve been tossing events around without much ceremony. But not all messages are alike. Some carry data. Some request actions. Some announce facts. Understanding…
Previously on Advent of Enterprise Integration Patterns: in Day 3, we explored message channels: Point-to-Point, Publish-Subscribe, Datatype, and Invalid Message channels. We built them first in pure code, then mapped them to RabbitMQ and Kafka as examples. Today we’re stepping away from Chronicle to focus purely on patterns. Chronicle needs some housekeeping (the boring-but-essential work…
Welcome back to the Advent of Enterprise Integration Patterns! In Day 2, we introduced Message Endpoints and the Messaging Gateway pattern to decouple our application from the transport layer, plus Pipes and Filters for composable event processing. We ended with a bit of a cliffhanger: running producers and consumers as separate processes worked great for distribution, but left us with fragmented…
Welcome back to the Advent of Enterprise Integration Patterns! Yesterday we got our hands dirty with Point-to-Point Channels and Competing Consumers, building the bones of Chronicle using OTP actors. We can send audit events and have them processed reliably by multiple consumers racing to grab work. Pretty cool! But here’s the thing: that implementation is tightly coupled to OTP. What…
Welcome to Day 1 of the Advent of Enterprise Integration Patterns! Today we tackle the foundational question every distributed system must answer: how do systems share information? Can someone just call me!? Chapter 2 of Enterprise Integration Patterns lays out four fundamental integration styles. Understanding these options and their trade-offs is essential before diving into the patterns…
Since returning to Zapier, I’ve been building the features enterprise customers crave for managing automation at scale. Event-driven architectures, message routing, guaranteed delivery, audit trails. The stuff that enterprise buyers expect as table stakes. It’s reminded me how much this foundational knowledge matters. One thing keeps coming up: audit logging. Every enterprise customer…
My kids have been deep into a 90s-movie phase lately — which is surreal because these were my high-school movies and now apparently they’re “vintage.” So we’ve been watching The Truman Show, Dark City, and The Matrix together. The themes in these movies always felt familiar — identity, perception, reality, control. None of that is new. But watching them again today, after decades of living…
Previously on Elixir Basics, we explored using GenServer to set up multiple workers that would print a message out at a random interval. Today, I’m going to expand on it by adding AMQP to the mix, having our workers publish messages on an interval and add a solitary consumer process to consume the messages. Overview If you recall from last time, we had a simple setup of an application, a…
A common code sample I like to write when learning a new language is to have multiple threads or processes “do something” on an interval. For example, publishing to Kafka or RabbitMQ, invoking a REST API, etc., as a way of simulating a multi-process worker system. As a starting point, I like to build these to simply output a string to the console and then build my way up from there.
Something that I noticed as I have begun building out applications in Phoenix is the complete lack of any kind of “Admin View.” Most frameworks like this don’t come with one included, but I guess I have just been spoiled by Django’s inclusion of these by default. I searched around and found several do exist and will maybe spend some time looking at them in the future, but…
While working on a new project recently, I needed to allow application users to enter a dynamic list of key-value pairs. Not knowing better, I reached for what I knew and built out a React component to dynamically add and remove inputs while also inserting into the websocket to sync the changeset with the server. Then on the server-side, I had to write some additional handlers to process those…
As I’ve started down the Indie Hacker road, one thing that has come up as I work on various projects is I need something to manage multiple versions of different binaries. I had used nvm and pyenv in the past (and rvm before that) so I started down the path of looking for the same type of tool in the elixir world. Thankfully, this led me down the path of more generic solutions to versioning…
I just got back from a month-long break in Cambodia, traveling with my family. I took advantage of the wide-open space with zero work expectations to give me some time to truly reflect on my career so far and what comes next. I’ve had a pretty successful career in tech spanning twenty years that, for the most part, paid off quite well. It’s afforded me a life that I never would have…
Previously I wrote about setting out on my solo journey across Germany and how I prepared. Today I’m going to break down everything about my time in Berlin: places to visit and tons of photos to share. Along the way, I hope to share a few tips and tricks to make your visit enjoyable. Arrival In Berlin After a 12 hour flight I landed in Berlin at 8:20am. Thanks to the sleeping gummies I took, I was…
10 years ago, as I was getting ready to take kids somewhere I heard them both laughing wildly in the backseat. I snapped this photo as quick as I could and to me, it just completely captures the companionship these two siblings have shared since the very beginning. The big grins on both of their faces is something I always hope to see every single day.
I recently returned from a week-long solo adventure in Germany, a trip that came about quite unexpectedly. Earlier in 2023, in my eagerness to reengage with speaking at conferences, I submitted a series of talk proposals to various events and one of my submissions was accepted at RabbitMQ Summit. However, it was only after this exciting acceptance that I realized the conference was in Berlin, not…
“Hey really quick, can you put together some cost forecasts for the coming year for our AWS and GCP costs?” At some point in your career, you may receive this kind of request from your manager, VP of Engineering, CTO, CFO, or CEO. You might explore Cost Explorer, put together some fancy graphs and breakdowns, and send them over, thinking you’re done. Not even close. This request will…
Reflecting on my professional journey and the various teams I’ve been a part of, I recently picked up a book that resonated with my experiences and offered some invaluable insights into building a truly cohesive and high-performing team. The book, “The Ideal Team Player” by Patrick Lencioni, delves into the characteristics that make someone an ideal team player and provides…
Looking back on the years of my life this morning, I was thinking it would be an interesting exercise to review my work experience from earlier in my life, before college graduation and before the true start of my professional career, and examine one thing I am grateful for from each experience. I think it is important to look back to humble beginnings and the lessons learned through hard work and…
Recently, I’ve been using Notion for a variety of purposes, including tracking my workouts. While there are many fitness tracking templates available, some of which are quite complex, I chose to rely on existing popular apps like Hevy, Strava, and Concept2 Logbook for detailed tracking, and use Notion as a central place for logging my activities. Initially, I was tracking my workouts…
For over a decade, I have enjoyed listening to podcasts. I subscribe to many and listen to them during long drives, while working out at the gym, or doing yard work. However, one downside is that I often hear many great ideas that I don’t capture in the moment. Taking my phone out and fiddling with a notes app normally detracts from my current engagement. Nine times out of ten, I won’t…
Let’s chat. Tell me about your project. Let’s create something incredible together.💡 Feel free to ✉️ email me or 📆 book a time for us to chat! Upcoming Speaking Engagements I’ve been on sabatical for 2024, new talks for 2025 coming soon! Past Speaking Engagements RabbitMQ Summit Supermanagers Episode Datadog Dash
I build systems that make other systems talk to each other. Twenty years in, that’s still the part I find most interesting: not the services themselves, but the seams between them, and everything that has to go right for a message to survive the trip. Two things drive the work. The first is efficiency. Distributed systems burn a startling share of their budget on coordination alone: latency,…
👋 I am so glad you are here at $COMPANY and I am looking forward to us working together. I like to provide this manager readme ahead of time as kind of an operating manual so you can understand how I operate and what to expect. This isn’t exhaustive, but hopefully we’ll fill in the gaps as we work together. My Values Be Transparent - Unless there is a need for confidentiality I am…
Life is a journey with endless possibilities, each one presenting an opportunity to grow and evolve. However, without intentionality, these opportunities can easily slip away, leaving us feeling unfulfilled and stuck. In today’s fast-paced world, it’s easy to get caught up in the hustle and bustle of everyday life, leaving little time for reflection and intentionality. Being intentional with…
While taking my career break I have been doing some reading around organization structures and building high performing teams. One of the books that came up in my list was Dynamic Reteaming by Heidi Helfand. I picked this book up because in my career I had seen a several examples of how team structures evolve and change over time and figured it would be great to really dig into some of the…
Yep, that’s me over in the top right, with my kiddos meeting the legendary John Romero at Strange Loop 2022.
Recently a member of the Rands Leadership slack went through and took some very detailed notes of the #engineering-effectiveness channel dating all the way back to 2017 to the present day and shared their notes with the rest of the group. I was floored by how much good information was in the document and plan to go over it in detail, but here are some of the key points I highlighted when it came…
In any organization, the role of a manager is crucial to ensure that the team performs to its full potential. As teams grow larger and more complex, managing them effectively becomes increasingly challenging. At a certain point, you need to subdivide the team into several teams and make the decision to either promote or hire additional managers. This is where a director takes on the role of a…