Recent content on HarrisonSec — Long-form writing on agent runtimes, distributed systems, and security
Here is a failure that survives every reasonable thing you can check. One app, on an iPhone, on cellular data, talking to a machine on your own VPN. The app pairs. It fetches the host’s info. It loads the list of things you can launch. Then the one connection that actually matters, the streaming session, hangs for ten seconds and times out. Switch the phone to Wi-Fi and everything works.…
An agent is asked to send a payment confirmation email. It calls the send_email tool. The tool returns {"status": "queued", "message_id": "msg_abc123"} . The agent writes back to the user: Done. Confirmation email sent. Two hours later a support ticket arrives. The customer never got the email. The provider’s queue had dropped the message during a regional failover, and no one on the path…
A team I was helping watched their model bill jump to several times its usual size in a single month. The token meter had not predicted it. The first question in the room was the one almost everyone asks: did the provider raise prices, or should we move to a cheaper model? Both were the wrong question, and they were wrong in an instructive way. The same engineers, if their AWS egress bill had…
An agent I was reviewing was asked to rename a function across a repository. It ran for a while, made its edits, and returned a clean summary: Done. Renamed parseUser to parseAccount everywhere. The summary was confident, well formatted, and wrong. Six of eight call sites were updated. Two were not, one buried in a string that built the name dynamically and one in a test helper the search had…
There is one failure mode I have learned to fear more than a missing check. A system with no check for something is at least honest about it. The gap is visible, the uncertainty is real, and everyone downstream knows not to lean on that part. A system with a wrong check for the same thing is worse, and it is worse in a specific, dangerous way. It answers with confidence. It overrules the signals…
I have now written this same sentence three times in three pieces, so it is time to write the sentence underneath it. The first companion piece, Generative AI Builds Shapes, Not Games , argued that generative AI produces plausible shapes and that correctness has to come from structure and verification wrapped around the generator. That answers whether you need something outside the model. The…
I keep writing variations of the same sentence. Agent memory has to terminate at a source of truth. An agent loop has to terminate at a check. Generative 3D has to terminate at a verifier. The probabilistic part proposes, a deterministic part disposes. Four pieces, one shape. That is usually a sign there is a law underneath, not four coincidences. This is my attempt to write the law down, and then…
I sat down to benchmark a tool and ended up with a map of a wall. Higgsfield shipped a Minecraft “prompt-to-build” feature: type a prompt, get a structure in-world about a minute later. I ran eight building prompts through it, scored each one, and walked through the results. The point started as “how good is this tool.” It ended somewhere more useful, because the shape of…
Higgsfield shipped a Minecraft “prompt-to-build” feature: a mod that drops a “Supercomputer” block into your world, takes a free-text prompt, and generates a structure in-world a minute later. I spent one session putting real building prompts through it to see what it actually does, not what the landing page says it does. Eight prompts, fixed screenshots, an in-world…
In April 2026, Anthropic published a blog post called “The advisor strategy: Give agents an intelligence boost” , naming a pattern they had been A/B-testing in production: a cheaper model runs the agent loop end-to-end, an expensive model is consulted only when the cheap one hits a decision it can’t solve. They reported concrete numbers — Haiku + Opus advisor on BrowseComp at…
The prior post in this series, Agent Retrieval Is a Cost Curve Problem , argued that a viable LLM-symbol-graph would need to satisfy six specific conditions — and that no existing tool had hit all six. The post went live on 2026-05-25; seven days earlier, CodeGraph had hit GitHub trending with exactly those six properties satisfied. That’s the easy version of the update: framework predicted…
Two weeks ago CodeGraph hit GitHub trending — tree-sitter + SQLite/FTS5 + MCP for Claude Code, 19k+ stars in a week. The team published a benchmark on 7 repos showing 35% cheaper, 57% fewer tokens, 46% faster, 71% fewer tool calls vs. baseline. Those are big numbers. They’re also numbers from a benchmark designed by the team that built the tool, on repos they chose. Designer bias is the #1…
This post is one half of a pair. The other half — Agent Retrieval Is a Cost Curve Problem — argues that Claude Code’s within-session code retrieval avoids RAG because the cost curve says it should. This piece argues something parallel about cross-session memory : the lossy auto-capture systems being marketed as “AI memory” are, in classical distributed-systems vocabulary, caches…
There’s a popular interview question making the rounds: “Why doesn’t Claude Code use RAG to retrieve code? Why grep?” The popular answer goes: chunking breaks code structure, vectors approximate when code demands exact, indexes go stale, cold-start is slow, retrieval is a black box. All five are real. None of them are the reason. They’re symptoms. The reason is older…
It’s 3am. The Kafka consumer pod that’s been running cleanly for six weeks gets OOM-killed. Kubernetes restarts it. Five minutes later: OOM-killed again. Restart. OOM-killed a third time. By the fourth restart I’ve shelved the dashboard and started reading runtime/chan.go . The code that died fit on one line: events := make( chan Event ) I want to tell you that line is the bug.…
The Setup I was optimizing an AI agent for a production system — a creator agent that handles user requests like “make this character fiercer” or “rename this entity.” The agent runs a 5-layer pipeline: Perceive → Cognate → Decide → Act → Express, with real LLM calls at each step. Quality was bad. Not “it doesn’t work” bad — “it works 40% of the…
The Problem Nobody Talks About AI can write code, generate content, analyze data, design systems, and manage projects. It’s getting better every month. The natural question: what’s left for humans? The wrong answer: “AI will replace us.” The other wrong answer: “AI is just a tool, nothing changes.” The right answer is uncomfortable: stop picking the best AI. Run…
Most discussions of TypeScript/Node vs Go concurrency stop at the surface: Node is async, Go is threaded. That framing isn’t wrong — it just isn’t deep enough to be useful when you’re picking a runtime, debugging a tail-latency problem, or explaining to your team why one of the services keeps falling over under CPU load. The real difference is not async vs threaded. It’s a…
The model is roughly 10% of what makes an AI agent work in production. The other 90% — context engineering, memory, validation, tool-call reliability — is where every team’s agents quietly stop working at scale. This video walks through the four-layer failure model and where each kind of bug actually lives. Related Blog post: The AI Agent 90% Problem Companion long-form: The 90% AI Agent…
Episode Summary Building an AI agent that works is easy. Building one that keeps working is where most teams fail. This episode breaks down the hidden 90% of agent engineering: context management, memory, tool execution, state recovery, and loop closure. I use Claude Code as the reference point, compare it with more fragile agent designs, and show how production quality often comes from code…
Long-form podcast version of the 90% Problem thesis. Goes deeper than the 7-minute video on each of the four layers: context, memory, validation, tool-call reliability. If you’re scoping AI agent reliability work, this is the layered mental model the readiness review uses. Related Blog post: The AI Agent 90% Problem Short video: Why Your AI Agent Keeps Failing
Your Agent Works Great. Until It Doesn’t. You built an AI agent over the weekend. It calls tools, remembers context, follows instructions. You demo it to your team. Everyone’s impressed. Monday morning, a user types “rename Ember to Infernia.” Your agent loops 15 times, burns through your API budget, and returns a response that doesn’t contain the word…
You’re debugging a gnarly race condition. Claude Code has been going at it for 10 minutes — reading files, forming theories, running tests. Then it hits a wall. Same hypothesis, same failed fix, third attempt. What if you could call in a second brain — a completely different model with fresh eyes — without leaving your terminal? That’s what the Codex plugin for Claude Code does. It…
Every AI coding agent runs the same core pattern: send context to an LLM, get back text and tool calls, execute tools, feed results back, repeat. LLM talks, program walks. Claude Code’s implementation lives in query.ts — a 1,729-line async generator where the while(true) loop spans from line 307 to line 1728. That’s 1,421 lines of production state machine logic handling context…
This is Part 4 of our Claude Code Architecture Deep Dive series. Part 1: 5 Hidden Features | Part 2: The 1,421-Line While Loop | Part 3: Context Engineering — 5-Level Compression Pipeline This article replaces and deepens our earlier analysis, Claude Code’s Memory Is Simpler Than You Think . The original focused on limitations. This one focuses on why — the first-principles tradeoffs behind…
This is Part 3 of our Claude Code Architecture Deep Dive series. Part 1: 5 Hidden Features | Part 2: The 1,421-Line While Loop | Part 4: Memory Tradeoffs Why Context Engineering Is the Real Moat Every AI agent has the same fundamental constraint: a fixed-size context window. Claude’s is now up to 1M tokens. That sounds massive — until you realize a real coding session can easily generate…
This is Part 2 of our Claude Code Architecture Deep Dive series. Part 1: 5 Hidden Features | Part 3: Context Engineering | Part 4: Memory Tradeoffs Why Claude Code's Agent Loop Is 1,421 Lines
Adding AI API calls to an existing backend is where most teams’ observability and billing instincts break. The calls look similar to any other RPC — send a JSON request, receive a JSON response. The difference is what happens to the meter. An ordinary RPC costs you deterministic compute: a few milliseconds of CPU, a few KB of network. An LLM API call costs you between $0.0001 and $1.50…
Updated: This analysis has been superseded by Part 4: Why It Uses Markdown Files Instead of Vector DBs — a deeper first-principles tradeoff analysis. The original article below focused on limitations; Part 4 focuses on why those design choices were made. The Hype vs. The Source Code After Claude Code’s source leak , one of the most talked-about discoveries was Kairos — a “permanent…
This is Part 1 of our Claude Code Architecture Deep Dive series. Part 2: The 1,421-Line While Loop | Part 3: Context Engineering | Part 4: Memory Tradeoffs What Happened Anthropic shipped Claude Code v2.1.88 to npm with a 60MB source map still attached. That single file contained 1,906 source files and 510,000 lines of fully readable TypeScript. No minification. No obfuscation. Just the raw…
Same first-principles framework as the 15-minute video — LLM talks, program walks — but with deeper exploration of each layer and the seams between them. For listeners who want the audio-first format. Related Original 15-min video: The AI Stack Explained Even longer deep dive: The Complete AI Architecture (48 min) Blog post: LLM talks, program walks
The longest cut of the AI Stack material. Where the 15-minute video gives you the framework and the 22-minute podcast gives you the talking-points walkthrough, this version goes layer-by-layer through every concept — from how tokens stream out of the LLM to how a multi-agent orchestrator decides who acts next. For viewers who want the full mental model in one sitting. Related 15-min original: The…
Episode Summary This is the extended version of Episode 1. Same first-principles framework — LLM talks, program walks — but with deeper exploration of each layer, more examples, and discussion of real-world implications. If Episode 1 is the executive summary, this is the full technical report. What’s Different From Episode 1 Deeper exploration of tokenization and why it matters for costs…
There’s an impulse, when someone first learns about consistency models in distributed systems, to want to classify the taxonomy into neat drawers. Strong here. Eventual there. Linearizable above it. Read-your-writes below. Study the diagram, pass the interview. That taxonomy is real, but it’s not useful the way people think. Production systems don’t pick a consistency model and…
LLM, Token, Agent — They're All the Same Thing. (AI Stack Explained) Watch the full 15-minute video walkthrough with animations. LLM. Token. Context. Prompt. Function Calling. MCP. Agent. Skill.
Episode Summary LLM. Token. Context. Prompt. Function Calling. MCP. Agent. Skill — 8 concepts that confuse every engineer, until you realize they’re all the same thing. An LLM can only output text. It can’t browse the web, call APIs, or take any action. The program around it does everything else. LLM talks, program walks. That loop is how the entire AI world runs. What We Cover LLM — A…
LLM. Token. Context. Prompt. Function Calling. MCP. Agent. Skill — 8 concepts, 1 pattern. An LLM can only output text. It can’t browse the web, call APIs, or take any action. The program around it does everything else. LLM talks, program walks. That loop is how the entire AI world runs. This video peels back every layer of the AI stack from the bottom up, building a mental model that makes…
gRPC interceptors are the middleware pattern, specialized for gRPC. If you’ve written HTTP middleware before, the shape is familiar — a function that wraps a call, can observe or modify the request, pass to the next handler, then observe or modify the response. The difference: gRPC’s type system makes the flavors (unary, server-stream, client-stream, bidi) explicit, and chain ordering…
Go 1.18 shipped generics in March 2022. The two years before that were dominated by hopeful blog posts (“finally, a real type system!”) and the two years after by the predictable backlash (“why did we even bother, Go was simpler”). I’ve written production Go before and after. The honest answer is somewhere in the middle and closer to “useful for a narrower set…
Go’s performance culture has a ritual quality. “Use sync.Pool.” “Avoid interface boxing.” “Preallocate slices.” Copy-pasted from blog posts and applied without measurement. Sometimes helpful. Often hollow. The honest answer is that Go performance work is mostly just profiling . Good profiling tells you what’s actually slow. Bad profiling — or no…
sync.Pool is one of those Go features that shows up prominently in “how to write fast Go” blog posts and then gets applied to everything. The result is a codebase sprinkled with pools that don’t help and sometimes hurt. Most Go code I review does not need sync.Pool . The code that does need it often uses it wrong. This is a working engineer’s take on when pooling actually…
Episode Summary Fail fast is widely accepted as a best practice in software engineering. But in distributed systems, blindly failing fast during infrastructure transitions — like Redis Sentinel failover, NATS leader election, or Kafka partition rebalancing — can turn a 12-second self-healing event into a 12-minute outage. In this episode, we break down why this happens and walk through a concrete…
It’s 2 AM. PagerDuty fires. Redis master is down. Your application, trained to fail fast, dutifully fails — every single request, all at once. By the time Sentinel promotes a new master 12 seconds later, you’ve already generated 40,000 errors and three escalation calls. The system recovered on its own. Your application didn’t let it. This is the story of how “good…
A team I worked with once migrated an order-placement path from gRPC to NATS because “it’s decoupled and faster.” The old flow was simple: the web service called PlaceOrder via gRPC, got back an order ID, rendered success to the user. The new flow: web service publishes order.place to NATS, an order-service consumes it and processes asynchronously. Within three weeks they had…
The number of times I’ve watched a team pick a message system based on “Company X uses it” is depressing. Right behind it: the team that picks the one they already know, regardless of whether it fits the workload. NATS, Kafka, and MQTT get lumped together because they all pass messages between processes. That’s like lumping trucks, sedans, and motorbikes together because…
“A Docker container is basically a lightweight VM, right?” No. That sentence alone causes more architectural misunderstandings than any other in modern backend engineering. A VM virtualizes hardware. A container is a set of Linux kernel features — namespaces, cgroups, overlay filesystems — wrapped in a nicer CLI. Same host kernel, same memory space, same attack surface if the kernel…
I worked with a team that rewrote a critical service from Go to Rust because “performance.” Six months later, the service was 30% faster, the team was miserable, and feature velocity had dropped to a crawl. Meanwhile the competitor team, still on Go, had shipped four new features. We did the postmortem eventually. The service handled maybe 2,000 requests per second on a 4-core machine.…
I’ve reviewed enough Go backend test suites to notice a pattern. The services with the most unit tests are often the ones with the most production incidents. Not because unit tests cause incidents — because the teams writing unit tests and calling it a day weren’t testing the things that actually broke. Production bugs in distributed Go backends don’t usually look like…
A team I worked with tried to build their billing system on top of their tracing pipeline. The idea was clean: every operation already generates a span; spans already have duration and attributes; adding user_id and billable_units to each span lets finance query the trace store to compute invoices. One pipeline, less infrastructure. Beautiful. Six weeks before the first billing cycle, the wheels…
The bug was alive for three weeks. On a normal day it cost nothing. On the day it activated, it nearly took the service down. The pattern was simple. An HTTP handler had to fetch data from three downstream gRPC services and merge the results. The team had done the disciplined thing: set a 5-second deadline on the request context, propagate it all the way through to the handler, use errgroup for…