If It Has to Happen, Don’t Put It in the Prompt
When agent instructions turn into all caps rules, the fix is often to move the requirement out of the prompt and into a workflow that can check it.
Skylar Payne
Building AI that actually works. Evaluation-driven development for engineering teams.
Latest posts
TIL: Annotate AI Traces
Logs tell you what happened. Annotations tell you what it meant, why it failed, and whether the fix helped.
TIL: Break AI Workflows Into Parts You Can Grade
One giant prompt can hide five separate jobs. Split the work so each part has a smaller contract and a failure you can actually name.
TIL: Evaluate RAG Retrieval Separately
A bad RAG answer does not tell you whether retrieval failed, generation failed, or the product asked an impossible question. Split the blame before fixing anything.
TIL: Instrument AI Calls Before You Debug
If an AI answer goes sideways and you cannot see the prompt, model, latency, tokens, retrieved context, and failure path, you are debugging from vibes.
TIL: Make AI Pipelines Safe to Replay
If every eval run emails a customer, updates production state, or fires a webhook, you do not have an eval harness. You have a hostage situation.
TIL: Model Agent Workflows as State Graphs
Agents get less spooky when they have named states, constrained transitions, and a record of how each decision moved the process forward.
TIL: Quality Check RAG Chunks
Before blaming the model, inspect the chunks. Duplicate, empty, bloated, or low-signal chunks can wreck retrieval quietly.
TIL: Record and Replay AI Workflows
When a multi-step AI run fails once and then refuses to fail again, replay beats superstition. Capture the calls, context, and intermediate state.
TIL: Put Approval Before Risky Agent Tools
Agents should not get to delete files, send messages, spend money, publish content, or mutate production just because the next step looks obvious.
TIL: Validate RAG Citations
A citation is not proof just because the model printed a source name. Verify that the source exists and actually supports the claim.
TIL: Structure LLM Outputs at the Boundary
If the rest of your app needs data, make the model return data. Do not make downstream code scrape nice-sounding paragraphs forever.
TIL: Make AI Features Boring to Change
AI features get scary when prompts, logs, evals, schemas, fallbacks, and product code all live in the same pile. Give the weird part one stable interface so changes have a place to go.
GPT-5.4 in OpenClaw doesn’t suck. Your prompts do.
Anthropic changed OpenClaw billing. We ran evals, tuned the bootstrap files, and GPT-5.4 got a lot better.
If DSPy is So Great, Why Isn't Anyone Using It?
Any sufficiently complicated AI system contains an ad hoc, informally-specified, bug-ridden implementation of half of DSPy.
Evals That Actually Get Used
A streamlined system for AI evaluation that closes the gap between seeing problems and fixing them.
Bringing Data Science Back to AI Engineering
A Practical Way to Evaluate Multi-Step and Multi-Turn Agents
Patterns for AI Reliability
Artificial Intelligence for Business
A Strategic Guide to Success
The Answer is Always Evals
40% Better, 75% Faster
How Frigade Slashed Latency & Boosted User Helpfulness
Why RAG Is (Still) Not Dead
The Enduring Value of Retrieval in the Era of Expanding Context Windows
The Art of Interface Design: Making Good APIs that Scale
Quality Assurance for AI
Bootstrapping AI Systems with Synthetic Data: 4 Approaches
5 Strategies for Improving Latency in AI Applications
AI Observability is Just Observability
Why Most Companies Fail to Build Strategic Assets with AI
An AI Maturity Model
The Art of Iterative AI System Development
A Practical Guide to Evaluation-Driven Improvement
Building AI Products That Actually Work: A Hard-Won Guide
The Real Secret to Landing Your Dream Tech Job
It''s Not What You Think
Building an Automated Email Reply Agent with Pydantic AI
A Story of Simplification
Finding True North
How LinkedIn''s Candidate Recommendation Project Changed My View of Data-Driven Development
What Effective Leaders Get Right
Building an Experimentation Mindset in AI Teams
A Leader''s Guide
Why Your Team is Slow
The Hidden Cost of Missing Business Context
The Hidden Bottleneck in Knowledge Work
Why Working Harder Isn''t Working
From AI Ambitions to Action: A Mental Health Startup
How HealthRhythms built their mood score system