95% Cheaper, 75% Faster: Tuning an Agent with Pairwise Evals
Pointwise vs. Pairwise LLM Agent evals and how I used a custom eval tool to speed up my agent by 4x and cut costs by 95%.
erik.wiffin
Recent content on erik.wiffin
Latest posts
You are Failing at AI Because You Haven’t Built a Learning Organization
LLMs have broken the traditional software playbook: demos are misleading, expertise is shifting too fast to hire for, and the conditions surrounding AI adoption are the worst possible for building the learning organization you actually need.
Your AI Agent is a Distributed System
Mid-way through building an agentic application, I got stuck. I found myself needing to make it work, make it right, and make it fast, all at once, and progress was slow.
Reframing it as a distributed system helped me recognize where my problems were coming from and got me going again.
Building Chart Chat with Plan and Execute
Observability, tool design, and model selection are harder than the orchestration loop. Lessons from building an agentic charting tool with Plan and Execute.
Vibecoding Audit
I spent the holidays building an event platform with an AI “vibecoding” tool to see if the results were actually production-ready. While the speed of development was incredible, my audit reveals the critical gaps—from database safety to broken SEO—that still need a human touch before you hit publish.
Patient No-Show Prediction: Improving Operating Room Utilization in a Major Academic Medical Center
Executive Summary 
 A major academic medical center faced challenges with high patient no-show rates for scheduled surgical procedures, leading to suboptimal utilization of limited operating room capacity and lost revenue opportunities. To address this problem, I was contracted to develop a machine learning model to predict the likelihood of patient no-shows. By integrating this predictive…
AI Strategy & Implementation
I provide AI strategy and implementation services. If you think I can help you, contact me at erik.wiffin@gmail.com . 
 Services 
 
 AI Strategy: I will work with you to help you define your company’s AI Strategy. 
 Zero to One: Proof of Concept Implementation: I will develop a proof of concept to validate a product idea or lay a foundation for your existing engineering team.…
Proof of Thought
With large language models capable of generating convincing documents without real thought, this insightful article explores the critical need for new “Proofs of Thought” to ensure genuine intellectual engagement and prevent workplace processes from becoming mere facades.
Evergreen Links
A collection of blog posts and resources that I’ve found useful and frequently refer back to. 
 
 Death by PowerPoint: the slide that killed seven people 
 How to Socialize Big Changes at Work 
 A Rubric for Evaluating Team Members’ Contributions to an Inclusive Culture 
 The Ultimate Burnout Guide: Symptoms, Causes, and Prevention 
 The Boy Scout Rule ~Robert C. Martin…
What to do when someone tells you to "take initiative"
I have always hated the phrase “Don’t bring me problems, bring me solutions!”. It feels lazy. It incentivizes people to sweep things under the rug if they can’t figure out what to do. It doesn’t give any useful direction to the person bringing you problems, because if they had solutions, they probably would have brought them already! At its best, it’s unhelpful, at its worst, it’s actively…
Limiting Work In Progress as a Manager
The demands on a manager’s time are endless and sometimes it feels like you’re being pulled in every direction at once. These demands can make it hard to focus, they make it hard to move from reacting to problems to anticipating them. In this article, I’d like to talk about how I’ve limited demands on my time, counterintuitively doing more with less. 
 The story 
…
How to get the most out of your 1:1s
As a Director of Engineering, I have monthly 1:1s with all of my direct reports. A 1:1 (one-on-one) is a recurring meeting with no set agenda between a manager and one of their reports. The internet is full of valuable insight into how to run them from my perspective, (ex. The Update, The Vent, and The Disaster ) but somewhat more limited in advice on how to make the most of them if you’re on the…
Stripe For Outpatient Billing
In this article, I will explain why I think Stripe should start treating health insurance cards like any other 3.3”x2.1” pieces of plastic with 12 digits on them and take their slice of a $4.1 trillion industry. 
 The situation 
 Outpatient billing is expensive and unpleasant. 
 According to a study published in 2017 , doctors spend 21.8% of gross receipts on it. That means over 20…
Data Engineering for Other People
Or: Why is this software engineer being so difficult? 
 So you’ve got some data. In my case, I’ve been keeping track of how successful my taco truck has been. 
 
 
 
 
 There’s nothing crazy here: a date column, how many of each kind of taco I sold, and where I parked my taco truck that day.
At this point, I’d like to create some sweet…
Design Plans: A Reference
When confronted with a complicated task, the first response of any developer is to dive in and start coding. This is precisely the wrong thing to do . I might sound hyperbolic, but it’s true. Every single design plan I have ever seen has resulted in a better understanding of the problem, and more importantly, a better solution delivered faster. 
 What is a design plan? 
 A design plan is a…
Story Points As Effort + Uncertainty
The only way to perfectly estimate how long a task will take, is to do the task and measure how long it took. In most situations where you’re endeavoring to produce an estimate, this is not a practical way of providing it. At the other end of the spectrum, you can pick a random number and base your timeline on that. While this is a quick and easy way of coming up with an estimate, it is rarely…
The Missing Manual - Professionalism Among Agency Developers
What does it mean to be a professional software developer at an agency? 
 If you’re familiar with some of the big names in books about software design - books like Design Patterns, Code Complete, Domain-Driven Design - you may have noticed that they all share a particular assumption: you are working at a product company, on a single codebase, and there is a good chance you will be working on…
AI Systems Audit

 
 
 
 
 
 Two-week engagement · $5,000
 
 AI Systems Audit 
 
 A focused, two-week review of your AI stack. Find where you're
 overspending on tokens, where new agents get stuck on their way to
 production, and what happens after they ship.
 
 
 Book a free 30-minute call 
 
 
 
 
 
 
 
 
 
 
…
Fractional Tech Lead

 
 
 
 
 Fractional Tech Lead 
 
 Technical leadership without bringing on a full time employee.
 
 
 Book a free 30-minute call 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 Who you are 
 
 
 
 You're an early or growth-stage startup that needs real technical
…