RSS Amplifier

DHK On Data and AI · Jul 23, 2026

Evaluate What You’re Building, Not What You’re Paying

0
Sign in to vote or save

Dave Holmes-Kinsella (DHK) · DHK On Data and AI

Example of wingman output, based on synthetic data

Everybody is becoming more sensitive to the cost of AI. “Token-maxing” has crept into our language—first as a hack, and now as a complaint.

That concern is reasonable. But it seems to me that the more powerful question isn’t how much we spent. It’s what the spending produced.

I can tell you, to within a few dollars, what I spent on Claude Code last week. What I can’t tell you is how much of that went toward building a feature, understanding an unfamiliar codebase, debugging a difficult problem, writing documentation, or investigating an idea that turned out not to be useful.

Billing data also can’t tell me whether that work left me, my team, or the underlying system in a better state.

That’s the gap I think is worth exploring.

Over the last few weeks, I’ve been building a tool called Work Ledger. It looks across Claude Code sessions, reconstructs the work being attempted, connects that work to its cost, and begins to identify the patterns underneath it.

The first question is familiar: Where did the money go?

The more useful questions help us decide what to do next:

  • What kinds of work keep appearing?

  • Where is AI creating genuine leverage?

  • Where am I repeatedly asking it to solve the same class of problem?

  • Which interactions should become reusable prompts or shared practices?

  • Which should be replaced by a deterministic tool that’s faster, cheaper, and more reliable?

The purpose isn’t simply to reduce AI usage. It’s to increase the quality and impact of the work.

Token-maxing rests on the assumption that using more AI is evidence of progress. Work Ledger starts somewhere else:

Where and how are we creating value?

That leads to the larger question informing the project:

For an individual, “good” might mean developing repeatable ways of working instead of beginning every session from scratch.

For a team, it might mean recognizing successful patterns, sharing them, and turning them into common tools and practices.

For an organization, it might eventually mean using that evidence to redesign workflows, remove unnecessary steps, and build new capabilities around what AI makes possible.

The progression I’m interested in isn’t simply from higher cost to lower cost. It’s from observation to understanding; from understanding to shared practice; from shared practice to better tooling; and eventually to changes in how the organization works.

Work Ledger begins with observation.

The encouraging news is that the thing I built is working. It’s already showing me patterns in my own work that I couldn’t see from billing data, transcript history, or intuition alone.

This article introduces the project. In the next few pieces, I’ll describe what Work Ledger is finding, how it reconstructs work, what it would take to become a complete product, and how tools like this might support the larger transition from individual AI use to shared practices and organizational change.

I’m still working through the implications, and I’d value feedback from people thinking about AI effectiveness, engineering practices, process redesign, organizational learning, or the difficult question underneath all of them:

How do we determine whether AI is actually making the work better?

More to come. I hope you’ll follow along.

In the meantime, a couple of links:

Read the original on dhkondata.substack.com

Comments

Nothing yet. Say the first thing.

    Sign in to join the conversation.