Everyone is talking about “return on tokens.” Vendors love it (as long as the news is good). Companies that had long since given up on any sort of value architecture or on understanding the ROI of product work are suddenly chomping at the bit. Which makes sense: they haven’t been able to lay off enough people to cover their token budgets while still hitting their promises to the board.
It is hard to know what sits below the surface: a genuine effort to understand how AI might augment human capabilities, or some version of “how many people can we safely fire?” Or both?
It is as if the whole world suddenly shifted from a buffet model to buying each dumpling one at a time, and now everyone is freaking out about the ROI of each dumpling. But we’ve been here before. We’ve been here with hours. We’ve been here with capacity. And the problems have always been roughly the same:
Spending more time on the “I” side than the “R” side of ROI.
Myopically choosing shorter-term, easier-to-attribute use cases for the “R” side.
Gravitating toward whatever is easiest to measure. And tokens are very easy to measure.
This post takes a trip through hours and flow metrics, detours into an alternative way of thinking about ROI, and eventually gets to TOKENS—everyone’s favorite dumpling.
My goal is to show that the underlying measurement problems are remarkably universal. AI introduces some genuinely new twists, but in the end, it still hinges on a theory of value.
With these comparisons, ask yourself, “How might this relate to the current return on tokens question?”
Hours, as commonly used to understand investment, represent a major construct validity problem. There’s nothing inherently wrong with measuring “time we spent on things” (hours) to understand better where effort is going, provided you make it clear that:
Time is not a fungible thing that can be infinitely allocated/re-allocated. This holds across skill sets, team context, and even across a “normal day”. For example, the “magic hour” of uninterrupted morning productivity can be vastly more productive than trying to wrap your head around something complex between 4 and 5 PM. The developer deep in the onboarding space, an expert in onboarding analytics and the user journey, working primarily with what customers see/interact with, can’t be immediately swapped into a deep, legacy backend refactoring effort.
Where you allocate time tells you nothing about the “quality”, efficiency, or efficacy of that expenditure of time. The highly skilled developer reactively called in to debug an area of the code they have no experience with will not have the same leverage as someone more junior, but more contextually situated. Ten people spending ten hours each is not equal to one person spending one hundred hours. Sometimes the entire system is constrained by a single specialist, a single decision, a single dependency, or a single review. Spending time elsewhere has negative leverage.
The context switching tax is real. Most efforts to understand the allocation of time, whether explicit or implicit, leave out large time investments. Imagine alternating between two tasks every ten minutes for a six-hour day. You’ll spend three hours (at least) on context switching and calibrating/orienting around the new task. Coordination is multiplicative, while time accounting is additive.
The relationship between time and outcomes is non-linear. 2x-ing where you spend time doesn’t 2x the outcomes. You might bang your head against a problem for eight hours and get nowhere, and the ninth hour “unlocks” all the value. The metric treats the input as continuous even though the production function contains thresholds and fixed setup costs.
Where we invest focus can have a long-term effect. Imagine spending 5% longer on every enhancement to pay off a bit of debt and “garden” the codebase. Three years later, a version of the team—some people have stayed, but some have left—is absolutely flying. Where do you “book” that time? And how do you connect that investment to the long-evolving, game-changing “return”? Some hours multiply future options. Some hours limit future options.
Time allocation schemes often force allocations into individual buckets, but there are typically causal relationships between those buckets. They aren’t mutually exclusive. One impacts the other.
Helpful, when focused.
Lean/flow metrics include cycle time, lead time, practical WIP limits, and throughput (e.g., N stories per week). Most of these metrics have a storied history in manufacturing, which is both a positive—there’s actual math, theory, and practice behind them—and a negative when misapplied to software development.
Note the following:
In manufacturing, it matters what you are producing. Things that have sufficiently different work processes, value profiles, etc. are always divided out. This is sometimes referred to as “classes of service,” and it is a critical component of using these metrics effectively. Without that segmentation, changes in throughput, cost, or efficiency may simply reflect changes in the mix of what was produced rather than actual changes in performance.
A lot of product work is more akin to R&D, experimentation, and initial “design” than to manufacturing and mass production. In this setting, “waste” isn’t an inherently bad thing. If you try five things and one works, then you’ve purchased the knowledge for that fifth thing.
What constitutes a work unit is also mushy. Say you have a more open-ended stream of experiments meant to move a metric; what do you measure? Experiments per week? The lead time of the whole stream? Both? Where do you put the opportunities that get discovery work, but ultimately don’t get greenlit?
When we talk about “capacity” in manufacturing, you are talking about the sustainable ability to meet a particular type (or types) of demand. “This factory can produce N of this variety of shoes every month.” You invest in a factory that can meet this forecasted demand, with optionality to scale up the factory based on new demand. So, going back to some of the issues with hours, capacity is not (always) fungible and is not something that is “spent” at the time of use. It is an emergent quality of the factory’s design.
The boundaries of the system matter enormously. “Lead time” from when? Customer request? Commitment? Ticket creation? First code change? And when is something done: merged, deployed, adopted, or producing value? Moving the boundary can dramatically change the metric without changing the work.
You have to be ready for the uncomfortable reality that “work” spends a majority of its time not being “worked on”, as well as facing all of your bad habits when it comes to keeping people busy.
Many of these metrics describe a relatively stable flow system that remains stable over a sufficiently long observation period. If the system is rapidly changing and has a lot of variability in work types, you can be lulled into a sense of apparent precision.
Don’t get me started on story points. Any effort to use story points for anything beyond a disposable tool to spark discussion, right-size, or reality-check a near-term cycle commitment is a dereliction of duty. We have decades of evidence to support this.
The main thing with manufacturing is that it is essentially a demand-based pull system. The things being produced have a price, and there is a clear distinction between “sitting on the lot” and “in the customer’s hands.” That makes the economics an order of magnitude easier to reason about. Theoretically, we could imagine features going unused as inventory, unfinished work sitting in queues as WIP, or capabilities built ahead of actual demand as overproduction.
But the manufacturing use case makes much more explicit assumptions about the “value exchange.” There is demand for a particular thing. The thing has an economic value. And value is realized when it reaches the customer.
“But how about throughput accounting!?”
Say you have a very stable work stream, with a lot of historical throughput data to draw from, with a very tight line to value, with relatively homogeneous units of work and stable demand characteristics; you might be close to saying something like, “We allocated N% of our capacity for that demand this quarter” using these metrics. This would more closely resemble throughput accounting.
The caveat is that those conditions are not necessarily ideal in product work for various reasons: 1) you want to do work that might have longer-term leverage, 2) you might want variety, 3) you might want to shift things to meet emerging demand, and 4) some of the most valuable work may deliberately change the system rather than simply flow through it. The more dynamic, exploratory, and leverage-seeking the work becomes, the less comfortable you should be treating historical throughput as a stable measure of allocatable capacity.
All to say that certain shapes of work are a better/worse fit, and you have to be aware of the differences.
To be clear, I think a lot of these Kanban-style/manufacturing metrics, along with PR-level metrics like cycle time, review latency, and merge frequency, can be very useful for understanding the health of the system, especially when combined with classes of service/work. I consider all of these metrics/ideas “required learning” for product developers. But their primary use is for continuous improvement: helping a team optimize around a particular type of demand, understand where waste is accumulating, identify bottlenecks, and improve flow.
We will get into tokens and AI in a sec, but first an alternative to points and flow. The only way I’ve seen the ROI discussion work reasonably well is to keep the cost side deliberately broad and apply much more rigor to the causal model linking that investment to value.
Treat the team as the cost unit. Start with something like, “This team costs roughly $1.5M per year.” Don’t pretend you can meaningfully decompose that investment into hours, story points, or individual features.
Connect the team to a causal model. Each team should be linked to at least one model that connects actionable inputs and leading indicators to mid- and long-term sources of differentiated growth.
Clarify the demand the team is expected to serve. Be explicit about the types, sources, and variability of demand. This gives you some basis for judging whether the team’s capacity and demand are reasonably matched. If they aren’t, that becomes a separate question, or at least something you should call out explicitly.
Fund durable lanes of focus when possible. Invest in enduring areas of responsibility or opportunity rather than constantly funding one-off projects, features, or stories.
Maintain a business case for the team. Document the causal assumptions, validity of the proxies, expected returns, major uncertainties, and what would cause you to change the investment. The team business case should transcend projects. You know you’re doing it wrong when the whole rationale for a team was a project finished years ago.
Be explicit about the business/product lifecycle. An emerging idea, a scaling product, and a margin optimizer/commodity should not be evaluated using the same expectations or metrics.
Judge return through movement in validated proxies. Look for evidence that the things you believe lead to differentiated economic value are actually moving, rather than trying to attribute revenue to individual units of work. Always ask yourself, “Are we more or less confident about this value hypothesis?”
Use operational metrics for their intended purpose. Use flow metrics for continuous improvement. Use hours to understand time leakage and broad patterns of effort. But don’t use either as the proxy for investment itself.
I still can’t get over the fact that companies rationalize $1m+ “teams”, but then choose to bury their value under layers and layers of pyramids, programs, points, etc.
For more on this approach, some topics of interest might be causal inference, econometrics, measurement theory, structural causal models, proxy validity, Bayesian updating, leading indicators, and real-options thinking.
Imagine for a moment that someone in finance greenlights investing $50,000,000 in AI spending (measured as input and output tokens). Let’s first compare that to an equal investment in “human capital”. Consider all the gotchas and open questions:
Organizational capital / positive externalities. Employees don’t only produce assigned outputs. They also create knowledge, relationships, routines, and institutional memory that benefit other work. This matters much more when AI is augmenting people than when it is simply replacing a task.
Fixed vs. variable cost structure. Salaried labor behaves more like a committed/fixed cost over the relevant planning horizon; token consumption behaves much more like a variable cost. That distinction becomes especially important as inference gets cheaper and more commodity-like.
Capital formation / residual value. What asset remains after the expenditure? Human work can increase human and organizational capital. AI consumption may leave useful code, data, workflows, or intellectual property, but only if the organization actually captures it. Otherwise, much of the spend is simply consumed.
Marginal versus average returns. The first $1M in AI spend may be extraordinarily productive because it targets obvious, high-value use cases. The fiftieth million may be going toward increasingly marginal applications. This gets even trickier if higher-priced models are being purchased for differentiated capability.
Neither human capital nor AI spend produces value in isolation. Their returns depend heavily on the complementary assets around them. In augmentation scenarios especially, the relevant system is often human + AI, not AI by itself.
Both suffer from the long-term costs of overproduction. Producing more than the system can absorb creates inventory, maintenance, coordination, review, and cognitive overhead. Cheap generation can simply move the bottleneck downstream.
A cheaper/faster build is not necessarily a cheaper asset. In software, most economic consequences occur after the initial act of production. A compelling substitution case at build time may still have poor lifetime economics.
Are they comparable?
A lot of these points hint at the same two tensions. Will AI ultimately augment or replace humans? And will it provide a commodity input that people add value to, or will it be a premium capability? Or both?
What is the dominant mental model at your company? How did it get that way?
We have an unusually precise measure of one input sitting inside an unusually uncertain economic system (tokens). We can know exactly how many tokens were consumed and what they cost. Tokens give you a beautiful meter and invoice, which can create a much stronger illusion that you have a denominator suitable for ROI.
But we are still reasoning through much fuzzier questions: Augment or Replace? Commodity or Differentiated Capability? Or are we paying for differentiated capability? What complementary assets does it require? Does it create durable organizational capability? Does it increase maintenance costs? Move bottlenecks? Encourage overproduction? Change the economics of the work over its lifetime?
A serious AI business case is less “What’s the ROI?” and more “What theory of the system are we betting on?”
We believe using AI for __________ will reduce __________ without materially increasing __________.
We believe AI will primarily augment/replace __________, resulting in __________.
We believe spending __________ on AI will be economically preferable to __________ because __________.
We believe making __________ cheaper/faster will not simply move the bottleneck to __________.
We believe increased AI-enabled production of __________ can be absorbed without creating excessive __________.
We believe the primary long-term asset created by this AI investment will be __________, rather than merely cheaper production.
We believe the human + AI system will outperform __________ on __________, after accounting for __________.
We believe the downstream costs of AI-generated __________ will remain below __________ over its useful lifetime.
We believe paying a premium for __________ will produce enough additional __________ to justify the premium.
We believe this investment works economically only if __________ remains true and __________ does not become the limiting constraint.
So rather than:
We will spend $10M on AI and generate $30M of value.
the intellectually defensible version is closer to:
We believe this AI spend will change this production system in these specific ways, under these assumptions, with these expected benefits, costs, externalities, and downstream effects.
Then ROI is almost the output of the hypothesis, not the hypothesis itself. This is extremely important given that everyone is learning at the moment. No one has figured this out.
I’m struck by one thing. Imagine for a moment that we understood team funding as a combination of salaries and AI spend. A team might look at itself and say, “Geez, we cost $1.5M a year in salaries plus $1M in tokens. What is our business case here?”
I understand that this is somewhat utopian. In many companies, the goal is explicitly to use AI to eliminate some portion of that team, its managers, or the surrounding organization, based on some substitution math.
But if you think about the problem at the level of the team + AI system, the solution is remarkably similar to the “alternative” described above. What demand are we serving? What causal model connects our work to differentiated value? What durable areas are we investing in? What assumptions have to hold? What evidence would make us increase, decrease, or redirect the investment?
In fact, this kind of business case may be a prerequisite for giving teams meaningful responsibility for AI spend. If you want teams to make intelligent tradeoffs between people, models, tools, and other forms of capacity, they need a model of the investment they are responsible for and the value they are trying to create.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.