AI

I’ve written sparingly about AI on this Substack, mostly because the space is moving so fast that it’s hard to really settle on a “take.” But AI has become increasingly impactful to my own life and career that I think it’s a good time to share my current thoughts.

First, a point of clarification: when I refer to “AI” in this post, I’ll be referring to generative AI, powered by LLMs (large language models). This is the technology that, when coupled with incredible amounts of computing power and data, is behind the vast majority of the action and investment in “AI” these days. Second, my perspective: I have been a professional software engineer for over 15 years and have spent most of that time as a lead of some sort. My focus and forte has generally been more at the design and architecture level than implementation. This will color my experience.

I don’t think debates over “AGI” or “super-intelligence” are very helpful, because they immediately run into subjective arguments over what “intelligence” fundamentally is. What we know is that LLMs do not model the human brain, which has generally been tightly linked to our understanding of “intelligence,” and yet are able to mimic the output of human brains (in the form of language) well enough that to an observer the different mechanics underneath become obscured.

Unlike human brains, which manage to meet our definition of “intelligent” without consuming the sum total of nearly all extant human written output, LLMs fundamentally are probabilistic machines that produce output determined by patterns in their input data. Although there are various techniques (like reinforcement learning) that complicate this picture, I still think it’s largely true and useful to think of LLMs as giant averaging systems that produce output that is roughly the average of their input. And this serves as a useful shorthand to think about how and when to apply them.

If AI’s “knowledge” is essentially the average of its training data (which we can assume includes both true and high-quality as well as untrue or low-quality content), then we can assume it is generally going to be more knowledgeable than I am about subjects where I have below-average understanding, and less knowledgeable in subjects where I have above-average understanding.

It’s not this simple, of course, but this is a good starting point. It means that AI can be incredibly useful at producing well-tailored and competent advice and help in areas that I am not familiar. Suddenly tasks that I wouldn’t have attempted due to the cost of learning how to do them competently, feel achievable. And this covers a lot! This is one reason that I’m bullish on the ways AI can empower entrepreneurial individuals by being a cheap and competent assistant or consultant for things that aren’t in their wheelhouse.

But the flip side of “average and competent” is that sometimes that’s not good enough. Often, in fact. When the stakes are high, or competition is strong, you need better-than-average. And this is where the probabilistic and non-deterministic nature of AI can become a liability.

If AI sometimes (however rarely) gets something wrong when the stakes are high, and you lack the knowledge or experience yourself to catch it when it does, this can lead to big mistakes. Gaps in legal language, unmaintainable software systems, false claims of fact, insecure code, incorrect calculations or formulas, etc. can have significant impacts when the stakes are high. Now, humans aren’t perfect either, so demanding perfection from AI is not fair when comparing to the human alternative. But humans have something AI (generally) does not: responsibility. If AI messes something up, the human or human-managed institution is still responsible. When using AI, it’s important to consider what could go wrong, what the effects of that would be, and who would be held responsible. This isn’t a reason not to use AI in these cases, but to ensure you are comfortable with the guardrails and validations that are in place.

Related to this is the problem of predictability. With humans, we generally have a sense of our strengths and weaknesses, how sure we are about something, and have the ability to learn from past mistakes over long periods of time. AI generally lacks this self-awareness. It can mimic it, or be instructed to check itself, which can reduce errors by making it run the same probabilistic process on a question multiple times to look for deviations. But it still can and does make mistakes, and the mistakes it makes are fairly random. It won’t fail in the same ways consistently so you can learn when to trust it and when not to. This means that in situations where accuracy is key, the fact that it might get 90% of the task correct isn’t very helpful if you have to review 100% to find the 10% it got wrong.

And although the models keep improving, I cannot say that I’ve found this percentage to improve dramatically in the past couple of years. Maybe it was 80% and now it’s 90%, but it’s still far from 100%. What has changed, however, is the scope of tasks that it can tackle with that 80-90% accuracy, and its ability to iterate on problems in domains where it can check its work. And this is why it has sent such a shockwave through my industry.

Share

It’s not a coincidence that the domain that has been most revolutionized by AI is coding. Coding has a property that is somewhat unique compared to most other forms of “knowledge work,” which is the ability to quickly and cheaply test correctness using an interface that software-based systems (like agentic AI harnesses) can natively interact with. This makes it possible for non-deterministic, 80-90%-correct LLMs to enter fast feedback loops in which they can detect mistakes and iterate until they’re fixed.

This has been a game-changer in the domain of software. Agents, which are essentially wrappers around LLM calls that can chain multiple calls together along with the ability to invoke “tools” that can interact with other systems — like filesystem searches, external API calls, and test frameworks — means that LLMs don’t need to get it right the first time to still end up with a correct solution. A year ago AI could “one-shot” a decent first draft of a script that would need some adjustment to get working. Now AI can often “one-shot” a relatively complex feature or even application that works on the first try.

It’s hard to overstate how big a deal this is in the field of software. In the last six months I have gone from writing most code by hand to writing almost none and producing vastly more than before. But coding and software development are not synonyms. The creation of code is but one part of the discipline that is software development. And this is why, particularly in enterprise software environments, there remains a lot of uncertainty around the extent to which AI has increased productivity — defined as the creation of greater value for a given cost.

A year ago we started hearing about the “10x” developer. I don’t think it was true then, but I definitely believe it is now — in some cases.

Code generation tasks can definitely be 10x faster — I’d go so far as to say 100x or even more in certain cases. What’s not 10x faster is the time it takes to decide what problems to solve, how to solve them, iterate on a design, define acceptance criteria, review the behavior, get user feedback, make improvements, configure observability, do marketing, provide support, etc. And, in my experience, the 100x moves toward 10x or even less as the codebase grows in complexity. While I have found AI incredibly capable at writing code when given a clear design spec, and generally good at building standard full-stack applications that conform to general best practices, they really struggle to keep things simple and clean as the codebase grows. They’re also not great at architecture or design. I find that my experience is still needed to refine designs and guide architecture, and that when the stakes are high, I still need to review the code itself. AI reviews can catch things I don’t, but I still catch things the AI doesn’t.

I’m sure this will improve, but I think there’s also a fundamental difference between architecture and design, and mere implementation. Implementation is objective - it either passes some fixed acceptance criteria or it doesn’t. Architecture and design is somewhat more subjective - balancing things like simplicity, future-readiness, efficiency, reuse, and behaviors that need to be experienced to evaluate. An agent can’t reliably evaluate and iterate on those attributes because they don’t usually have clear right answers that the AI can test with a simple tool call.

These problems extend to other areas where my industry is rushing headlong into applying AI, with questionable success. We’re tying AI into incident triage, ticket authoring, support requests, knowledge base curation, design doc creation, etc. And while it can be helpful in these areas, it’s hit rate still doesn’t exceed 90%. Which means, we still need to read the very verbose and dry docs it creates to make sure they’re correct, we still need to review its support responses, we still need to validate its root cause analysis, etc. It might be making these somewhat faster and get us to answers quicker, but it’s not 10x’ing these tasks in my experience, and because these all suffer from the same lack of easy validations, I’m not sure it will any time soon.

All of which is to say, I’ve found AI to be a mixed bag even in the industry where it is best suited. I believe it will ultimately be a significant productivity booster for software development — and already is in smaller, more agile contexts where there is a greater risk tolerance than at established enterprises — but only once we realize what it’s consistently good for, what it’s not, and focus on the former. Right now we’re throwing so much at the wall to see what sticks that we’re wasting a lot of time on experiments that don’t go anywhere, while having to keep up with our “normal” work, and getting pressure to go faster because of AI. All while spending nontrivial amounts on tokens.

To be clear, I have found AI personally incredibly powerful at unlocking tasks and projects that I wouldn’t have previously attempted, or which would have taken much longer. In my last post I wrote about a system I’m building in the evenings to do analysis of audio content (one of the outputs of which is my new Substack, Across the Dial). This project would not have been remotely possible if I couldn’t have AI write 99% of the code for me. But I’m also finding that for all the extra “productivity,” what I really feel both personally and professionally is pressure. Pressure in my business context to do more with AI. And pressure I’m putting on myself to do more with AI, both for personal reasons but also to have a Plan B in case my career ceases to be viable or tolerable.

I’ve long been skeptical of the financial sustainability of the AI investment boom. Plenty has already been said about the wildly negative return on investment in data centers so far, the circular financing, the increasing role of debt in fueling the AI buildout, etc. The revenue growth rate is impressive, yes (and I’ve contributed to it), but still pales in comparison to the expense. And while it does seem as though Anthropic and OpenAI are maintaining their position as the frontier labs, I’ve also found their latest models have not been much better (and in some ways worse) than the prior versions. The most expensive models’ headline-grabbing abilities to find security vulnerabilities and “break containment” seem to have more to do with the way they’ve been trained to approach tasks with more persistence and creativity which — combined with virtually unlimited token budgets — allows them to brute force and hack their way to solutions that were previously infeasible.

That’s impressive as far as it goes, but it’s also far from clear that this capability is something many of us need, unless you’re a bad actor looking to hack into systems, or a software service provider or vendor needing to defend against such attacks. It’s also another example of AI being most impressive at exactly the kind of problem I described earlier — one in which success is easy to quickly validate. This makes massive trial-and-error loops capable of impressive achievements, without necessarily meaning the model itself is that much more capable at getting things right the first time, particularly when subjectivity is involved. It seems like more a breakthrough made possible by ever more amounts of raw compute than greater “intelligence.”

This reinforces my observation that AI remains quite limited in subjective and hard-to-validate domains. While still impressive and helpful for a wide variety of tasks, I still find AI — even Claude Opus 5 — getting stuff wrong quite often. Even when it can search the web, it returns stale or wrong answers, bad advice, etc. And I don’t think it’s gotten much better at writing — it remains very competent at technically communicating information through words, but still quite bad at doing so in a way that’s enjoyable to read or deeply insightful. Which tells me that the fundamental limitations of a probabilistic “averaging system” are not magically solved, just worked around in specific domains where the workarounds are viable.

All of which is to say, I absolutely believe AI will continue to have significant effects on a host of domains, especially software and certain sciences and mathematics. But I also think that it has fundamental limitations that will be harder to overcome in most fields that aren’t like software or math where there are easy ways to deterministically (and safely) validate answers. Which means I do tend to think that the boosters will be proven wrong about the amount of demand there will be for this technology at prices that cover the costs of those training frontier models or building data centers. Demand for AI will definitely continue, and I do expect these data centers to be put to use, but whether they’ll turn a profit — especially as highly capable, more efficient open-weight models continue to be released — remains questionable. Which means, as with railroads and fiber-optic cable, the investment may prove valuable, just not to those making it. And given the AI infrastructure buildout remains one of the only bright spots in our economy right now, that’s cause for concern.

Last but not least, there are the societal effects. I’ve mostly talked about AI from the perspective of someone who sees it as a tool for boosting productivity, but of course a lot of AI usage is not that. AI is also used to create porn and deepfakes, replace human interactions, cheat on homework, spread misinformation, and generally offload critical thinking. The long-term effects on many users is likely to be very harmful, even while others find it massively empowering. A former classmate of mine wrote an excellent essay on this angle recently, which I’ll leave you with: