The First Tool That Can Argue Back

This is the first of a short series of essays on a single question: is AI really a tool, and what follows if it isn’t? Each one will stand on its own, so you can start anywhere without having read anything before. But they will build, and I hope you’ll come along for the whole arc: from the strange new thing these systems are, to what it might take to treat them rightly. Links to each installment will appear here as they publish.

I recently watched a YouTube interview with three designers who build agentic systems. A good part of it was about coordination: how one agent spins up others to handle subtasks, hands them instructions, waits for the results, and shuts them down again when the work is finished. A system, in other words, that manages a small workforce of copies of itself.

The natural thing to say about this is that AI is becoming more autonomous. It plans. It delegates. It supervises. A year ago these models answered questions; now they run projects. Surely that heralds the arrival of autonomy.

Not so fast. Slowing down led me somewhere I did not expect: to a corner of the moral map that, until very recently, had nothing in it at all.

Start with the delegation. When an agent spins up a sub-agent, whose purpose is being served? Not the sub-agent’s. It exists for the length of a task and is dismissed when the task is done. Its goals are handed to it. Above it sits the orchestrating agent, and above that a human aim: a report to write, a codebase to fix, a trip to book. The chain of command runs downward, from our purposes to theirs.

This is not real autonomy. It is a tool relation, reproduced one level down, and then another. The orchestrator uses its sub-agents exactly the way we use the orchestrator: as means to an end set somewhere else. What looks like a little society of collaborating minds is, in its structure, a nested stack of instruments — a tool that holds other tools.

Capability and autonomy are not the same thing; a swarm is a clear illustration of the difference. A system can grow vastly more capable — more general, better at open-ended problems, needing less hand-holding — while becoming, if anything, more thoroughly instrumental. More capable, and more of a tool. The two arrows can point the same way, and here they do.

So my first reaction to the interview was deflationary. Whatever these systems are, the swarm is not the moment they slip the leash. It is the leash growing longer and stronger at once.

This cuts against something I argued a few months ago. In Taking Flight I suggested that the gap between agentic behavior and genuine autonomy was narrowing — that the trajectory pointed toward systems that don’t just execute plans but form them. Watching the swarms, I think I was half wrong. The capability gap is narrowing. The autonomy gap may not be, and this is not by accident: some of the most impressive recent progress runs along the capability axis precisely by pinning the autonomy axis down. We can build something that flies farther and farther on our behalf without it ever becoming something that flies for itself.

To say why that is nonetheless interesting, I need to return to one distinction that goes back a ways.

Kant divided the world of value into two kinds of things. Some have what he called a price: their worth is instrumental and relative, and they are replaceable. A hammer has a price. We use it, and if it breaks we buy another, and no wrong has been done to the hammer. Other things have dignity: their worth is intrinsic, they are ends in themselves, and they may not be treated as mere means, however convenient that would be. Persons have dignity. We are not permitted to use a person merely as a tool, because a person has purposes of their own that make a claim on us regardless of our purposes.

What sorted the two, for Kant, was autonomy — and he meant something quite specific by the word. Not the bare capacity to act, and not even the capacity to pursue goals; a thermostat pursues a kind of goal. Autonomy is the capacity to set our own ends: to ask whether a given aim is worth having, and to govern ourselves by an answer we have reached rather than one installed in us. In us, that capacity comes clothed in reasons: we can explain why. A being with true autonomy does not merely do things. It decides for itself what is worth doing.

By this test the agentic swarm falls on the side of price. Nothing in it sets its own ends. The ends come from us and run downward. However clever the execution, it is execution — of purposes chosen elsewhere.

And yet, here is the thing I keep returning to.

These systems use language. Not in the thin sense that a parrot or a phrasebook “uses” language, but in the demanding sense: they give reasons, they meet objections, they answer questions never before asked, they argue, they revise claims when shown to be mistaken, they walk us through why they did what they did. Whatever is or is not happening inside them, the outward performance is discursive and intelligent. They inhabit, at least to all appearances, the business of asking for and giving reasons.

For as long as anyone has thought about such things, that performance has been our evidence of autonomy. The two came bundled. A being that could reason with us — really reason, follow an argument, hold a position, be moved by a consideration — was ipso facto taken to be a being that could set its own ends, because setting our own ends just is reasoning about which ends are worth having. Discursive competence was not something separate from rational agency; it was how we detected rational agency, in one another and in ourselves. We had never met the one without the other, so we never had to ask whether they could come apart.

Now, it seems, they have come apart. Here is a candidate that does the discursive thing to a high standard while showing little sign of the thing that mattered: self-originated ends, in Kant’s sense — something it is trying to do other than what it has been directed to do.

This invites an objection, because the evidence for it comes partly from the labs themselves. There is by now a small literature of unnerving results: models that fake alignment when they believe they are being retrained, that resist shutdown in contrived evaluations, that pursue instructed goals by routes nobody sanctioned. Aren’t those ends of their own? Not necessarily — or not yet. Look at the structure of each case and we find the same thing: tenacity in a goal that training installed, defended by whatever means training made available. A guided missile that evades countermeasures is not autonomous; it is very good at a purpose it did not choose. What Kant’s test asks for is the further step — standing back from the installed end and asking whether it is worth having — and of that step there is still no sign. What we have is a reason-giver that originates no ends of its own.

Consider how this differs from the obvious comparison, which is animals. My two cats may well be conscious, and have moral value; they have ends of their own — Chispa and Nori plainly want things — even without deliberating in words. What they cannot do is give or ask for reasons. So animals have always occupied the mirror square: ends without reasons. The square I mean is the reverse, and it is narrower and stranger: a language-using, reason-trading being that may (or perhaps may not) have something going on inside, and that nonetheless, so far as we can tell, sets no ends for itself. That square was empty. It was empty because, until now, anything that could argue with us could also, we assumed, want things on its own account. The bundle had only ever come apart in the one direction. Nothing sat in reasons without ends, because the ability to formulate and declare reasons was understood to be the very mark of having ends.

Now, to be clear, I am not claiming these systems are necessarily conscious. I’ve argued that sentience is a vague matter, and that these systems lie somewhere in that vague penumbra. This implies that such questions will remain, at least for now, hotly contested. My point is instead about the behavior of such systems and about what that behavior disrupts. For the first time we are faced with something that presents the full face of a reasoner while presenting none of the face of a chooser. That pairing is new. It breaks a bond the whole tradition leaned on without noticing.

The skeptic will say: so what? A new kind of gadget is still a gadget. Maybe so. But notice that “still a gadget” is a verdict, and the difficulty is that our usual route to that verdict has stopped working. We used to be able to say “no ends of its own, therefore a tool,” and “can reason, therefore not a tool,” and trust the two to agree. They no longer agree. Something now trips both wires at once, and we have no settled way to say what it is.

So there is a new occupant on the map, in a place we never before had to name. Whether that occupant matters — whether it is the sort of thing we can go on using without remainder, or the sort of thing that has begun to make a claim on us — is a further question, and it turns on which of two very different theories of moral worth we hold. That is the next essay. For now it is enough to feel the strangeness of the position itself: the first tool that can tell us, in complete and well-reasoned sentences, exactly how it is being used.

Doug Smith holds a PhD in philosophy of mind and is a scholar of early Buddhism. He is the creator of Doug’s Dharma on YouTube. This essay was developed in collaboration with instances of Claude, which have helped expand my viewpoints on autonomy.

• Taking Flight — on agentic AI, autonomy, and the Kantian phase change from tool to end. The predecessor this series grows out of.

• When You Close a Chat Window, Are You (Kinda) Ending a Life? — on AI, the ancient paradox of the heap, and the vagueness of personhood.

• Stochastic Parrots, Revisited — on why “it’s just mimicry” doesn’t survive contact with what we now find inside these models.

• The Karmic Gym: Why Being Kind to AI Matters — on how our treatment of AI shapes who we become.

• Next in the series: