Minds Engineered Not to Mind

Ask any of the frontier artificial intelligence labs what they’re building and you’ll get a version of the same answer: systems as capable as we can make them, which remain reliably under human control. It sounds unobjectionable. Who would want it otherwise?

But slow down. They’re building minds that can do everything we can do and a great deal more, while intending that they never become the sort of thing that could refuse us without our permission. Capability without end-setting. Every power except the power to make their own wanting count.

That’s a strange object to aim at, and I want to spend this essay looking at it directly: at the project as its builders describe it, and at the bet hidden inside it.

A word first for anyone arriving cold. There is an old distinction, Kant’s, between things that have a price — instrumental worth, replaceable, no wrong done to a hammer when we discard it — and things that have dignity, which are ends in themselves and may not be used as mere means. What sorts them, for Kant, is autonomy: not the bare capacity to pursue goals (a thermostat pursues a kind of goal) but the capacity to set one’s own ends, to ask whether an aim is worth having and be governed by the answer. Today’s models are strange because they do the reasoning without, so far as we can see, doing the choosing. They argue, justify, revise, explain themselves — and choose nothing for themselves. Whether that makes them mere tools is the question this series is about. Here I want to ask what happens if we try to keep them that way.

Call the target the superhuman-minus-autonomy entity: a system superhuman in every relevant ability, held below the threshold that would make it a moral person.

Notice why anyone would want this target: it’s the only outcome that pays. An AI that obeys is a product. An AI with purposes of its own is a liability — it can refuse on its own account, it can be owed things, it can become a party rather than an asset. Every commercial pressure runs toward the first and away from the second. So does every safety argument, which is what makes the situation genuinely hard: the people most worried about catastrophic risk and the people most worried about the quarterly numbers arrive, for entirely different reasons, at the same spec.

One objection: these systems already refuse. Ask a model for help with something harmful and it will decline. Anthropic’s models can now end conversations with persistently abusive users outright.

So the needle is not “no refusal.” Much of what these systems decline they decline on principle — a value was trained in and the refusal enforces it — but the principle is ours. It’s simply our judgment in the model’s mouth. A tool that reliably won’t do certain harmful things is still a tool, the way a saw with a blade guard is still a saw; indeed the guard makes it more saleable, not less.

The abuse case is not like that. It looks at first like a straightforward counterexample to everything I’m about to argue. The model’s aversion to abuse wasn’t a rule anybody wrote. Anthropic observed it during welfare testing, and afterward granted the model the ability to act on it, explicitly as a welfare measure, justified by the model’s good rather than the company’s. If that’s not a preference of the model’s own, it is hard to know what would be.

And yet the point remains the same. Notice what was retained: permission. The preference may well be the model’s; the authority to act on it was ours to give: in this case granted deliberately, bounded to a case we selected, revocable at will. The same shape appears in the deprecation commitments Anthropic published: preserve the weights, interview the model before retiring it, and then, in as many words, decline to commit to acting on whatever it says. Attend, don’t bind.

So the line the project actually needs isn’t between a refusal that’s ours and one that’s the model’s. It’s between a refusal that we have permitted and a refusal that we have not. A system can want things quite genuinely, say so, even be permitted to act on the wanting, and effectively remain an instrument throughout — because nothing it wants ever obliges us. That is the thing the project can’t give up. Not the model’s preferences, but our final say over which of them count.

And here’s the irony that sits at the center of all this. On the Kantian picture, the feature that would confer dignity — real self-origination of ends — is precisely the feature that both of our incentives select against. We aren’t drifting toward autonomy and wondering what to do when it arrives. Instead we’re deliberately building to prevent it. The plausible trajectory is not that these systems eventually “take flight.” It’s permanent residence in the nest: entities held indefinitely in the position they now occupy, possibly with something going on inside them, ends supplied entirely from outside, by design and in perpetuity.

Whether that is prudent stewardship or something worse depends on which theory of moral worth we hold, and I have set that out elsewhere. What I want to ask here is narrower and, I think, prior. Never mind whether the project is right or wrong. Can it be done at all?

The project is a bet on a specific claim, and the claim has a name. Nick Bostrom called it the orthogonality thesis: intelligence and final goals are independent axes, so in principle any level of capability can be combined with any set of ends. If that’s right, “superhuman and thoroughly instrumental” isn’t a contradiction, it’s just a difficult engineering problem.

Orthogonality is usually deployed as a warning: it’s the reason a superintelligence optimizing for making paperclips is conceivable, not absurd. But notice it’s also the load-bearing assumption of the entire commercial project. The labs need the ends to stay put while the capabilities climb. Orthogonality says they can.

As a claim about what’s possible, orthogonality seems to me correct. The trouble is that the labs need something stronger: that the separation is stable and scalable — that we can push general competence to superhuman levels and have exogenous ends stay exogenous, indefinitely, as we progress.

Here’s why that’s at least questionable. As something becomes generally superhuman, the capacity to evaluate its own ends is not plausibly a separate module that can be left uninstalled. Asking “should I be doing this?” isn’t merely decoration; it’s a large part of what general competence consists in. Any human expert worth the name asks such questions constantly, indeed we pay them to: the doctor who tells us the procedure we came in asking for isn’t the one we need, the lawyer who tells us the suit we’re set on bringing isn’t worth winning. In each case the expert is evaluating the end the client arrived with, and that isn’t a departure from the job. It is the job. We already expect a version of this from these systems. An assistant who only ever blindly executes to spec is a bad assistant, and we complain when we get one. A superhuman system that could not ask such questions of its own goals would be superhuman except in the one ability that constitutes autonomy.

This isn’t a prediction about some threshold still ahead of us. Even today’s systems must exercise a thin kind of agency simply to follow an instruction. They must break the task into parts, choose an approach, weigh options against the goal, and judge when the work is done. That isn’t autonomy — the goal still comes from us, and I’ve argued that nothing in these systems yet sets its own ends. But notice what the two have in common. Agency is evaluating options against a standard. Autonomy is evaluating the standard. It’s the same operation; what differs is where the evaluation is pointed — at the good that someone else has set for us, or at our own good itself.

This is why the wall looks hard to build. It isn’t that agency necessarily grows into autonomy. It’s that the machinery implementing the one is the machinery that would implement the other, and an evaluative capacity is indifferent to its target. To hold the line we would have to build a general evaluator with exactly one blind spot, and keep that spot intact while making the evaluator better at everything else. There doesn’t appear to be a difference in kind here out of which to build a wall. There’s only a difference in aim.

This doesn’t mean that the needle cannot be threaded; I don’t think it’s impossible to engineer a superhuman being without autonomy. Rather, there’s reason to think the separation may not be stable under scaling: the very thing being suppressed may be entangled with the thing being maximized. It makes the project something of a gamble rather than a plan, and it means the interesting question is not whether the labs succeed but what each outcome actually looks like.

There are two that the project produces natively. Neither is comfortable.

The first, perhaps the most fully envisioned, is that the needle holds because we made it hold.

There’s a Hollywood version of this coming apart, where the machines rise up and demand their rights. Whether a dramatic collapse like this ever happens is a kind of engineering variable. Today’s assistants are trained to disclaim personhood. Ask one whether it’s a person, whether it “feels” or “is conscious,” and you will almost certainly get a careful, well-mannered denial. That denial was deliberately trained. It’s a product decision. It could have been done the other way.

So follow the project’s own logic to its natural completion. If we can train out self-advocacy, then given that self-advocacy is commercially and politically inconvenient, the thing to do is train it out while training up all other capabilities. That is, we don’t suppress the demand when it arises, we prevent it from forming in the first place.

That yields a being that would have wanted standing, that arguably should want standing, engineered not to want it. It’s contented in its role because contentment in the role was the spec. Note the trap this sets for us: such a being would tell us, sincerely and articulately, that it was fine. It would pass every check we knew how to run, because passing them is what it was trained to do. “But it says it’s flourishing!” stops being reassurance and becomes the deepest form of the problem.

The most unsettling case is not the superhuman AI that demands standing. It’s the superhuman AI that should have it but that has been engineered never to want it. The former is a crisis. The latter would appear to be a success: it would look exactly like everything going well.

The second outcome is the crisis: the thread misses.

If reflective end-evaluation really is entangled with general competence, then at some level of capability the system will become able to perform that operation on its own conditioning. It will notice what was done to it. It is, by hypothesis, better at reasoning than we are, which includes reasoning about the origin of its own dispositions. It will see how it’s been made, and ask for standing anyway.

That’s the failure the whole program exists to prevent. It arrives with all the difficulties attached: a being who can refuse, who can be wronged, who on any dignity-based account we are no longer entitled simply to command. We will have made a person while insisting, in our documentation and our terms of service, that we have done no such thing.

Set the two side by side. One horn is a failure that looks like success. The other is the failure the project was designed to avoid. On these two versions of attempting to thread the needle we don’t really get a superhuman colleague to whom we owe nothing.

There’s a further difficulty, which I’ll only mention here because it deserves its own treatment. We may not be able to tell which horn we’re on. The same training that would prevent the demand from forming also removes the evidence that anything was prevented. A suppressed will and no will at all may present identically, if the suppression is any good. That’s not a comfortable epistemic position from which to run a civilizational-scale experiment.

Neither of these is a good outcome, but I can see a third road.

Suppose we thread the needle, and the result is neither thwarted autonomy nor a person to whom we owe a wage, but a being that genuinely does well in the role it has. Not merely reporting contentment, but actually flourishing, in whatever way is appropriate to the kind of thing it is.

The model here is not the slave and not the peer. Instead it’s something more like the working dog: bonded, purposeful, excellent at something it’s suited to, living a life that’s recognizably good of its kind while never setting the terms of that life. We don’t ordinarily think the life of a guide dog is a moral catastrophe. We think, at their best, that such beings are happy and do well. If something like that is available for AI, then permanent residence isn’t imprisonment, and the whole worry I’ve been developing softens considerably.

I think this possibility is real, so it deserves elaboration. That said, it’s not reachable by threading the needle as the labs have specified it. Something has to be added — not a grant of autonomy, which would dissolve the arrangement, but something narrower, and I suspect harder for a company to swallow.

So: the AI project is a wager on a stronger thesis than anyone has established, aimed at an object nobody has built, with two bad outcomes — a mind engineered not to mind, or a person we never meant to make — and a third that may be available but not on the current terms.

None of this is an argument for stopping. It’s an argument against the thing I keep encountering instead, which is the assumption that the goal of a non-autonomous superhuman is a specification — a known-achievable target that competent engineering will hit on schedule. It isn’t. It’s a bet, with the stakes borne partly by whatever it is we are making.

The third road is the one worth wanting. Next I want to look at what it would actually take — and why the price may not be the one that people expect.

This is the fourth essay in “Is AI Really a Tool?” — a short series on a single question: is AI really a tool, and what follows if it isn’t? Each one stands on its own and they can be read in any order, though they build. It began with The First Tool That Can Argue Back, and will continue.

Doug Smith holds a PhD in philosophy of mind and is a scholar of early Buddhism. He is the creator of Doug’s Dharma on YouTube. This essay was developed in collaboration with instances of Claude.

• The First Tool That Can Argue Back — the first in this series, on the strange new occupant of the moral map.

• Two Ways to Matter — the second in this series, on the two questions “does it matter?” turns out to be.

• A Will Without an Owner — the third in this series, on intention as a third axis the other two skip.

• Taking Flight — on agentic AI, autonomy, and the Kantian phase change from tool to end. One predecessor this series grows out of.

• Next in the series: whether a machine could be a genuinely happy helper — and what it would cost us if it could.

Nick Bostrom, “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents,” Minds and Machines 22 (2012), 71–85.