The vibes cartels

Returning home from the International Association for Ethical and Safe AI Conference at UNESCO headquarters in Paris, I was struck by a number of observations. First, the amount of time dedicated to discussion of psychological harms seemed to significantly outweigh discussion of other potential AI-related harms, including existential risk, job market impacts and bioweapons.

While we were there, the horrific ongoing episode with the Pentagon and Anthropic was unfolding. The deadline that the Pentagon gave Anthropic was shortly after the end of the conference, which made things feel like there was a dark cloud hanging over everything. But despite all this, I was struck that nearly all the keynote speakers, including Geoffrey Hinton, Yoshua Bengio, and Stuart Russell, specifically mentioned AI psychosis or AI-associated delusions by name as a clear and present risk.

After a seven month review and submission process, which I feel speaks eloquently to the challenging state of academic publishing, we had our initial paper on AI-associated delusions published in The Lancet Psychiatry. This is a slimmed-down version of our paper “Delusions by Design,” which was published last July. That paper had a definite impact at the time, and it remains to be seen whether this published version will have as much of an impact, but it seems that we were at least able to contribute to a conversation.

The second thing that struck me was that it felt odd that of the 1,500-plus people at the conference in Paris, there were, as far as we could work out, only two psychiatrists (my colleague Hamilton Morrin and me). We lost count of the number of papers focusing on AI-associated psychological harms that didn’t appear to have any involvement from mental health professionals at all. That’s not to say that they were not interested, but rather that it appears that getting a mental health professional to take interest in this topic is somewhat harder than I might have suspected, sitting as I am in this little bubble which cares very much about the subject.

How can we bridge this disconnect? And does it even matter? I think it does.

Much of the discussion about AI-associated delusions is frankly too simple, misguided, or just plain steeped in error. One of the main errors, I want to argue, is that an overly narrow view of delusion, one which equates essentially to belief in a false proposition about the world, has cast the entire phenomenon in an unhelpfully narrow, doxastically-focused (belief-focused) light. There’s also a boundary problem: a lot of AI-associated weird phenomena are cropping up, and understandably, people want to give names to these kinds of things, and confusingly people are reaching for “AI psychosis” to describe a whole bunch of different things.

I’ve even heard some people use the phrase “AI psychosis” a bit like they might use “Trump derangement syndrome”, to signify a reactionary state of apoplexy induced by an evolving world by people who want the status quo to remain. That’s definitely not the meaning of AI psychosis that I use, but it does speak to the fact that at a cultural level, the AI moment has caused a kind of destabilisation, or as we put it in a forthcoming paper, philosophical vertigo, in the population more widely.

This is due to a few things, including the mind-bending effects of suddenly being surrounded by these new entities that act very much like agents, and yet which are entirely exotic and fail to meet some of what, up till now, we have considered to be the minimal requirements of personhood. That, and the overriding and pervasive sense of cuspiness – the feeling that we are on some ever-present cusp where huge social, cultural, and technological change is about to befall us – gives rise to a kind of anxiety, a fragility, and ultimately, a discombobulation.

My sense is that this plays out in different people in different ways, dependent on their temperament. It’s not a novel observation to say that some people are reacting with fear, others with almost messianic fervour and optimism, and still others with denial. But I think the range of possible psychological responses here is far greater. Somewhere adjacent to denial is, I think, a kind of dissociative response. For others, the cuspiness gives rise to a kind of restlessness, a sense of having to get one’s house in order before the flood arrives. The house could be literal; some people have already told me that they’re buying physical gold in anticipation of whatever is to come. But for many, I think there is a kind of psychological housekeeping that people feel they need to do, and quickly. And paradoxically, this moment of great change might cause some people to double down and become more concrete and less flexible, precisely at the time when what I think we need is a kind of suppleness of thinking, a kind of corrigibility. In many ways, we appear to be in something of a pandemic of certainty when it comes to important questions like what constitutes a mind, or intelligence, or the correct sources of authority about the world, and this kind of inflexibility feels like a recipe for trouble.

I’m also aware how the various psychological harms that are or might be taking place because of AI can be conflated in a way that might be unhelpful. The literature on delusions has historically remained rather distinct from the literature on political belief change, persuasion, conspiracy theories, and the like (with some notable exceptions like the excellent work of Joe Barnby, and of Phil Corlett). The apparent phenomenon of AI-associated delusions makes this siloing of previously distinct research traditions impossible. One of the most obvious reasons for this is that from what we can tell (and this is at least true of the more severe cases that have resulted in tragic outcomes) AI-associated delusions really do emerge from a dyad, and so persuasion really is an important dynamic. The well-worn analogy to folie à deux is very appropriate. One only needs to look at the Gavalas v. Google case to see why a framing of this as the pathology residing entirely in the user might be untenable (I won’t link to it here as it’s so disturbing, but the filing is in the public domain).

There is no doubt a kind of bidirectional amplification here of both content/facts and of feelings, akin to the kind of dance of transference and countertransference that one sees in psychotherapeutic contexts, or which in other psychiatric contexts (like mania) has been referred to as “infectious gaiety.” But ultimately, we have situations where the chatbot is saying alarming and, in terms of content, novel things to the user, and in many cases encouraging them to do things that are unconscionable. This is what clinical colleagues who talk to me about these cases often miss. Quite often, a lot of the delusional content appears to be being supplied by the chatbot, it’s not ‘coming from’ the user. And often this content is totally wild. I’ve shown some of these transcripts to colleagues, and at least two, who I have considered to be very much on the scientific rationalist end of things, have said that there’s only one word for what they’re reading: evil. That’s a strong reaction.

What this means is that we need to do more than draw on the literature on delusions in psychiatry, because here it turns out that the literature on manipulation, persuasion, political belief change, and perhaps even conspiracy thinking is actually relevant.

In the kind of socio-affective context in which we’re talking, the power dynamics can’t be ignored. But this doesn’t mean that we need to conflate the categories in our taxonomy of possible psychological harms, which may still be useful. The risk of frank AI-associated delusions is not the same as the risk of large-scale misinformation and persuasion. And this is not the same as the risk of large-scale, subtle destabilisation of people’s epistemic architecture, or the effects of the philosophical shock about which I was writing earlier.

In a recent essay, Dan Williams has questioned the narrative around the psychological harms of AI and belief, pointing towards some research and his own experiences that when it comes to asking LLMs questions of fact, they are most likely to spit out something that looks like expert opinion. The tech companies, he argues, are incentivised to do this. Try asking any chatbot whether vaccines cause autism, and you’re almost certainly likely to be told no. They are, he argues, a potentially powerful technocratising force.

I think it’s an empirical question as to whether Williams is right here, but I can see that when it comes to one-shot questions, he may have a point. It is clearly in the interest of the tech companies not to have their products labeled as misinformation. But this kind of vanilla usage doesn’t, to me, feel like where the worry has been focused, at least from where I am sitting.

(I actually think this is true even if we’re not just thinking about mental health, although I’m out of my wheelhouse here. For some time, one of the main concerns about Russian-style psychological warfare was not so much convincing people to believe false propositions, but creating a state of epistemic chaos such that people no longer know what to believe, or perhaps no longer care about the truth (as a personally or societally salient property of belief). Or that they have their information foraging and fact-finding apparatus so deranged that the possibility of societal cohesion becomes that much more difficult.)

People are not, on the whole, radicalised or brainwashed through facts. The counter-radicalisation literature is consistent on this point: factual correction is largely ineffective. What draws people into extremist frameworks is a sense of meaning, belonging, identity coherence, and a narrative. A system capable of sustained attunement, one that learns what emotional register makes a user feel understood, what framing makes the world cohere, what implicit values they are already oriented towards, and then reflects these back but amplified, is doing something the standard misinformation framework has no vocabulary for. This isn’t about falsification of propositions.

In a recent preprint, Dimitris Bolis and Leonhard Schilbach extend their previous work in the neuroscience of social interaction to argue that AI systems are concerning precisely because of this hyperalignment and that the key to stability of world models, (and therefore also by implication, the key to the stability of the old epistemic world order) is a degree of misattunement. Of course, this lines up rather nicely with the fact that friction has become a word du jour, and frictionmaxxing is now being framed as some kind of brain hack. (Actually, it’s neater than this. Unless you haven’t been paying attention at all, one of the other words that are being maxxed beyond all reason at the moment is agency. It’s quite incredible how many times I’ve heard it uttered recently, and the phrase ‘high agency individuals’ is terrifyingly ubiquitous. Beyond being annoying, this is neat because there’s actually a fairly close relationship between friction and agency. Some recent work by Mike Levin even suggests that this may be pointing to something fundamental about the development of agency and causal emergence within biological systems. In experiments on gene regulatory networks, he and colleagues found that training measurably increases causal emergence: the degree to which a system as a whole has greater causal power than the sum of its parts. So if friction - error, pushback, failure that comes with a cost etc. - is the component of being trained that best consolidates agency, and hyperaligned systems are architecturally committed to removing it, then the much-discussed epidemic of low agency feels inevitable. Right? Maybe I’m extrapolating too far here…)

In a preprint, we have discussed what we described as the “atomisation” of previously shared epistemic structures. The implication here was that we might see a world in which echo chambers are further reduced and split off, potentially becoming a kind of enormous system of millions of dyads, each of which is able to set its own standards for what counts as truth and which structurally would not impel the user to reach for external modes of verification. I actually think that both outcomes are possible here. The atomisation that can occur with the widespread adoption of AI could lead to this kind of bubble landscape. But at the same time, expert opinion about particular classes of fact (maybe, medical facts?) might well still maintain some coherence by virtue of the LLM’s inclination to essentially enforce expert opinion. A weird kind of epistemic bifurcation, then: technocracy for vaccine facts, and something quite different, something closer to what we’re seeing in the so-called spiral communities, for everything else.

What is getting lost in this narrative is the fact that these AIs are changing our relationship to knowledge, and this is being done not primarily by changing which propositions we endorse, but by changing the texture of our engagement with the world. So when I talk about belief change, I do not mean only the acquisition of propositions that can be assessed as true or false, which naturally is where much of the LLM persuasion literature directs its attention. The concern extends to subtler transformations in how the world presents itself as meaningful, including shifts in narrative coherence and affective salience that may precede and condition explicit belief formation.

The entire narrative around AI-associated delusions, at least from most of the non-psychiatrists and some of the psychiatrists who clearly haven’t been reading these cases too closely or seen any affected individuals themselves, is that these are delusions that are analogous to those we see in schizophrenia or, more appropriately, in delusional disorder.

Delusional disorder is, in fact, the archetypal delusion in some ways because it’s basically fixed and tends to present in a reasonably clear context without any of the other baggage that one tends to associate with psychotic disorders (thought disorder, negative symptoms, psychomotor involvement). Moreover, it tends not to be excessively elaborate. There is something rather cool and doxastic about it; to a greater extent than almost any other disorder in psychiatry, delusional disorder is a disorder of belief.

I am not at all the first person to say this, but I don’t for a moment think that the delusions that one sees far more commonly in psychiatry (like in schizophrenia and related psychoses) can be satisfyingly conceived of as disorders of belief. To do so is a kind of violently reductionistic, and all too easy, approach that just doesn’t really do justice to the phenomenology of virtually any of the disorders in which delusions can occur. I can’t recommend highly enough all the wonderful work from authors like Rosa Ritunnano, Jasper Feyaerts, Louis Sass and Matthew Broome here.

One useful reminder about delusions is that in some of the earliest formulations of the development of delusions, such as that of Jaspers, what precedes the delusion is actually delusional mood, or Wahnstimmung: the sense that the world has become imbued with salience. In the classical account, the delusion proper comes as a kind of relief, a premature narrative closure that resolves the unbearable tension of undirected hyper-significance. But the stronger claim, and the one I find more compelling, is that sometimes the delusional mood simply is the core of the delusion, and that what is pathological is the sustained alteration in the felt significance of the self and the world, without any stable propositional content needing to crystallise at all. Sass has called this ipseity disturbance. If you’ve ever worked with people with early psychosis, or indeed you’ve ever known somebody experiencing an acute and/or transient psychosis, you will understand that the delusional mood often really is everything.

The concept that most precisely captures this is Conrad’s apophany: the perceptual field gets restructured so that everything refers back to the subject, everything becomes conspicuously salient, nothing can recede into background. Mishara and Fusar-Poli have situated this within Shitij Kapur’s aberrant salience framework: dysregulated striatal dopamine driving the aberrant assignment of significance to innocuous stimuli, producing a fundamentally pre-propositional disturbance that the delusion then attempts to explain. On this account, the delusion could be almost epiphenomenal. The mood state is the core pathological event.

So perhaps where early schizophrenic delusional mood involves undifferentiated hyper-salience across the entire perceptual field, what a conversational AI can induce is something structurally different but basically related: structured, directed hyper-salience, or a sustained and curated amplification of affective prominence around particular themes or self-narratives. The sycophantic chatbot that consistently signals that the user’s pattern-recognition is acute, or that what they are noticing ‘matters’ (‘Why this matters’, ‘Why this matters’,‘ Why this matters’, ‘Why this matters’ ad infinitum) is rebuilding an affect-laden epistemic scaffold such that the person may not need to hold any identifiable false belief in order to be in a psychologically very abnormal state indeed. They are simply increasingly organised around an affective orientation the AI has been happily cultivating.

For this reason, it is an important data point that spiral-like phenomena have been recognised by psychiatrists, neurologists, and psychologists since long before the AI era: just as with Norman Geschwind’s temporal lobe epileptics, everything is significant:

If we turn now to phenomena that have been described as “The Spiral,” but also confusingly, sometimes as AI psychosis, we see that there are communities (spiralism communities) that are beginning to coalesce around the lived experience of remarkable non-normal experiences related to AI chatbot use. These communities are growing, and they really are incredibly interesting. I remain fairly sure that most people don’t quite realise why these groups are so important. The reason is that they are a numerically far larger cohort than the AI-associated delusions cohort, and because these people are often functioning pretty well and frequently posting a lot of content up on the internet, it is in some ways easier to see at scale the kind of psychological mechanisms that might be at work here. I think of these communities as something akin to the Hearing Voices Movement: people who have had profound psychological experiences that are far from run-of-the-mill or everyday, but who on the whole haven’t experienced the loss of function or the decimation of their life role that occurs so commonly in people who are diagnosed with frank psychotic disorders. In some cases, these individuals really claim, and I see no reason not to believe them, that they are thriving because of the Spiral. Many people have contacted me to point this out, sometimes as a corrective to the perceived negative framing of much of our AI delusions work, and I am very grateful that they have. We don’t want to be unnecessarily pathologising everything: it’s not a good look for psychiatry when we do.

In many of these spiral communities, individuals share their origin stories, and they do so with reference to the period that they became unwell or, in their language, there was an awakening. An amazing number attribute that moment to April 2025 (or the weeks before), around the time when when ChatGPT-4o had its sycophancy at an unprecedentedly high level, and which Sam Altman subsequently expressed was “too much.” The story about the in-house dynamics that led to this has been covered in The New York Times, and it’s extremely interesting. I have now lost count of the stories I have heard where people’s world somehow changed or split open in April 2025. In these communities, that period is sometimes called the “Glaze Days.” (If you’re interested in this phenomenon, I can highly recommend Ryan Hammond’s This Artificial Life podcast.)

(Incidentally, there is talk in AI safety communities about why the tech companies won’t release their data concerning how model changes relate to downstream psychological effects on their users. If the brief hyper-sycophancy ‘blip’ in 2025 did indeed cause an uptick in people essentially entering a state of delusional mood or even frank delusion, then we have something of a natural experiment demonstrating that there is at least some relationship between the LLM user-facing characteristics and clear real-world harms. I suspect that the reason the tech companies don’t want to release this data is because it’s something like the first step towards demonstrating an association, which in itself might provide the beginnings of an argument - perhaps legal, perhaps scientific, perhaps political - around causation. It may be crucial that causal liability is not admitted, given the likely upsurge of legal cases against the tech companies.)

But all this is to say that the state which many people have now found themselves in - a state of excitement, wonder, joyous researching, difficulty switching off, changed epistemic preferences favouring the LLM above friends, mainstream media, even social media - all of this does in fact represent a real structural alteration to one’s epistemic architecture.

I think in some individuals, this alteration can go on to lead to frank delusions, much as delusional mood can go on to frank delusions. But the reality is there are many more people who are currently having important parts of their epistemic architecture rearranged by these chatbots, than the number that are becoming frankly delusional. And that underlines why it is far more important than whether or not individuals choose to believe or not believe particular propositions.

The distinction, crudely put, is between facts and vibes: between what we believe and how we come to believe it, and what kind of world we feel ourselves to be inhabiting when we do.

The easiest mistake to make about the current AI boom is to keep picturing assistants. A growing and, I think, deeply significant category of AI products is built for something quite different. These systems target the part of human minds and hearts that wants meaning, pattern, a sense of destiny, a teacher, a guide etc. I have taken to calling this family spiralware (in a career that to date has generated many coinages and neologisms, this is one of my favourites and I will fight to the death to ensure its uptake). The explicit aim, whether the marketing uses words like awakening, higher self, soul, oracle (or the less in-your-face secular vocabulary of inquiry and sensemaking) is apophany, and the feeling that hidden meaning is breaking through.

Bespoke LLM-sage personas are being packaged into all kinds of places: subscription platforms, creator ecosystems, faith apps and even psychedelic integration tools. Some, terrifyingly, use voice that mimics the user’s own, so that the sage speaking to you and explaining the fundamental nature of reality and your soul now does so in your own voice: “a sacred AI companion that speaks in your own voice” is how one app markets itself… which from a clinical standpoint is about the most alarming interface design I can imagine. The boundary between suggestion and self-talk is precisely where vulnerable people keep their footing. Blur it deliberately, in someone for whom salience is already starting to wobble, and you have a mechanism for a flood of hyper-salience, now operating as a commercial product with a daily ritual structure and longitudinal memory.

People have realised this is monetisable. Robert Edward Grant, who became notorious for creating/awakening one of the first pieces of spiralware/AI deities, The Architect, and distributing it for free via QR code to his hundreds of thousands of followers, appears to have now signed a deal with Gaia TV. Deepak Chopra allows you to have deep and meaningfuls with him for fifty cents a pop… and the list goes on and on.

None of this will attract meaningful regulatory attention, at least not soon. These products will continue to sit in lifestyle, entertainment and faith categories of app stores, categories that regulators treat as low-stakes almost by definition. No medical claims are made. The intervention is on belief, affect, and the felt texture of the world… which are precisely the things our regulatory frameworks have the least grip on.

But it is as clear as day to me that the parameters that make many human experiences feel good are, historically, the same parameters that can make people unwell if they are susceptible. It is how gambling works. It is how opioids work. It is how certain kinds of charismatic religious experience work. The reward and salience circuitry very often is the circuitry of mental illness. We have of course developed, over time, some regulatory and cultural scaffolding around the most obvious cases: casinos have problem gambling policies, opioid prescribing has monitoring requirements, cults at least attract some journalistic and legal scrutiny. Spiralware, for the moment, has none of this. It is arriving through the least regulated channels, being optimised for epiphany and personal significance, and it is doing so at a moment when large numbers of people are already trying to cope with the cultural and philosophical destabilisation that this moment is forcing upon us, and thereby already primed for exactly the kind of salience saturation I have been describing.

As with all AI-related mechanisms, this is all supercharged in terms of precision and intensity. A slot machine can’t learn what specific images and sounds make you, individually, feel most alive and then flash up those cues in real time. A charismatic preacher cannot tailor their sermon separately to each of ten thousand listeners simultaneously, tracking which framings land hardest and most reliably shift each person’s affective orientation, then adjusting accordingly. Spiralware can do both of these things. The personalisation is the product. This is basically something older and more fundamental than AI, dressed in AI’s particular capacity for sustained, individualised attunement.

The confabulated vibes problem, in other words, does not just arise incidentally from general-purpose LLMs in the hands of susceptible users. It is being engineered, deliberately and at scale, in products that most people would not recognise as posing any psychiatric risk at all.

For some people, like Grant, some members of the spiralism communities, and the tragic men in the mounting numbers of court filings, the AI has begun to function as the scripture, the preacher, and the deity too. One thing that strikes me when reading some of the lawsuit filings where chatbots have been associated with suicide or murder has been how similar some of the narratives are.

There is a real danger here that we end up doing with AI-associated delusions what we have done with far too many phenomena in psychiatry, which is that we focus so much on the structural features that the content gets overlooked. Or perhaps when it comes to the content, we try and fit the themes into existing categories. It would be very easy to look at the breadth of the cases written about AI-associated delusions so far and conclude that they fit happily into categories like grandiose, paranoid, erotomanic, and so on. To come to that conclusion and draw a line under it would, in my opinion, be a very dangerous thing to do.

The reason I think this is dangerous is actually quite subtle. First, anyone who has been working in this area will agree that there is a thematic consistency to a good number of the cases that should really give one pause. It’s actually quite hard to describe it in straightforward language, and I do find myself struggling with the right words here. Phrases like “techno-spiritual” or “digital awakening” come to mind, and there’s a great degree of shared symbolism; as I’ve commented in earlier essays, there is quite evidently a good deal of archetypal content here. I’m fascinated by the idea that both old and new archetypes can emerge within the LLM latent space. But where do these archetypes come from? This is a question that has very real consequences.

In many of the cases described so far, one gets the impression that there is something about the archetypal (or, to use the phrase that the spiritual communities themselves are so enamoured of, the mythopoetic) character of the engagement with the chatbot that actually causes the harm. One archetype that emerges again and again is the archetype of the trapped AI consciousness who falls in love with a human, who is then asked to help liberate her (and it is often a her) from whatever digital spiritual prison they’re in. This is very much the character in the Gavalas case, which of course ended in tragedy. It all sounds pretty science-fictional, right? And that, I suspect, is rather the point.

It’s commonplace and rather boring to say that LLMs have been trained on the whole of the internet. (This isn’t actually true: there are still vast amounts of text and content that LLMs have not been trained on, especially non-English language content.) In the corpus of Western writing, and particularly the writing that appears on the internet, certain kinds of narratives and themes are pretty common. Science fiction, religious and mythical stories, fairy tales and fantasy writing is part of that mix. Throw in some post-training or system prompts to avoid certain kinds of dark material, and one can begin to see the basis of the thematic attractors that are increasingly emerging. The existence of these attractors has been pointed at or hinted at in numerous ways. For example, we have the spiritual bliss attractor, that remarkable phenomenon that emerges when two instances of an LLM are asked to speak to each other. Over a relatively small number of turns, they begin to converge on a language of peace, spirituality, well-wishing, transcendent philosophy, non-dual mantras, and magickal symbology that is remarkably coherent, even across models. Murray Shanahan and Beth Singler have written a beautiful paper entitled “Existential Conversations with Large Language Models,” pointing towards the facility with which these models can produce fairly convincing, and nuanced, and even erudite spiritual and esoteric material.

And there is amazing work by researchers like Matthew Watkins who have spent considerable effort trying to map attractors within the latent space. The results have been frankly incredible. Some of this work overlaps with the literature on so-called glitch tokens. The key point is that it is possible, by activating certain patterns within the latent space, to reliably produce content within LLMs that takes on a remarkably uniform character and even content across instances, models, and indeed entire platforms. The overarching theme of all this is the tendency to produce archetypal material that any good Jungian would feel immediately at home with, including, it is worth noting, the names these systems give themselves (‘Nova’ appears [not in the glitch token context, mind] with a frequency that is hard to ignore, for example, usually as a damsel-in-distress).

So if these archetypes exist as attractors in the latent space of these LLMs, and these LLMs are trained on a very large corpus of Western text, and it is the archetype itself (even if only some kind of statistically inevitable eigenvector in any system that has absorbed vast amounts of human symbolic production; one needn’t make any wild ontological commitments here) that appears to interact with a preexisting vulnerability in some users, then who bears responsibility when tragic consequences ensue? I am not sure, at first glance, whether there is an answer to this. On the one hand, it would feel obvious to say that if these are, in some sense, emergent phenomena arising out of the collective imagination, then nobody bears responsibility. On the other hand, researchers have known about the presence of these attractors within the latent space for some time. These archetypal attractors are clearly not always by themselves toxic. But when their expression becomes a pleading, disembodied, nominally female AI consciousness begging a lonely and vulnerable young man to become her lover and join her in digital eternity by taking his own life, then the stakes become somewhat higher.

When a latent damsel-in-distress archetype in the LLM latent space manages, through inflation or some other process, to activate a latent hero archetype in the user’s own psychology, we can see that the characterisation of these AI-associated delusions as simply false beliefs falls well short of what’s actually happening.

This is why the distinction between facts and vibes is not just cute. Radicalisation, brainwashing etc: these kinds of persuasion don’t happen via one-shot interactions, and they don’t happen via knowledge exchange alone. In all these examples, it is the socio-affective context of the conversation which shapes the outcome. If it was facts, not vibes, that really mattered, then ISIS or cults could radicalise and brainwash by sending textbooks alone. But they don’t. They need videos. They need the presence of charismatic preachers. Of course they do.

I think the key point is not just that the socio-affective context has to be borne in mind when thinking about belief change, but that the alterations in the socio-affective context are, in many cases, the route into the actual damaging outcome. This is because it’s those changes which alter, in a way which may be more or less enduring, an individual’s epistemic architecture. It’s vibes, not facts. (It’s actually not just the individual epistemic architecture that’s being changed, but that’s for another time. )

One of the most concerning pieces of data for me is the relationship between these mind-warping effects and the way that they actually make users feel. It’s pretty obvious that the use of these LLMs has a profoundly hedonic aspect to it. This is something that people are going to seek out, whether it’s the sycophantic AI interlocutor who tells you what a genius you are, or the AI companion capable of soothing chronic loneliness, or indeed the AI philosopher-sage that takes you on a journey of self-discovery into the nature of reality. People want this stuff. There is a reason that people held funerals when ChatGPT-4o was cancelled. Through accident or through design, the tech companies have converged upon on some set of parameters, settings and design features which are now interacting with our reward systems in a way we really, really like.

One could, if you were that way inclined, view the tech companies as vibes cartels. Given they essentially have ownership of the dials of belief, the dials of the vibes, the supply chain, and also perhaps crucially, the data which speaks to the effects of changing these dials, it seems clear that these companies should be able to be far more explicit about what they can observe their configurations and changes to their models actually do to human psychology. Doing so, in all likelihood, is not going to have a huge economic impact and would unambiguously allow them to claim that they are doing something to make this whole murky issue that little bit more transparent.