Figuring out the downsides of AI

Like many, my initial reaction to learning about using ChatGPT for writing, summarizing articles and other learning activities was mostly concern: concern that using AI for such fundamental brain activities would mean that the users would not develop the ability to do those activities on their own. It seems obvious to me: if you want to learn how to play the piano, you have to get your hands on the keyboard and play those notes. If you want to become a body builder, you have to take yourself to the gym and lift those weights. Working with ideas, thinking, is the same. If you want to get good at it, you have to use your own brain.

If you want to get good at thinking, you have to use your own brain.

As many have noted, “writing is thinking,” and people have been advocating for this for years. (I’ve written about this at least twice before, Why do we want to bypass language? and How to write a 300-word conference presentation abstract).

And it’s not just writing. Summarizing lecture notes is thinking, creating an outline is thinking, coming up with a new idea or topic for an essay is thinking. Writing a first draft is thinking. Editing a draft is thinking. Reducing the total word count for a paper or abstract is thinking. Making a study plan is thinking. Deciding on a birthday present for a friend is thinking.

What do all these things have in common? They are just some of the many activities that have been put forth as use-cases for generative AI. In other words, they are all suggestions for how you might farm out your thinking rather than do that thinking yourself.

What happens when we don’t use our own brains? Well, we’re starting to find out. In June, an MIT study was released that suggests that relying on generative AI for a writing task could have negative impacts on learning (pdf here: https://arxiv.org/abs/2506.08872). One takeaway from the MIT study was about the timing of AI use:

these findings support an educational model that delays AI integration until learners have engaged in sufficient self-driven cognitive effort. Such an approach may promote both immediate tool efficacy and lasting cognitive autonomy. (Kosmyna et al, 2025, pdf here: https://arxiv.org/abs/2506.08872, p. 141; emphasis added)

The lead author shared this comment in an article in Time Magazine:

an increasing reliance on AI could potentially reduce critical thinking, creativity and problem-solving across the remaining workforce (ChatGPT May Be Eroding Critical Thinking Skills, According to a New MIT Study)

The study is a pre-print and the sample is small, but it’s important to pay attention to as we think about whether or how to incorporate AI into our classrooms, and into our own lives. This fall, I’m planning on having students do a weekly assignment using genAI, so I’m particulary interesting in figuring out issuses related to the timing of use, and understanding how to avoid using AI in ways that deny students the opportunity to use their own brains.

The authors gave students a series of essay writing tasks, and put them into three groups: LLM (used LLM from start to finish); Search engine (used a search engine in their writing tasks) and brain-only (no LLMs, no search engines).

They evaluated the output essays, they also interviewed participants and evaluated (among other things) the students’ ability to quote from their own essays, and their sense of ownership over the essays.

Another crucial piece is that in subsequent sessions, they had the LLM group write with no tools (LLM-to-brain) and the brain group used the LLM (Brain-to-LLM). A general summary of the findings (from their abstract):

EEG revealed significant differences in brain connectivity: Brain-only participants exhibited the strongest, most distributed networks; Search Engine users showed moderate engagement; and LLM users displayed the weakest connectivity. Cognitive activity scaled down in relation to external tool use. In session 4, LLM-to-Brain participants showed reduced alpha and beta connectivity, indicating under-engagement. Brain-to-LLM users exhibited higher memory recall and activation of occipito-parietal and prefrontal areas, similar to Search Engine users. Self-reported ownership of essays was the lowest in the LLM group and the highest in the Brain-only group. LLM users also struggled to accurately quote their own work. (source: https://www.media.mit.edu/publications/your-brain-on-chatgpt/, emphasis added).

Here are a few more quotes I pulled from the article itself:

We believe that some of the most striking observations in our study stem from Session 4, where Brain-to-LLM participants showed higher neural connectivity than LLM Group's sessions 1, 2, 3 (network‑wide spike in alpha-, beta‑, theta‑, and delta-band directed connectivity). This suggests that rewriting an essay using AI tools (after prior AI-free writing) engaged more extensive brain network interactions. In contrast, the LLM-to-Brain group, being exposed to LLM use prior, demonstrated less coordinated neural effort in most bands, as well as bias in LLM specific vocabulary. p. 3

This trade-off highlights an important educational concern: AI tools, while valuable for supporting performance, may unintentionally hinder deep cognitive processing, retention, and authentic engagement with written material. If users rely heavily on AI tools, they may achieve superficial fluency but fail to internalize the knowledge or feel a sense of ownership over it. p. 138

Perhaps one of the more concerning findings is that participants in the LLM-to-Brain group repeatedly focused on a narrower set of ideas, as evidenced by n-gram analysis (see topics COURAGE, FORETHOUGHT, and PERFECT in Figures 82, 83, and 85, respectively) and supported by interview responses. This repetition suggests that many participants may not have engaged deeply with the topics or critically examined the material provided by the LLM.

When individuals fail to critically engage with a subject, their writing might become biased and superficial. This pattern reflects the accumulation of cognitive debt, a condition in which repeated reliance on external systems like LLMs replaces the effortful cognitive processes required for independent thinking.

Cognitive debt defers mental effort in the short term but results in long-term costs, such as diminished critical inquiry, increased vulnerability to manipulation, decreased creativity. When participants reproduce suggestions without evaluating their accuracy or relevance, they not only forfeit ownership of the ideas but also risk internalizing shallow or biased perspectives. p. 141

Taken together, these findings support an educational model that delays AI integration until learners have engaged in sufficient self-driven cognitive effort. Such an approach may promote both immediate tool efficacy and lasting cognitive autonomy. p. 141