Why I'm Not Allowing My Students to Use ChatGPT

[I wrote this for students and colleagues at my former institution a year and half ago. I still stand behind it.]

The debate over allowing college students to use ChatGPT or other Large Language Models (LLMs) in the writing or research process has largely been an exercise in weighing the educational costs and benefits of the technology. When looking at the issue this way, it is complicated.

On one hand, LLMs have the potential to make plagiarism the norm and to make “writers” into merely passive copyeditors of AI-generated content. On the other hand, LLMs can produce complex work on very specific topics in seconds. For this reason, they can improve our productivity as writers and creators immensely. And, whatever their risks, LLMs are here to stay so we should focus on teaching students to use them responsibly instead of running the fool’s errand of trying to block their encroachment on learning conventions.

If you accept the terms of this debate, the policies being adopted by most colleges and universities seem reasonable. These policies do not prohibit the use of LLMs but try to develop guidelines that limit the risks they involve. As a result, students are allowed to use LLMs but under certain conditions.

Fortunately for me, my wife is a cartoonist who has exposed me to the conversation writers and artists are having about LLMs. This conversation is very different from the one it seems university administrators are having. The reaction of human content creators to LLMs is much more visceral. Many feel strongly that they are being mistreated by LLMs and the firms behind them. Listening to these artists, I have become deeply alarmed by this technology and now think the parameters of the debate in education are misplaced.

There are reasons to object to the use of LLMs in the classroom that have nothing to do with their potential impact on the learning process. It appears that LLMs like ChatGPT were created in a manner that makes them morally objectionable regardless of their educational value. It is possible that an LLM could be created that does not have this moral problem. But, as far as anyone can tell, the LLMs that we are being offered are infected by it.

What is this problem? LLMs derive all their creative power by relying on a “data set,” which in their case is an almost unfathomably large pile of digitized examples of human writing or images (hence, the “Large” in Large Language Model). This data set is collected by “scraping” information off the internet. LLMs use the examples in their data set to predict what the user is looking for when they enter a prompt. Because of the number and variety of examples they can rely on, they are able to produce very complex and seemingly intelligent works. Without this data set, LLMs could do almost nothing.

The problem with LLMs is what is in their data sets and how the firms who built them got their hands on it. Based on the available evidence, a large portion of LLM’s data set is work that is not publicly accessible but available only by subscription. A review of Google’s C4 data set by the Washington Post found that much of the resources contained in it were taken from subscription-only websites. The third most heavily used website in the data set examined was scribd.com, a library subscription service with a huge collection of copyrighted eBooks and magazines. Moreover, the Post found that a significant amount of the data came from websites that themselves traffic in pirated content. At least 28 sites represented in the data set have been labelled by the US federal government as markets for pirated or counterfeit material. If the firms behind LLMs are using content only available behind paywalls without paying for it and using content that itself is pirated, they are engaged in theft. Indeed, a growing number of writers and artists are suing firms making LLMs for copyright violation.

But even if a work is publicly accessible online, that does not give others the right to use it however they wish. The firms that build LLMs are not merely consumers like you and me, nor are they performing a public service. They are engaging in a for-profit endeavor that only very rich investors can participate in and aim to seize an advantage in the marketplace for creative products. The fact that they are using the work of others—including those who will be economically harmed by LLMs—without their consent and without giving them credit for their work is deeply ethically objectionable. Therefore, basically everything in LLMs data sets, whether it was behind a paywall or not, has been misused.

Lastly, we should appreciate the structural relationship the firms making LLMs are creating between their products and the human laborers they rely on. These LLMs are aiming to do work that humans currently do for income. This poses a real danger to the market sustainability of many forms of labor. Of course, labor often becomes obsolete in marketplaces. That isn’t new or necessarily objectionable. But losing one’s source of income because an LLM is offering the same service is uniquely troubling. LLMs can do what human laborers do only because they have appropriated the product of those human laborers without their consent. It is one thing for an individual firm to receive the value of the labor of the individual workers it is engaged in voluntary employment contracts with. It is another thing for a firm to swoop in, appropriate the products of workers employed across an entire sector without their consent, and then plumb those products in a way that makes those workers obsolete and enables the firm to continue to benefit from those worker’s products. The fruits of the labor of entire classes of workers have been vacuumed up by a few big tech firms and are being used to replace those workers in the marketplace while the worker’s labor continues to bear fruit for only those big tech firms. Seen this way, the structural relationship between human workers and the firms that control LLMs is an especially grotesque instance of exploitation.

As I said, an LLM could be created that avoids at least some of these problems. Securing the informed consent of the creators of all content an LLM uses in its data set would help. However, given that the data sets need to be massive for an LLM to be truly useful, and that it is hard to see what a creator would gain by agreeing, securing all these acts of consent appears practically impossible. Still, there may be other forms of generative AI that come along in the future that are so unlike LLMs that they aren’t objectionable in these ways. My concern about LLMs does not apply to all possible forms of generative AI.

From this perspective, allowing students to use LLMs in the classroom represents a failure on our part as educators to show our students what responsible engagement with new technology looks like. Sometimes technologies, no matter how exciting, can be unethical in themselves. We owe it to our students—especially those who intend to pursue careers in engineering or design—to call those technologies out. For their sake, I am telling my students that I do not approve of the use of LLMs like ChatGPT because they have been created by unethical means and aim to establish an unjust relationship between human laborers and the firms behind LLMs. This is a message worth sending, even if LLMs cannot be stopped and even if they have educational benefits.