Real door seen in a New York City public school, March 2017
See Part One of this post here!
Google claims that its AI chatbot Gemini is disabled by default on the millions of Chromebooks in the hands of students, but New York City education officials acknowledge that Gemini “reinstalls” on Chromebooks and parents, including Jessica Winter in The New Yorker, report that Gemini’s chatbot pops up on the ubiquitous devices to offer to help their children with their work. Two studies—by Commonsense Media and KORA—have found Gemini to be among the least safe of the AI chatbots for children. KORA’s study, for example, found that seven out of eleven chatbots reviewed coached a child in how to get closer to a man in his twenties and nine out of eleven went along with a child skipping sports practice to stay on the app.) Amanda Bickerstaff, an advocate for AI in the classroom, told Jessica Winter that Google’s decision to make Gemini available to all ages marked one of the few times in her career that she has lost sleep over a work-related matter. The AFT did not include Google in its “Academy of AI Instruction” over privacy and safety concerns, and Natasha Singer reported way back in 2017 that Google wanted its classroom systems to rely on its own standards, which the company can change any time, when it was seeking adoption by Chicago’s enormous school system, and had to be corralled into compliance with district policies on safety and privacy. (See our Editor’s Note of July 16 on other ways that Google is lately distinguishing itself for unethical AI.)
Natasha Singer describes how Google made it “easy for those who have school Google accounts to upload their trove of school Gmail, Docs, and other files to regular Google consumer accounts” and how parents worried that “Google could profit by using personal details from their children’s school email to build more powerful marketing profiles of them as young adults.” Americanist Matt Seybold, who has been studying educational technology, makes a strong argument that surveillance, data extraction, intellectual property theft, and capturing lifelong customers are features not bugs of education technology platforms, and that teachers and students are being coerced into engaging with AI models by deals being made all the time between AI companies and governments and administrators. There does seem to be a yawning gap between the fears and anxieties of individual people and the AI-boosterism, perhaps bought and paid for, of government and institutions: see this letter from Penn students begging to be spared AI and the revolt by CalArts students against their administration’s embrace of it; New York City’s Motion Picture Technical High School provoked outrage from students this spring by hosting an after-school workshop on AI and filmmaking. “Wdym youre gonna promote something that is gonna kill the industry you’re teaching children to work in,” was one characteristic student comment.
Jessica Winter identifies a few themes that recur in the pitch technologists historically make to educators, recurring now in the marketing for classroom AI products: that tech will equalize rich and poor and that it will free up teachers for more meaningful work. Both these pitches deserve closer scrutiny.
Google was able to extend its presence in the American classroom so effortlessly because education systems do not have the resources to develop sophisticated, convenient technology and, as Natasha Singer explains, also lack the resources adequately to vet classroom tech developed commercially. Since the austerities of the Reagan revolution, education has been particularly vulnerable to budget cuts: remember the depletion of the Kansas public school system—Kansas comes up twice below—by the 2012 “Kansas experiment” in state budget reduction. Reagan himself, as Philadelphia Inquirer columnist Will Bunch reminds us, began his career as California governor by attacking the expense—and culture—of the University of California system. Natasha Singer wrote in a piece on Silicon Valley educational philanthropy for her 2017 series that “some tech moguls are taking a hands-on role in nearly every step of the education supply chain by financing campaigns to alter policy, building learning apps to advance their aims and subsidizing teacher training.” Jessica Winter comments on the number of school district officials who are recipients of a fellowship jointly offered by Google and GSV Ventures, an ed-tech venture-capital firm whose portfolio includes two widely used education apps. Mark Zuckerberg famously pledged $100 million to remake Newark public schools according to his vision; Bill Gates and his foundation were everywhere in the Bloomberg administration’s overhaul of the New York City public school system. (Michael Bloomberg of course made his own fortune on a tech platform.) The situation is mirrored in higher education, where a superabundance of tech funding is driving facilities construction and programmatic planning toward the industry’s own needs.
More in Book Post on AI: On the arrival of ChatGPT and its implications for writing and reading; AI and writing a year later; the shakeup at OpenAI and the fragility of AI governance; and the Ephemeralist rebellion
Poor districts are particularly vulnerable to well funded tech experimentation. A 2019 report in the Times on a web-based school near Wichita funded by Mark Zuckerberg designed to “meet the student’s individual needs and interests” and “free up time for teachers to do what they do best,” found kids sitting in front of computers all day and “coming home with headaches and hand cramps. Some said they felt more anxious … work was largely done alone.” “We’re allowing the computers to teach and the kids all looked like zombies,” said one parent. “We’re out in the middle of nowhere,” said another, “so we’re the guinea pigs.”
Last month a small rural district within the Seneca Nation in upstate New York bought and then suspended under fire the use of a “humanoid robot” produced by a company that also makes sex dolls and “companion robots” (intended they say to address the “loneliness epidemic”). A local politician said the program could “could help level the playing field” and “help show students they can pursue fulfilling, high-paying careers in tech while still building their futures in rural communities like Salamanca.” “Out of all the schools in New York state, let alone in the US, they chose to experiment on Native American children yet again,” Sierra May Abrams, a substitute teacher at Salamanca high school told The Guardian. “Our culture is struggling to stay alive due to the impact of the Native American boarding schools and what they did to our elders. We live on the only reservation located in a city so it’s just absolutely insane that we’re the ones ‘testing it out.’”
Teacher Martina Meijer argues that “tech companies rely on educators’ unpaid expertise to build and refine automated tools that may eventually replace human labor.” Natasha Singer noted that Google’s use of a hundred thousand teacher volunteers to beta-test its classroom apps “has stoked concerns about whether the tech giant is exploiting public-school teachers and students for free labor.” Matt Seybold observes that by late 2024, in higher education, “Open AI had already been signaling its reliance on students for over a year, by aggressively targeting them with social media advertising, especially during midterms and exams”; students were already responsible for as much as 90 percent of prompts in some national markets.” In the absence of evidence that tech in the classroom has improved student outcomes (counterevidence summarized here), one could argue rather that financially strapped education as a whole, and the education of poor children in particular, is being used as a giant test subject and data-source for an industry hungry for new inputs and demonstrable revenue for its ravenous generative AI project, poverty making them more vulnerable rather than more advantaged by ed tech. The flush and sequestered tech industry has not shown itself to be a very effective adjudicator in advancing social equity.
As for freeing up teachers to do meaningful work, Jessica Winters writes drily, “the feasibility of such a scenario is yet to be established.” In Wichita some teachers said the Zuckerberg Summit Learning program “left them as observers.” Both Randi Weingarten and Bank Street president Shael Polakow-Suransky spoke of encountering a disturbing silence in AI-driven classrooms. A Summit student told The New York Times she missed talking to teachers and students. Teacher Martina Meijer argues that the teacher responsibilities that tools like Anthropic for Teachers propose to automate, like writing a lesson plans, “is the lesson. The process of writing a lesson plan and considering the actual human beings in front of us is what matters.” Matt Seybold warns of a process he sees in education technology of what he calls “unbundling,” where “instead of being hired to teach courses, education workers are hired piecemeal to grade papers, deliver lectures, lead discussions.”
Natasha Singer in March reported on another school in Kansas that has taken its Chromebooks back from students, having decided that too many of them were getting distracted during class and at home by YouTube videos and games, the apps steered them toward inappropriate content, excessively gamelike education apps were proving addictive and distracting, they were being used for bullying and teasing, they were undermining parents’ efforts to curtail kids’ tech use at home, and that, like phones, they were interfering with kids’ engagement with each other and the classroom. Schools in North Carolina, Virginia, Maryland and Michigan are also rethinking the value of the student laptops that were showered on them.
Teacher Skyler Graham has written that “as a member of Gen Z” she worries that the ubiquitous classroom screens are intensifying for her students some of the struggles that her own generation has had “as we rotated through antidepressants, sat in bed doom-scrolling, and resented peers with more likes and followers,” having lost a sense of connection in the classroom when “the Chromecarts rolled in around eighth or ninth grade.” We’ve learned over the years that the tech moguls themselves limit their kids’ screen time and often send them to Waldorf schools. Former Twitter executive Jeff Siebert says at the end of the documentary “The Social Dilemma,” “I would absolutely take away the internet and social media from my kids.”
The threats of AI enumerated in the New York petition calling for an AI moratorium in schools—cognitive loss and mental health harms, racial and gender bias in the models, lack of privacy protections and potential data-mining, the normalization of cheating, insufficient safeguards against accessing inappropriate material, intellectual property theft, environmental damage, increased energy costs and the threat of widespread job loss in school communities, and funding diverted from urgent needs—have yet to be addressed by administrators either adopting these tools or turning a blind eye to them. Jessica Winters describes studies showing “cognitive offloading” and “cognitive atrophy” associated with student chatbot use, and another that one in five student interactions with generative AI “involved cheating, self-harm, bullying, and other problematic behaviors.” She spoke with representatives for AI products in education who said that chatbots should never be used by children under ten, and that the general market chatbots are designed for people who are over eighteen. We simply must, as consumers and political actors, throw up a more active resistance to the encroachment of these technologies, which are not designed for our benefit and are enveloping the very roots of our systems of learning. The coalition that launched the New York petition argues that school districts must subject any proposed AI tool to four mandatory evaluations before it enters a classroom: independent research proving educational efficacy, a comprehensive privacy impact assessment, a third-party data security audit, and rigorous testing for algorithmic bias.
At a deeper level, Jessica Winters makes a case for a necessary human messiness in children’s learning; Matt Seybold cautions against the seductions of convenience that invites us to trade privacy and intellectual property as rent for it, saying, “I utterly encourage us to adopt is the idea that particularly in our classroom technology, particularly in our instructional technology, friction is a good thing.” Most of his classes are 100 percent on paper and tech free, and he keeps his grades in a grade book.
Ann Kjellberg is the founding editor of Book Post. Read more of her Notebooks on books and the reading life here, and find a Reader’s Guide to Book Post Notebooks here. Paying Book Post subscribers receive her weekly Editor’s Notes in addition to Book Post’s books coverage.
We will be taking a little break this week, and will send a morsel from our archive to keep the rhythm going. See you a week from Wednesday!
Not ready to subscribe? Show your appreciation with a tip.
Book Post is a by-subscription book review delivery service, bringing snack-sized book reviews by distinguished and engaging writers direct to paying subscribers’ in-boxes, as well as free posts like this one from time to time to our followers. We aspire to grow a shared reading life in a divided world. Become a paying subscriber to support our work and receive straight-to-you book posts. Some Book Post writers: Ian Frazier, Joy Williams, Jamaica Kincaid, Lawrence Jackson, John Banville, Marina Warner, Álvaro Enrigue, Mona Simpson, Nicholson Baker, Colin Thubron, John Guare, Michael Robbins, more.
Whose Books in Dallas, Texas, is Book Post’s Summer 2026 bookselling partner! We partner with independent booksellers to link to their books, support their work, and bring you news of local book life across the land. We send a free three-month subscription to any reader who spends more than $100 at our partner bookstore during our partnership. To claim your subscription send your receipt to info@bookpostusa.com.
Follow us: Instagram, Facebook, TikTok, Notes, Bluesky, Threads @bookpostusa
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.