July 21: Pangram and Tips for Detecting AI in Writing

Editor’s note: We’ve launched a new Journalist’s Toolbox page on AI data centers. In includes links to datasets on data centers by country, locations for mapping, etc. We’ll be adding more resources to the page in the coming weeks.

Let me begin by saying there is no perfect tool for detecting AI in writing. But I do use some detection tools as part of a larger formula when evaluating news stories, student assignments, press releases and even customer reviews of products.

Many studies have shown how AI detection tools are unreliable and can return false positives and false negatives with AI text detection. There was widespread panic in academia — trust me, I lived it — in 2021 and 2022 when professors randomly used AI checkers built into tools like TurnItIn.com and found several false positives. Students were suspended, denied graduation and accused of cheating.

But what many journalists and academics haven’t taken into account is how much these tools have improved over the past four years. This was a point Pangram CEO Max Spero made during a Hacks/Hackers AI Real Talk webinar earlier this month.

Pangram is among a handful of AI detection tools I use when evaluating writing. Spero says the fee-based tool boasts some impressive testing metrics for measuring AI detection accuracy rates:

1. It has a false positive rate of 1 in 10,000 (99.999 percent accurate)

2. It has a false negative rate of 1 in 100 (99 percent accurate)

The first number is the one I’m most concerned about. It’s why I don’t rely exclusively on detection tools like Pangram, GPTZero, Copyleaks and WinstonAI when evaluating student work that I suspect was written with AI.

Pangram has a monthly pay model or a $180 annual fee.

I use at least two of the tools to measure the differences in results. I check for sources and citations. Nearly all of the stories require a source to be interviewed, so I contact them to fact-check. I look for suspicious quotes, odd phrasing and often compare the writing to the student’s previous work.

Flaws in the Models
One flaw with detection models is evaluating non-native English speakers’ work. They write in simple sentences or use AI-driven translation tools that sometimes set off the detection tools. I take this into consideration when evaluating my international students’ work. In four years, it’s never been an issue to me.

I also look at the student’s original Google Document of the story and check the previous drafts (press the clock icon at the top of the doc). If 1,000 words suddenly pop into a draft a 4 a.m., it raises more suspicion.

I do allow my students to use AI to write headlines, edit stories and even assist with paraphrasing. That often sets off the detection tool, but typically returns a small percentage of AI detection, which I’m fine with.

Sponsor: visualping

Please welcome our new sponsor, visualping. This is a tool I’ve been using for a long time, and its AI features added in the past few years have taken it to the next level.

So there’s no single, surefire way to detect AI writing, but the aforementioned process gives me a clear picture. I’ve caught a few students over the past couple of years who have violated my AI writing policy in my syllabus. I go over it with them in week 1 and even demo the tools and discuss the process I use for catching them.

How are Models Trained?
Spero, who has a master’s degree from Stanford and previously worked as a Google software engineer, explained in detail on how the detection models are trained on millions of pieces of writing, ranging from news articles to academic journals.

It uses a process called synthetic mirroring, and studies traits of a human-written example vs an AI model. The bottom line question it asks: How confusing is the text to the AI model?

For example, the sentence: “The boy ate a bowl of soup” wouldn’t be confusing to the tool’s predictive model. However, the sentence, “The boy ate a bowl of spiders” would off alarms. This approach establishes a 95 percent accuracy model. The remaining 4 percent accuracy comes from intense training and texting.

A simple YouTube search on “Pangram” includes videos on how to use the tool, but others show how bypass Pangram detection with AI-written text.

Spero said most detection tools can catch copy that’s been “humanized” by online tools. But as Large Language Models get more widespread research to pull from, it makes it more challenging for detection models.

While today’s models are capable of detecting AI, will they keep up with superhuman models of the future? Only time will tell.

Footnote: Spero mentioned on the webinar that Superhuman just bought GPTZero in June. Superhuman tools work to humanize content, so it appears that the company wants to play both sides of the detection coin by adding GPTZero, which was built by Edward Tian when he was a senior in college several years ago.

Read more about OSINT tools and fact-checking on the Journalist’s Toolbox.

Sponsor: Nota

Be sure to check out the incredible production tools suite with our longtime sponsor at HeyNota.com

Revise
A new, browser-based word processor with a built-in AI agent. Revise is designed for complex, full-document edits. It supports every major model (GPT, Claude, Gemini, Grok) in one place, with visual revision history so every AI change is trackable and reversible. Free model with upgrades starting at $8 a month.

We Detect: AI or Not?
A free, no-login image checker that returns a probability rather than a yes/no, and deletes the image right after analysis. It catches roughly 85% of Midjourney, DALL·E, SDXL, and Flux images, and it publishes its weak spots instead of hiding them.

AI for Newsrooms
Everything about AI for journalism in one place — curated for editors, reporters, and local newsrooms. Track real initiatives, policies, research, tools, and a daily newsfeed.

Indicator: Seven Web Scrapers Put to the Test

Higgsfield Explainer
Turn a topic into an explainer video

AI Top Hub
A directory of more than 5,000 AI tools organized by category with ratings and reviews

Data + Journalism, 2nd Edition

Samantha Sunne and I co-authored the 2nd Edition of the textbook, “Data + Journalism: A Story-Driven Approach to Learning Data Reporting” that will be available in August through Routledge and other booksellers. It’s an introductory- to intermediate-level guide to learning data storytelling from A to Z. The second edition features new tools, datasets, exercises and AI tools.

The Journalist’s Toolbox

My book, “The Journalist’s Toolbox: A Guide to Digital Reporting and AI” was published by Routledge in 2023 and focuses on concepts and tools still used today. You can order it here.

"AI can make us faster, smarter and more informed, but it can't make us more caring, empathetic or trustworthy. That's still our job." — Shep Hyken, customer service and experience expert

Follow me @itsmikereilley | @journtoolbox | Subscribe on YouTube | Subscribe to this newsletter