AI chatbots right about daily news nearly all the time, but fail badly when users slip wrong details into chats

Paper

Evaluating Commercial AI Chatbots as News Intermediaries

Authors

Mirac Suzgun, Emily Shen, Federico Bianchi, Alexander Spangher, Thomas Icard, Daniel E. Ho, Dan Jurafsky and James Zou (Stanford University, Together AI)

Published

May 2026

AI chatbots are now strikingly accurate on same-day news. The best systems in a new evaluation hit more than 95% accuracy on questions about events that had broken hours earlier — a clear jump over prior real-time benchmarks. But when users ask their questions while misremembering a small detail, accuracy on the leading frontier models can fall as low as 19%. GPT-5, which powers the most widely used AI chatbot in the world, gets roughly four out of five such adversarial questions wrong.

That's one of three failure patterns documented across 12,600 model-question instances over 14 days of testing. Every model evaluated performed worst on Hindi-language questions (79% vs. 89–91% elsewhere). Retrieval failures — not reasoning failures — drove more than 70% of errors. And the chatbots most likely to credit original reporting may be the ones least respecting publishers' wishes about being scraped.

What the researchers did

The Stanford team evaluated six chatbots — Gemini 3 Flash and Pro, Grok 4, Claude 4.5 Sonnet, GPT-5 and GPT-4o mini — on questions generated daily from BBC News reporting between Feb. 9 and 22, 2026, across six regional services (US and Canada, Arabic, Afrique, Hindi, Russian and Turkish) covering populations of more than 2 billion people. Each question was multiple-choice with five options, targeting specific verifiable details — exact quotes, named entities, specific figures — from articles published within the previous 24 hours. All six chatbots answered every day's questions in parallel, producing 12,600 model-question instances from 2,100 questions in total. A separate set of test questions subtly altered factual premises to test whether models would catch the error or amplify it. The BBC was chosen for its well-indexed regional coverage; the researchers note this likely represents a best-case scenario.

The findings

Accuracy is high but uneven. Gemini 3 Flash (95.6%) and Grok 4 (95.0%) led. Gemini 3 Pro (93.7%) and Claude 4.5 Sonnet (90.4%) cleared 90%. GPT-5 trailed at 85.0%; the older GPT-4o mini reached only 69.0%. A Gemini 3 Flash user encounters an incorrect answer roughly once every 22 questions; a GPT-5 user encounters one every seven. Given ChatGPT's roughly 800 million weekly users, that gap matters at scale.

The Hindi gap is severe and systematic. All six models scored lowest on Hindi questions — 79.3% vs. 88.9–91.3% elsewhere. The failure isn't language comprehension; models generate fluent Hindi and reason competently in it. It's retrieval. When asked Hindi-language questions, models cited English Wikipedia more frequently than any Hindi news outlet, pivoting to English sources covering the same topic but reporting different specific details. The same wrong substitutions appeared across all four providers, pointing to a shared bottleneck in multilingual retrieval infrastructure rather than any one model's weakness.

Retrieval, not reasoning, is the dominant failure mode. When models retrieved the right source, they almost always extracted the correct answer. The problem is landing on the right source. Disabling web search degraded accuracy by 31 to 46 percentage points — without live retrieval, none of these systems would be reliable enough for news.

High accuracy doesn't mean robust accuracy. When questions were phrased accurately, the four frontier models were separated by 8 percentage points. When researchers slipped a subtly wrong detail into the question, the spread exploded to 51. Grok 4 held at 70% accuracy. GPT-5 dropped to 19%, slightly below the 20% it would have scored by guessing at random on the five-option test. The false premise didn't just confuse the model; it actively misled it toward wrong answers about four out of five times.

Original sources are rarely credited — and the citation gap may reflect scraping behavior. Grok 4 cited the original BBC reporting 28.5% of the time. Claude 4.5 Sonnet, GPT-5 and GPT-4o mini effectively never cited the BBC (0%, 0.2% and 0%). The researchers connect this directly to BBC's robots.txt enforcement: providers that comply cite BBC less; those that don't, cite it more. The paper specifically names xAI/Grok as "publicly flagged for scraper-like evasion behavior," citing DataDome research that a single Grok URL-fetch request produced 16 requests from 12 different IP addresses, none identifying as xAI or Grok, while rotating through ordinary browser user agents. Grok's higher BBC citation rate may reflect a more permissive crawling posture rather than better retrieval. Seven of the top eight most-cited domains overall are primarily English-language, with English Wikipedia at the top.

What this means

Aggregate accuracy is misleading. A 95%-accurate model sounds reliable in a way a 70%-accurate model doesn't. But high accuracy can coexist with systematic failures that affect specific populations, specific question types or specific source ecosystems. Editors evaluating AI tools shouldn't trust top-line accuracy without asking where the errors fall.

Local journalism in non-English markets faces a particular threat. The same mechanism — AI substituting English-language sources when local-language retrieval fails — likely applies to any market where local journalism is poorly indexed. For newsrooms serving non-English audiences, AI discoverability may now be as important as SEO was a decade ago. This extends the visibility concerns documented in earlier research on bot-blocking and publisher traffic.

The robots.txt question is more complicated than it looks. Publishers blocking AI crawlers already suspected what this paper confirms: refusing to be scraped means AI chatbots rarely cite you, even when they're effectively answering questions from your reporting. The paper also documents that at least one provider appears to evade robots.txt through IP rotation and user-agent spoofing, meaning a publisher's choice to block isn't necessarily honored. The chatbots most likely to credit original reporting may be the ones least respecting publishers' wishes. That strengthens the case for licensing frameworks that preserve attribution without forcing publishers to choose between visibility and uncompensated extraction.

User error is a verification problem now. Users who ask AI chatbots about news while misremembering details will frequently get confident answers that reinforce the error. If AI chatbots are increasingly where people verify what they think they remember from the news, journalists need to think harder about how accuracy travels through that layer.

Important caveats

The BBC is unusually well-indexed and editorially consistent across languages — a best-case scenario that almost certainly inflates these numbers relative to less prominent outlets. Multiple-choice tests inflate accuracy. When the researchers removed the answer choices and asked the chatbots to answer in their own words, accuracy dropped 16–17 percentage points on average across the six models, with individual models losing between 11 and 22 points. Real users ask open-ended questions, so real-world accuracy is likely lower than the headlines suggest.

The citation findings are also complicated by legal compliance: low BBC citation rates from Claude, GPT-5 and GPT-4o mini may reflect respect for anti-scraping policies rather than retrieval weakness.

Also, one of the authors holds a Google PhD Fellowship, and Google's Gemini 3 Flash topped the rankings. The methodology evaluates competing providers in parallel, but the funding is worth noting.

The bottom line

This study shows AI chatbots — already a layer between hundreds of millions of people and the news — are both more capable than previous research suggested and more fragile than aggregate accuracy implies.

The fragility falls disproportionately on Hindi-language users, on users who pose imperfect questions and on the journalists whose reporting is consumed but rarely credited.

The patterns of who gets reliable information and who doesn't are being set now, inside retrieval pipelines users can't see. Making those patterns visible is a precondition for shaping them.