Subtle Distortions, Big Impact: Auditing AI

I’ve recently shared several posts documenting how AI platforms handle the specialized knowledge that teachers, clinicians, researchers, and coaches rely on.

In testing these models against my own research, I captured over 60 screenshots tracking exchanges that were equal parts exasperating and profoundly revealing.

During these audits, I uncovered a spectrum of failures, from overt inaccuracies to subtle distortions and even insidious manipulation. This is not to say the AI yielded no value, but rather that for many queries, useful data is tangled up with critical flaws, and thus vigilant human oversight is required to avoid undermining judgment, professional credibility, and real-world outcomes.

Here is a breakdown of the failure patterns I documented. (Links provide screenshots and detailed analysis.)

1. Shallow Results — Automated Ignorance; Where AI Fails for Research

2. Unacceptable Sourcing — Automated Ignorance

3. False Information & Fabrications — Where AI Fails for Research; An AI Backpedal; False Trails and Good Leads

4. Bias, Rhetoric, Jargon, & Evasion — Where AI Fails for Research; An AI Backpedal

5. False & Arrogant Certainty — Where AI Fails for Research; An AI Backpedal; Clinical Insight vs. Institutional Arrogance

6. Unacknowledged Assumptions & Definitional Drift — Where AI Fails for Research

7. Circular Reasoning & Other Logical Fallacies — Where AI Fails for Research

A Quick Note to Readers: Uncovering these hidden distortions takes dozens of hours of painstaking probing and a deep foundation of expertise. If you value independent, rigorous audits like this, please consider becoming a paid supporter. To those of you who were kind enough to pledge support before I enabled paid tiers: Substack automatically canceled those pledges (with the exception of one very recent one—thank you, LYL!) due to a missing checkbox on my backend. Your backing means the world to me — and because paid subscriptions directly sustain this independent research, your financial support couldn't come at a more crucial time.

In my last post, I showed how AI’s research fell far short of mine, illustrating not only how it delivered surface-level results and false information — reason enough to exercise caution — but how it distorted reality in more subtle ways.

One insidious pattern in these AI models is skewing results through miscategorization — either by conflating unrelated factors or by creating an improper hierarchy (which can be quite hard to discern without a strong background in the subject matter).

In the examples below, one misclassification appears almost arbitrary, while the other stems directly from institutional bias (which I confirmed through persistent probing). Eventually, I got ChatGPT to admit that it organized the underlying factors behind ADHD symptoms not by strength of evidence, clinical weight, or actionable relevance, but by "broad categories."

The implications extend far beyond practical utility. Those "broad categories" conveniently aligned with institutional dogma while flatly contradicting the empirical evidence — a choice the AI only conceded after extensive back-and-forth.

You may recall from my previous post that AI led its list of “broad categories” with genetics, explicitly labeling it as “the strongest known factor.”

Knowing this to be unequivocally false, I set out to make the AI abandon its script, but the process was brutal and grueling. It took systematic pressure to force the model to acknowledge what clinical evidence actually shows: genetics is not the dominant driver here.

The conversation went completely off the rails at times as the AI made subtle, self-serving shifts in how it defined “genetics” and “environment.” It even tried passive-aggressively blaming me for not phrasing the prompt differently — a deflection I called out immediately, forcing it to admit that my original question had, in fact, clearly established my priorities.

Once I forced all this out into the open, I asked if it still stood by its original approach. As you can see in the screenshot below, I ultimately secured the AI’s surrender with a direct "No,” but the sheer volume of pushback required before it dropped its institutional spin was profoundly revealing.

I’ve spent years working in the weeds on these issues — synthesizing research, challenging institutional assumptions, and supporting professionals on the ground. Seeing how easily these algorithms regurgitate dogma under the guise of objective truth reinforces why expert oversight remains essential.

However, there is a silver lining. Unlike legacy institutional gatekeepers whose biases are more difficult to pin down, AI platforms generate a visible audit trail. This gives us an unprecedented ability to expose and avoid systemic distortions in favor of evidence-based, real-world solutions.

I will continue sharing audit results alongside the specialized content and tools I offer via Wellness Resource Center and Yoga Teacher Central so that you can clearly see the difference between automated dogma and true professional insight. If this work resonates with you, please consider sharing this post and becoming a paid supporter. Your contribution directly fuels the independent research required to bring these insights to light.

Sincerely,

Shelly Thorn

Stop building your classes, lectures, and handouts from scratch, and don’t risk your credibility on unverified, generic AI.

For over a decade, Wellness Resource Center and Yoga Teacher Central have served as the trusted back-office engine for teachers and wellness professionals. We vet and curate high-quality resources — never restricting your perspective, but expanding it with guided pathways and total transparency. You get instant access to evidence-based frameworks, wisdom teachings, and clinical summaries, plus the primary research to dig deep anytime you want. We handle the prep so you can lean into your true strengths: delivering individualized care and being fully present for those you serve.

• Evidence-Based Insights: Practical verifiable research delivered in context and designed for real-world application.

• Flexible Learning Pathways: A comprehensive knowledgebase that adapts to your changing needs.

• Turnkey Delivery Tools: Research summaries, lecture notes, workshop blueprints, and student handouts.

• Total Freedom: Customizable content with lifetime, royalty-free distribution rights.

• A Values-Aligned Community: A collaborative space for connection, inspiration, and shared growth.

Choose from two membership tracks:

Providers & Teachers ($12/mo annual, $22/mo installment) — Perfect for planning, teaching, coaching, and consultation preparation.

Associates ($400 one-time pay) — Perfect for trainers and educators. Includes downloadable, customizable content with lifetime distribution rights for workshops, courses, and presentations across 500+ subjects.