Quality Research Shows the Real Impact of AI @ Work

Quote:

“If you know what you’re doing, AI makes you faster. If you don’t, it just makes you wrong faster.”

–Louis Hickman

In this episode I’m joined by esteemed Psych Tech @ Work, Alumnus and AI research machine, Louis Hickman. Our incredible conversation taps into Louis’ myriad research studies to unpack AI’s direct impact on work, domain expertise, and talent assessment.

And of course, this episode also marks the return of the now new and improved AI podcast co-host Mayda Tokens (2.0).

Besides telling dumb jokes- Mayda’s job is to remind us that AI isn’t just a tool — it’s becoming an active participant in how we think, question, and explore ideas.

In the course of our conversation Mayda and I coax some PROFOUND take aways from our friend Louis as he shares the practical outcomes of his research:

Scaling intelligence is easy.
Scaling judgment is not.

The organizations that succeed won’t be the ones that adopt AI the fastest.

They’ll be the ones that:

• Understand what they’re measuring

• Use AI to enhance — not replace — that understanding’

• Maintain control over how decisions are made
  

Louis pushes back on the idea that recent advances represent a fundamental shift in how we measure people. Instead, what we’re seeing is:

• Better models

• Faster processing

• More scalable systems

But none of that replaces the need for valid, reliable, and job-relevant measurement.

One of the most interesting ideas in this episode is how AI interacts with individual differences in expertise.

At a high level:

• For simple tasks, AI helps novices perform closer to experts

• For complex tasks, AI actually widens the gap- allowing experts to perform better
  

  Why?

Because experts know how to ask better questions, recognize when AI is wrong, and refine its outputs—while novices often lack the ability to judge quality, diagnose errors, or course-correct when things go off track.

A major “wow” moment in Louis’ research:

By running the model locally on the same class of hardware, fixing the model and prompt, and turning off sampling/randomness in the settings, you can make the system produce the same output for the same input every time.

At this point, AI adoption isn’t optional—it’s expected. Organizations are being pushed to move faster and scale, while vendors are rapidly building and deploying solutions, often without deep validation.

The resolution isn’t to slow down adoption—it’s to ensure we add and maintaining rigor.

The resolution is to ensure what gets scaled is built on clear constructs, strong design, and validated measurement, so speed amplifies quality—not noise.

The tension is clear: AI can accelerate work, but over-reliance without critical evaluation leads to lower quality, missed errors, and reduced trust.

This shows up in real ways—unchecked outputs, declining attention to detail, and growing skepticism in collaborative work.

The resolution is that AI doesn’t replace accountability—users still need to apply judgment, review outputs, and take ownership of the final result.

Tune in to get the full story on these profound revelations and hear Mayda’s stand up comedy routine.