The Power of Conversational Simulations

“There is only one plot; things are not as they seem.” ~ Jim Thompson

This past winter, I began experimenting with structured conversational simulations in my Rhetoric and Inquiry class at a local New Jersey university. For my class, I called them “Friction Bots” and made it clear that the conversation they were about to enter was designed to be difficult.

I built characters and scenes for different skills and topics that we were working through. One tested their ability to identify rhetorical moves. Another tested the ability to back up their thesis. Another tested their ability to explain a theme.

The characters in each simulation pushed back through a series of branching mechanisms I designed into the back-end to make the conversation feel more realistic and adaptable.

At the time, the most important re-frame I felt I needed to get across to students was that AI could be difficult, and not easy or a shortcut, if the experience was redesigned. For many, that is now a well-known development. But at the time, I could tell my students had little perception of what I meant. To them, there was only ChatGPT.

I told them the chat would be hard. They nodded. I repeated. “No,” I said, “I mean, you’ve got a task to do, but it’s not going to be easy.” They rolled their eyes.

I handed them a “student dossier” that laid out the scenario they were entering. In the first one I ran back in January, I aimed to create a diagnostic that would show me their argumentation skills, specifically. In that scenario, the students are sitting in a coffee shop. A fellow student sits down nearby. The fellow student notices them using AI and taps them on the shoulder. He wants to know what they think about AI and Writing.

Their job, I told them, was not just to make a statement but to legitimately engage in an argument with their fellow coffee shop patron. The character was designed to take the opposite side of the argument no matter what. If the student says AI is fine for writing, Fictional Coffee-Drinking Student A says it’s terrible. If they say AI is bad for writing, he says it’s fine.

But don’t just make a statement and walk away, I told them. Actually try to convince him. I’d be watching on the back end, and might give them feedback or asked them questions. Show me how good of an arguer you are. And the fun part? He can be convinced. He can also laugh you off.

“So,” I said, “decide on your thesis before you enter the chat. Jot it down. Come up with some supporting ideas. Or don’t. It’s your call. I just want to see how you handle it.”

A few students prepared, reading the dossier closely and asking me a few questions before beginning. Others dove right in after skimming the guide for no more than a minute or two. They leaned back in their chairs, expecting a breeze.

I built in branches below that. The fictional character can be convinced, I decided, by thorough arguments with logical evidence. That seemed obvious. But beyond that, it can also be moved by the user (my students) acknowledging what the fictional character just said, rather than blowing right through it. In other words, they could have the best argument in the world, but if they didn’t actually listen to the person across from them (and show that they were listening), the character wouldn’t change their opinion.

It felt like a good life lesson, baked into an argumentation diagnostic.

On the flip side, if the student does not acknowledge the opposing argument being made, the fictional coffee-drinking student digs his feet in more. He grows more stubborn, even a bit terse. His facial expressions and body language (written in italics in the chat) tell as much of the story as his dialogue does. They also hint to the student which direction the conversation is going. I was interested to see if they would pick up on the furrowed brow and raised eyebrow of the character.

I watched. The students sitting back in their seats began to sit up, then forward, and ultimately hunch themselves over the screen, their fingers pecking the keyboard with a modicum of ire. Then one of them turned to their friend and said, “Yo, this dude is trippin’.”

The students were engaging on a prototype app I had built with a developer friend. It gave me immediate visibility into their performance inside the simulation. So I opened up their dialogues and began reading.

Some students were slowing down, typing in full sentences, acknowledging the counterargument, using anecdotes and examples to support their thinking. I didn’t give them time to do research, so that was the best they could do in terms of evidence. Others were firing off five-word fragments, poor grammar, little-to-no logic or rationale, zero acknowledgment of the counterargument.

Later, we debriefed the experience. Several students found it interesting, engaging, and meaningful. The ones who found it frustrating (several) said, “He was annoying” or “It felt like he wouldn’t listen.” As we kept discussing, the other students chimed in, “But that happens in real life. That’s the point.” Some mentioned feeling like they wanted to win the argument. Perhaps it was an intellectual trigger, I thought to myself.

Still others relayed some version of, “I thought I knew what I wanted to say, but then I realized I didn’t actually know it as well as I thought I did.” They asked if they could start over and do it again. I said sure.

Later, I asked if they wanted to do more, similar simulations throughout the semester. They said yes. I asked how they would feel about having these conversations graded and receiving feedback on their approach. They said “that’s the only thing missing.” I pushed. “Are you sure? This wouldn’t make you feel uncomfortable?” They looked at me like I had two heads. “No, I want to know how I did,” they said.

Ok, then.

Share

As I worked with my students, I knew we had to try this in other disciplines. So began building simulations for other educators from middle school through graduate school and even into the business world. I partnered with Doan Winkel to begin shaping what this approach might look like in a scalable form.

We ran experiments with 12 institutions in total: 7 K-12, 5 Higher Ed. We ran experiments in high school science, math, English language arts, history and social studies, counseling, and language learning. At the Higher Ed level, we covered writing, entrepreneurship, marketing, sociology, and anatomy. They were not all debates, like “The Coffee Shop Argument.” Some took on a different perspective and tested a different set of skills. It often depended on the subject being tested and a host of other smaller variables.

Along the way, we interviewed each educator that ran the experiments and we surveyed the students. We wanted to know what was working and what needed improving.

I am going to begin sharing these stories with you, dear reader, because I believe there is a great deal of potential in this approach. And I’ve continued to build the prototype into a platform that Doan and I plan to launch this August, just in time for back to school. We hope to serve High School and Higher Ed educators looking for a way to redesign assessments and the classroom experience in the age of AI.

We’re hosting a free virtual launch event on Wednesday, August 19th at 1pm EST. We’ll demo the platform and give you a chance to “spar” with one of our Friction Bots, so you can experience what your students would feel. Register here: [AI Friction Labs Faculty Demo Event Registration]

Hope to see you August 19th.

Share

On a separate note, Aimée Skidmore and I co-authored a peer-reviewed chapter in the forthcoming “GenAI and Higher Education: Ethical Frontiers, Challenge, and Sustainable Pathways“ publication from Elsevier. It comes out July 31st. I have no financial relationship with Elsevier, nor do I make any profit on sales. I haven’t seen the other chapters in the book but will have a chance to read them when I get my copy after July 31st.

Our chapter presents a method for developing AI literacy, along with student data on how the experience changed them. It was a small pilot (21 students, all self-reported) so take the numbers as a first pass, rather than a set of grand conclusions. But we measured three things: skepticism, confidence, and approach to using AI, before and after the activity. We asked – did it make students more or less skeptical of AI? More or less confident? And would they change how they approach the tools going forward? If so, how?

Our thesis was that students who self-report increases in both skepticism and confidence can be considered to have built a higher level of fluency as it pertains to the tools. But skepticism and/or confidence alone doesn’t mean fluent, literate, or aware. For example, an unskeptical and highly confident person could reasonably be considered less literate than a meaningfully skeptical and highly confident person. In the latter case, the confidence is born of the skepticism itself. “I’m confident because I know not to trust.” In the former case, the confidence is born of ignorance. “I’m confident because I don’t believe there is anything to worry about.”

The cherry on top is the approach. Eighteen of the 21 students changed their approach to AI as a result of the activity. The “how” varied. Some said they now think more deeply about what they write into the chat window itself. Some said they trust the outputs less. Cross-reference all three and you have the beginnings of a portrait. An AI-literate person, or a not-literate one.

I’m pretty proud of the work Aimee and I completed. If you do end up picking up the book and have some ideas to share, I’d love to hear what you think. Don’t be a stranger. It gets lonely out here in the desert. Stay tuned. And thanks for being here.

I just finished watching Season One of The Lowdown on FX, a semi-farcical Oklahoman noir from Sterlin Harjo (Reservation Dogs), starring Ethan Hawke. It’s based loosely on the life of Lee Roy Chapman, a Tulsan citizen journalist who famously unearthed connections between one of Tulsa’s founding fathers, W. Tate Brady, and the Tulsa Race Massacre of 1921, among other notable news stories.

The show leans thematically on the work of Jim Thompson, an Oklahoman crime noir author from the 40s and 50s who I was only vaguely familiar with via the commentary of Stephen King [“Big Jim didn’t know the meaning of the word stop. There are three brave lets inherent in the forgoing: he let himself see everything, he let himself write it down, then he let himself publish it.”] Thompson is most well-known for The Killer Inside Me and for reinventing the stylistic norms of noir, even though his work was not widely acclaimed during his lifetime.

I only recently realized how much I actually love noir; I took a class called “Film Noir” in college without having any clue what I was stepping into. Twenty years later, it has dawned on me that its aesthetics are right up my alley. I am also watching Sugar on Apple TV.

In any case, Thompson’s line from the epigraph feels true to life. More than just a commentary on the craft of fiction writing (or a fun, sly comment to make on your way out the door, which I will most definitely be doing going forward), “things are not as they seem” appears to be the plot of every news narrative, stand-up comedy bit, dinner table story, cocktail party anecdote, and speech I have ever consumed. We can’t escape it. If that seems like a grand statement, consider the alternative. Have you ever consumed a story — real or fictional — where things were as they seemed? I’d venture a large bet to say “nay,” because if that were the case, you wouldn’t have a story at all. You’d have a sentence. A pretty boring one, at that.

And that’s how I feel about the AI in Education discussion. Things are not, in any way shape or form, as they seem.

Indeed.