What Happened When I Asked an AI Agent to Interact with a Chatbot?

I’ve been thinking about an experiement for a while. And the other day, I finally had the chance to run it.

Here was my question: could I use an AI Agent to engage with a chatbot and complete a full “grade the transcript” exercise (shout out to Mike Kentz).

This is what I did:

1. I used my AI Agent. In this case, I used the Claude Chrome Extension, which I link to my paid Claude account. (If you have a paid account, you can use this for your $20/month. It’s extremely easy to set up, and in my experience works much smoother than an AI browser like Atlas.)

2. I provided the AI agent with one of my “pressure bots.” These are custom bots that I built through Playlab AI to put pressure on my students’ ideas.

3. I asked the AI agent to engage with the bot, put the final transcript into a Google Doc, and then mark up that Google Doc to highlight its “metacognition” and to focus on the specific moves it made.

This is the sort of exercise that consultants have long presented as a solution to AI overuse, and a way to build students’ metacognitive muscles.

Here was the prompt I put into the AI Agent:

Not great. But serviceable.

Here’s the bot I asked it to interact with: https://www.playlab.ai/project/cmf941tvl0o7nmy0tvhrto3ts

I created this bot for my Writing Through Literature class at Berkeley, to encourage my students to defend their ideas and to anchor them within specific moments in the story. I was very clear with them: I’m so much more interested in what my students write than what the chatbot writes.

So, for this experiment, I watched as my AI Agent clicked into the custom chabot, completed the exercise, put it in a Google Doc, and then marked up the chat.

Here are some snapshots of what the AI Agent produced:

If you want to see the entire Google Doc, you can check it out here:

https://docs.google.com/document/d/1Yo9oWwMZ9hqJGPIi4tkcol3X9R64b6Ey8nAqtG5es9A/edit?usp=sharing

This is a layered activity, that incorporates a chatbot ito the activity and encourages metacognition.

And yet…

An AI agent was able to do it with very little prompting. I didn’t use any nudging or give additional context, though I certainly could. If I wanted to, I could ask the agent to write the comments differently, to focus on particular sections, or even to highlight sections of the text.

I was recently reading John Warner’s Inside Higher Ed article on Einstein AI. For those who haven’t heard of it, it’s an AI agent that will log into a student’s Canvas account and submit work (including quizzes, discussion boards, and essays) on the student’s behalf.

Here’s the section that really stuck out to me:

As GenAI advances, I become increasingly convinced that the transactional model of education is one of the biggest obstacles we have to adapting to an AI Age. (I wrote about this a few times. Check one article I’ve written about it here.)

In some ways, the AI Agent Movement changes everything. It’s getting easier for students (and anyone, for that matter) to bypass the hard work necessary for deep thinking. (I recommend checking out some of Anna Mills’s work on this. Here’s a great write-up she did on AI agents.)

In other ways, it doesn’t change anything. Making something AI proof or AI resistant is going to get harder, and I have a feeling it won’t be sustainable in the long term. I’d love to be proven wrong, on that front.

What we’re left with (and please push back if you want) is this. We’re left with the tough sell of convincing students that hard work and process are pivotal and worthwhile, and that handing them over to an AI agent or chatbot has pretty big consequences.