Thirty Minutes

A standalone chapter of Beyond the Model. The Same Way Every Time, Part One of Three.

Cooper came prepared to navigate the Applied AI Expo like a local navigates the Las Vegas weather. There were twenty-one parallel industry tracks that ran across two floors. He watched a river of lanyards move up and down the escalator between the floors while the bright lights of the giant video wall cycled through examples of AI agents handling tasks like closing a purchase order, or reading a chest x-ray. Other agents were handling the negotiations of freight rates, while an old-fashioned automated tool was softly playing a piano underneath the din of it all.

He was there because the firm had made the conscious decision to become its own first client. Cooper’s belief was that if you wanted to know where the legal industry was going, he needed to see the other industry who hit the wall first and take notes on what those industries built as a result of it. Oakes approved the trip without questioning it. Nora gave him the task of identifying anything that had a price tag. Maya asked for anything with a failure mode.

By the second afternoon he had filled six pages with notes on what he saw. Five of those pages were repetitive descriptions of the same demo performed by multiple presenters.

The performance never seemed to vary. A presenter walked the stage wearing a headset mic, presented a problems statement, and then let an agent loose on the problem while the audience watched the screen. The agents performed well, and didn’t look like simple demo magic that Cooper had built up the skill to identify. He wrote in his notes that the agents found the anomaly in the shipping manifest. The agent effectively drafted the outreach sequence. The agent pulled the market summary together along with the proper citations. The process looked absolutely brilliant, and the demo ran exactly one time. Never twice. In his two days on the floor, he’d seen some thirty sessions, and he had yet seen anyone attempt to run the same job as second time to show if the models could do it on a consistent basis.

He wrote that down too, and drew a box around it.

Unlike most of Cooper’s previous conferences, the legal track occupied a single modest room on the second floor. He gave it a fair chance, and attended both of the morning sessions. Both of those sessions he could have presented himself from memory. After sitting through the session on legal industry AI tools, and a demo on contract-review workflows, he decided it was time to excuse himself from the familiar and head down to the bigger rooms where whole industries were talking about their plumbing.

He had an hour before the afternoon keynote and decided to look around for something that would catch his attention. On the ride down the escalator, he overheard two developers arguing over which session they would skip or attend next. One developer’s answer gave Cooper the answer he was looking for.

"I’m headed to the insurance room. The claims woman. She runs the agent process multiple times."

He fell in behind them.

The insurance-operations track room was much larger, and had a lot more attendees. The session description on the door read Same Input, Same Answer: Agent Consistency in Claims Operations. The presenter was already on the stage setting up her own equipment. Plugging in the laptop into the podium while the room’s tech lead looked over her shoulder approvingly. She was a woman in her forties who had the calm of someone who had done the thing she is about to describe many times before. The muted colored slide behind her said her name was Dana Whitfield, found and chief product officer at Coverline, a company which built claims-operation software for mid-market insurers.

The demo jumped right into the job.

"When a claim goes sideways, somebody has to build the record," she said. "The policy language to the claim file to the adjuster's notes. Every email anyone may have sent about it. Then somebody has to read all of that and write up the details of what actually happened and then what we do next. My customers call this an escalation investigation. A regional carrier usually runs forty of these a week. Before last year, each one took up half a day of a senior examiner's time. My company's entire pitch on this is that we sell them that half a day back."

She clicked through to the next slide.

"We point a model at it, just like everyone at this conference. And the model worked great. The first investigation it wrote was already better than our template. But the fourth one came back in a new format and contained two opinions nobody requested along with a section that does not exist in our process. Investigation number nine suddenly brought in an email thread from a different claim. The report that comes out of this process gets attached to insurance coverage decisions and, eventually, it is tagged in litigation. We have a word for a system that produces wonderful work most of the time. Uninsurable."

The room chuckled. Dana waited it out with the patience of an experienced comedian who knew how to play the room.

"There's a catchphrase going around the industry about all this. You've seen it in the presentations this week." The slide behind her transitioned into a single sentence in plain type: It's not the model, it's the harness. She let it sit there for a few seconds. "Catchphrases are free," she said, and clicked past it. "Now, let me show you the code."

The next slide brought up her definition, and Cooper copied it word for word.

A harness is code you write around an agent to make it good at one specific, repeatable job.

"It does one job," she said. "Specific. Repeatable. The harness I built investigates escalated claims. And that’s all it does. By design, it cannot do anything else. I’m going to spend the rest of this session explaining why it cannot."

Then she fired up a terminal on her laptop, and the room finally got the presentation it had not seen all week.

She worked from a notepad and pasted a single link, a claim number she pulled from the demo dataset, and then hit enter. Instantly the screen divided into three terminal panes. The left displayed a log that began writing itself. Each line timestamped to the second. The center display held the work of the agent as the audience watched it request the policy record. The agent made its request for the policy record, and a layer between the agent checked the request, approved it, ran the process, and then handed back the result. The right panel held a folder tree, filling those folders with policy excerpts, claim timeline, and correspondences, all sorted as the run progressed. Each file was consistently named in a standard format and saved to disk as soon as it was gathered.

"There are four systems here," Dana said, while the process continued. "Policy administration, the claims database, the adjuster notes, along with the mail archive. Those are the only four doors in the building. Each one of them is read-only. The agent is allowed to ask for anything it wants. However, my code decides what the agent is actually allowed to reach. There is a budget assigned on this run. We monitor tokens and dollars and minutes, when the budget is gone the run ends whether the agent feels like it finished or not. Everything you're watching on the left is the log, which my regulators can read, my customers will read, and my lawyers have already read."

The whole procedure took about four minutes to run. The right-had pane displayed the final report. There were seven sections in total. All in a fixed order, with each finding linked to a file in the evidence folder. She expanded one of the rows so even those in the back of the room could see it. Finding 4: Adjuster requested an engineering inspection on June 6. No inspection appears in the files. Source: correspondence/2024-06-06_hendricks.eml. The room understood that finding, which Cooper thought was exactly why she picked this claim as the test case.

The room began a polite round of applause. Dana raised a hand and stopped it.

"That's the demo you’ve seen everyone give," she said. "Now, here’s ours."

She copied and pasted the same link and hit enter again.

The room went quiet in a different way, the way a room does when it realizes things just got interesting. The log displayed again. The folders filled again. Four minutes later the second report appeared right beside the first, and she adjusted them on screen together: the same seven sections, the same findings, the same citations to the same evidence files. She scrolled them in parallel, top to bottom, and the two reports tracked each other down to the individual line, with a single sentence in section five worded differently.

"Same input, same answer," she said. "I've run this specific claim over two hundred times now, because this is my test case, and every one of those two hundred reports resides in a folder my QA team audits. The variation you just saw highlighted is the largest variation we've recorded this quarter." She lowered the laptop screen half an inch, a small punctuation. "Everything else you’ve seen at this conference is a talent show. What I needed was an employee."

A hand shot up near the front asking which model was being used.

"That's usually the first, and least interesting question about this system," she said, and went on to answered it anyway, naming the frontier lab and moving on. "We've swapped models on this twice since launch. The reports came out the same, because the shape of the work comes from the harness we built. If you take one sentence out of this room, that’s the one."

Dana spent the rest of the session going through individual slides explaining what her product was not allowed to do. Cooper found it to be the strangest sales pitch he’d ever seen.

The tool cannot browse the internet. It cannot send email. It cannot write to the four systems it reads. It cannot exceed its allotted budget. It cannot skip the log. It cannot produce a report outside of the seven sections it was assigned. It cannot delete or edit a file from the evidence folder, including trying to correct its own mistakes. That stays in the record along with a correction noted beside them.

“I vibe coded the first version in about thirty minutes,” she said. “I did that on a Saturday morning while on my first cup of coffee. There were a total of sixty lines of code built around an agent kit that I downloaded online. Any of you can get that same kit. That version worked pretty well. And by pretty well, I mean it did the job about four times out of five. Eight percent is pretty good, but not something I’d stake my job or reputation on. The next slide held a calendar. “The version I put in front of you took another three weeks. That time was spent on the harness, and not the agent. The harness helped me with the restrictions, the budget, the log, the claims testing we ran each night to monitor any drifting. So, thirty minutes will get you a nice demo, but it took three more weeks before I actually had an employee.”

Another question from the audience asked what the hardest part had been. She had to think about it.

"The biggest things was the model fought me," she said. "Not in a bad way. But it ended up being helpful. The AI kept offering to do more. It would offer to email the adjuster for clarification. It offered to check a claims forum that was outside of its scope. I mean, every offer was reasonable, but every one of them was an option I had already decided to prevent. I spent a week teaching the code to say no on my behalf, so I didn’t have to keep saying it at two in the morning."

A woman near the back asked whether the process had ever failed anyway, walls and all.

"Yes. Back in March," Dana said, without hesitating." There was a rider form we hadn’t seen before. The report came out and it looked good, but it had one finding that was filed under the wrong section. One of the reviewers caught it within two minutes, because the output itself didn’t match the consistent formatting we were used to seeing. So the shape of the output just didn’t match what we were seeing from all the other results. I now have a line in my checklist that exists because of that failure. The system will not always be correct. The walls we created make it stand out in a shape that shows where it goes wrong.”

Cooper realized that the first six pages of his notes were no longer relevant. The two pages of dense ink from the past forty minutes were going to be the basis for most of his report back to the firm. As he was reading through, he started to see a pattern of something familiar. There was a budget with a hard stop. Door, and who could go through them. A log that couldn’t be touched. A named person who was responsible. He realized he’d read something similar from his own guidelines.

Cooper took his pen and marked in the margin with an arrow pointing along with a question mark: Our rules, compiled?

Cooper caught up to Dana as she was standing at the side door and the next speaker was setting up. He introduced himself and named the firm he was with. She looked at his conference badge and a smile crossed her face.

“Ha. Legal. I was wondering if any of you would make your way over here. Half of my list of restrictions exists because of my legal department.”

"I have a question," Cooper said. "We have a test we use at the firm, and I wanted to see what you would do with it. If your system files a wrong report one one of the next runs, and that wrong report costs somebody money, whose name is on that?"

"Mine." No hesitation at all. "This thing runs under my authority, my name literally prints in the footer of every report, and there is someone on my team who reads every single one before a customer does. When the report is wrong, the log shows me exactly what it read and exactly what it did, and the correction goes in the folder right next to the mistake." She tilted her head. "How was that? Did I pass?"

Cooper hesitated, then he asked a favor. "You mentioned you have a checklist. The three weeks, the doors, the budget, the nightly test claims. Does that exist somewhere?"

"Yes. It's twelve whole lines. I keep it boring for a reason. Every exciting thing I ever added to this has turned out to be a mistake." She had already pulled out her phone. "Got a card?"

They exchanged QR codes. On the escalator ride down he passed by the video wall again. The same agent he’d seen before was doing something with a supply chain. He viewed that differently this time, as the river of lanyards also looked different to him. There we two thousand demos and talent shows at this conference. He now saw one employee.

His room faced out over the Strip, and the window lit up with the lights of Vegas and held back the heat radiating even in the dark. He grabbed his notebook and laid it flat on the desk.

At the top of the page he place the date, the title of the session, and Dana’s definition. He copied them clean. Under that he placed a short line and spoke the year as he wrote it down. He wanted this to be his prediction, and he wanted to see if he believed it as he said it.

Two hundred runs of the same claim, filed in a folder the auditors actually read. The demo I trusted and believed was the demo that behaved the same way twice.

And directly under that:

We’ve spent this year hiring AI agents. We still need to write the job descriptions and sign off on the results. That work has a year assigned to it now. Start the 2027 file.

Her email came in as Cooper arrived at the gate the following day. He nodded in approval reading the subject line of THE BORING LIST. There it was. Twelve lines as promised. Numbered and no adjectives. As he read it for the second time, he forwarded it to Jesse along with the session’s title and booked a thirty-minute session with him for Monday morning. In the note field he put: Make sure to bring the whiteboard. I found something interesting in the insurance track of all places. As he stood in line with his boarding group, he tapped his notebook with his finger and thought about how his latest bit of knowledge made consistency sound like the most exciting thing to come.