How the best legal teams supervise agents (incl. practical tips)

Hello! I’m Laura, a Legal Engineer at Flank. I build agents and supervise them, making sure that we’re putting as much trust in them as they’ve actually earned.

Most conversations about legal AI focus on whether the agent got it right. This piece is about something that took me a while to figure out - whether the way we’re supervising is actually working.

You have an AI agent assigned to review an NDA. It flags 4 clauses out of 40 as non-compliant. Instead of jumping straight to those 4 clauses, you end up trawling through the entire document, top to bottom, “just to be sure”. By the time you’ve completed it, you’ve redone the entire review yourself - those 4 flags raised by the agent are never approached differently from anything else.

One of the easiest traps in agent supervision is spending so much time checking an agent’s output that you end up redoing the work yourself. This instinct isn’t irrational either. In a report published by Thomson Reuters, one US attorney stated: “Agentic AI, while exciting, to me removes oversight a step too far. I like the idea of prompting and reviewing a result”.

We’re clearly having to find the balance between two instincts, either: I don’t trust the agent being able to find every issue in this contract, so I’ll review the whole thing myself anyway - or, I have an agent configured with the perfect prompt and instructions, so I’ll hand off all of my work without a second thought.

Despite the growth of AI in legal, most legal teams haven’t fully reached a point where they can offload the bulk of their work to an AI agent. In fact, the same Thomson Reuters report found that only 16% of law firms and 18% of corporate legal teams use agentic AI tools.

Supervising agents is the blocker standing between where they are now and actually adopting agentic AI.

In legal AI, agent supervision is the process of a human checking an agent’s work before it’s relied on and sent to a counterparty. The goal isn’t always to avoid reviewing the whole task, but to make the level of supervision proportionate to contract risk and the trust the agent has actually earned. Fundamentally, the agent does the work, the human checks for exceptions, without needing to complete the task from scratch.

In the context of contract review, that might look like:

That’s the model in theory. Nowadays, most people (and most tools) treat supervision as something closer to blanket quality control:

I used to think that supervising agent outputs meant finding mistakes, but I was wrong. The value of supervision isn’t in the catch itself, it’s that the catch improves the next hundred outputs. It’s a feedback loop.

A pivotal moment in my time as a Legal Engineer was understanding that there is a clear difference between reactive and proactive supervision.

• Reactive supervision is fixing an error after it’s already been spotted.

• Proactive supervision is noticing that a playbook built to review your own template doesn’t work for reviewing a counterparty’s template, and rewriting the playbook before that mismatch causes a string of errors, not after.

It’s easy to fall into the trap of favouring reactive supervision simply because it’s easily measurable. For example, let’s say an agent’s output comes back with 10 redlines. If you mark 2 as genuine misses by the agent, you’ve got a clean 80% accuracy figure. Proactive supervision, on the other hand, is rarer because the biggest wins are invisible. Nobody saw you rewrite that playbook before it caused a single problem. Likewise, you can’t score a mistake that never happened.

I saw this play out with an indemnity clause. There was one contract that needed an indemnity clause but didn’t include one, and the agent kept missing it. At first, each miss looked like an isolated slip-up. It was easy to mark as a one-off correction and move on. But the more I started logging why the agent was missing it, the pattern was obvious. The playbook only told the agent to review the indemnity clause if it was already sitting in the counterparty’s contract. It never told the agent what to do if the clause was missing entirely. Three misses weren’t three isolated mistakes. They were one instruction the playbook never had in the first place.

Share

So, how can proactive supervision actually be implemented in a workflow?

Start with this: every Monday morning, before reviewing any new agent outputs, spend 15 minutes scanning last week’s corrections. Tag each one with a short reason - wrong fallback, missing context, incorrect analysis - whatever makes most sense to you. If the same tag appears three times or more, that’s not three unrelated mistakes. You need to flag it, fix the rule behind it, and continue your review.

Within a few weeks, you stop seeing one-off misses and start seeing patterns. Updating a playbook because you caught a pattern is proactive supervision. Updating it only after being prompted by a client to do so is reactive supervision in disguise. The difference really isn’t the tool, agent or system you’re using. It’s whether your supervision learns from its mistakes.