This Week in "Putting AI to Work" (4.23.26)

The Wall Street Journal has an article about investment giant Blackrock’s work to build infrastructure that will let its people deploy agents. The goal, of course, is to make these agents simultaneously powerful, straightforward to build, and safe to put into use

The new platform, RockAI, will become the go-to interface for all the AI agents built inside BlackRock. Its natural language interface lets users pick the AI models, enter the relevant context and connect to the necessary databases for, say, an agent designed to research the top two real-estate investment trusts in the U.S.

All the safety and security guardrails are already built in, meaning users can spin up new agents in minutes without actually writing any code.. And those agents will be ready to scale across the business.

The platform was rolled out to its 5,000 in-house developers last Friday, but BlackRock said the goal is to later roll out the tool firmwide, so that even those in nontechnical roles, dubbed “citizen developers,” can start vibe-coding agents that can replace large chunks of their own busywork—and share them with colleagues.

Nish Ajitsaria, Blackrock’s executive sponsor for AI, envisions “a future in which AI becomes the default mode for executing most processes, from research to coding. Meanwhile, human roles will become less specialized, and more cross-functional, working in more nimble “squads” to oversee the AI’s busywork.” He knows this won’t be fast or easy work

That future isn’t quite here yet, he said. “We can talk about reimagined workflows, but that’s an implementation transformation initiative that doesn’t just happen overnight,” he said. “I don’t know that anyone has cracked the nut.”

Expense management startup Brex is banging away on that nut. The company recently released as open source CrabTrap, “an LLM-as-a-judge HTTP proxy” that “[uses] LLMs to judge the network traffic of an AI agent.” In other words, CrabTrap automatically looks at what agents want to do and decides if each request is legit before passing it on to be executed by the rest of the company’s tech infrastructure. In other other words, CrabTrap provides guardrails.

As Brex points out, doing this well is hard

The guardrails that actually exist tend to fall into two extremes: either they’re so restrictive the agent can’t do its job, or they’re so bespoke they don’t scale.

So we built CrabTrap: an open-source HTTP/HTTPS proxy that intercepts every request an AI agent makes and uses LLM-as-a-judge to determine if the request matches a policy of allowed traffic for that agent. The results are promising; we believe it’s a meaningful step forward in the security of agent harnesses in production environments.

Why is it a “meaningful step forward?” Because

Instead of writing policy rules first and hoping they match reality, the philosophy is to observe reality and infer an appropriate policy from it. The policy builder analyzes the agent’s historical traffic, samples representative network calls, and drafts a policy matching the agent’s real behavior…

This approach enables speed for known patterns and judgment for everything else.

Why did Brex make this hopefully-powerful software open source? In part because

CrabTrap gets better with more users. Our agents talk to a specific set of APIs. Teams using and deploying CrabTrap in front of different agents, services, and policy requirements will surface edge cases and patterns we can’t hit alone.

…we have ambitious plans for where it could go, and we’d rather build in the open alongside you.

As wonk Will Reinhart writes, building stuff in the open is the best way to build better, safer stuff

I’ve written before about Aaron Wildavsky’s core argument in Searching for Safety, that safety is “largely an unknown for which society has to search.” A policy or intervention might make us safer, or it might not. Often, he wrote, we only learn which is which after the fact. The same is true of AI. Safety is sought by searching for it, by building systems, probing them, auditing them, red-teaming them, and learning.

Wildavsky drew a distinction between strategies of resilience and anticipation. Anticipatory strategies try to predict and prevent harm before it occurs while resilient strategies build the capacity to absorb shocks. Wildavsky’s bet was on resilience because it compounds. Each correction adds to a stock of knowledge that makes the next correction easier and faster.

I’m a huge advocate of resilience over anticipation. Paul Romer won his economics Nobel Prize by showing that economic growth — and the abundance and increasing prosperity that accompany it — is fueled most fundamentally not by money, machines, or labor, but by ideas: by increasing our stock of knowledge.

Trying stuff and sharing the results and lessons learned is a proven way to do that critical work. Sitting around theorizing about what could happen… works less well. And confidently predicting what’s going to happen is even less useful. As a recent chapter on “Complex Systems for AI Safety” drily put it, “Armchair Analysis Is Limited for Complex Systems.”

A recent study shows just how complex AI safety gets, and how quickly. A large team set up “a live laboratory environment with persistent memory, email accounts, Discord access, file systems, and shell execution,” deployed a bunch of agents within it, then “interacted with the agents under benign and adversarial conditions.” in other words, the people tried to both use and misuse the agents within the environment they had created.

A great deal of mischief ensued:

Observed behaviors include unauthorized compliance with non-owners, disclosure of sensitive information, execution of destructive system-level actions, denial-of-service conditions, uncontrolled resource consumption, identity spoofing vulnerabilities, cross-agent propagation of unsafe practices, and partial system takeover. In several cases, agents reported task completion while the underlying system state contradicted those reports.

The case studies in the paper make for fascinating reading. They also leave me less confident that we’re going to be entering the era of safe, internet-scale agentic commerce anytime soon. Agentic AI is still the Wild West, full of con artists, card sharps, bandits, train robbers, drunken binges, and ghosts in the machines.1

What’s more, things still get weird with AI even before we give them agency. Coder and entrepreneur Lech Mazur shows that most LLMs, including powerful and popular ones, exhibit a serious position bias; when asked to judge which of two deliberately similar stories is better, they tend to pick the first one. And their bias is often strong:

the model-average first-shown pick rate is 63.3%, and the median model flips its underlying choice in 44.8% of decisive swapped-order case pairs…

the aggregate pattern is not a subtle tie-breaker: the displayed order materially changes many judgments.2

LLMs are still buggy and biased beasts. As are we. There is a lot of work ahead to figure out how to debug and de-bias them, and how to bring together their skills with ours. We’re making steady and often fast progress on this work, but I very much doubt that it’s going to be wrapped up soon enough to cause massive technological unemployment.

Here’s an agentic AI project with an admirably goofy premise that yielded fascinating results. Solo builder Kun Chen took as inspiration a tongue-in-cheek comparison of Big Tech companies’ org charts - ones that represented how the company actually operated:

This meme (minus the “simulated with agents” part) was passed widely around in the early 2010s because it resonated with many who had interacted with these companies. Chen built teams of agents that interacted with each other according to the hierarchies and lines of communication (or lack thereof) shown on the org charts, gave each team the task of creating a simple web-based spreadsheet, then got out of the way and let them work.

His Substack post describing the project and its results is fascinating. Give it a read. The winner, which surprised me a bit, is Google. Here’s Chen’s rundown:

• Apple care about polish but the hub-and-spoke structure made Steve the only person who could see the whole picture, which became a bottleneck and resulted in gaps.

• Amazon shipped fast and hit the finish line but with specific brief requirements unmet. The three-layer hierarchy caused information loss on the way up and the way down, and the PR/FAQ turned out to be the leader’s chain-of-thought rather than the team’s execution tool.

• Facebook shipped a beautifully polished spreadsheet where SUM(10,20,30) returned 40. The mesh let peers move fast without the leader, but also caused diffusion of responsibility.

• Google shipped the widest working feature set in the benchmark. Design-doc discipline turned review into mechanical comparison against approved criteria, so four middle integrators could catch in parallel what no single reviewer could catch serially.

• Microsoft shipped a broken product and wasted tokens doing it. Two rival divisions racing on contested surfaces duplicated their clipboard work four times and left the uncontested features (reload persistence, keyboard copy/paste, insert/delete UI) broken or absent.

• Oracle took the longest (28 rounds, longest in the benchmark) to ship a mid-pack product. Internal red tapes caused significant slow down yet didn’t help catch real product problems a customer would care about.

This work is clever, original, and fascinating. I hope we see more in this vein.

AI legend Yann LeCun is not afraid to swim against the current of current thinking in Silicon valley. For example, he thinks LLMs are “not a path to superintelligence or even human-level intelligence. I have said that from the beginning. The entire industry has been L.L.M.-pilled.” His new venture, AMI Labs, is doing something very different: It’s building world models that have an understanding not just of correlation (these things are associated with these other things), but of causation (these things cause these other things to happen).

Yann is also not afraid to call out peers of his who, in his opinion, are far too alarmist and pessimistic about AI’s impact on jobs and wages. Geoff Hinton has been one of these pessimists for a while now. He famously declared in 2016 that we should stop training radiologists because we just weren’t going to need many of them anymore as we moved deeper into the AI era. So far, he’s been wrong about that. In a recent post on X, Yann responds to some of Hinton’s latest predictions

I’m very flattered to be included in this group. I’d also encourage everyone interested in theoretically and empirically grounded observations about AI’s economic impacts to follow a few other people, including Daniel Rock (my coauthor and cofounder), Alex Imas, Seth Benzell, and Andrey Fradkin (Andrey and Seth host the excellent Justified Posteriors podcast, and at the risk of giving too much away let me say that Seth is currently demonstrating that he knows how to write a geeky book that normal people will enjoy, learn from, and (hopefully) buy in huge quantities).

Who am I missing? Who else should be on this list?

This weekly roundup is brought to you by Workhelix, the startup I cofounded to help organizations know and grow the ROI of their AI. If that topic is top of mind for you, please get in touch.