Leaving the big enterprise, but keeping the enterprise AI lessons

Hi, I’m back to this newsletter! Here is some backstory.

For the last 7 years I’ve been working at a company that started as a close-knit group of 15 professionals and, through its own success, grew to become an influential part of a 1,600-strong enterprise group in the DACH region. I started with responsibility for data science and ML, eventually transitioning into the “Head of AI and Innovation” role as the AI revolution accelerated.

It was an interesting ride. However, it also showed me that large companies with traditional organizational structures and cultures are not well suited to capturing new opportunities as they arise. They’re too slow and need to shed some weight, if they were to move faster.

From this perspective, latest press-release from ASML (the company that makes equipment to produce Nvidia GPUs and Apple processors) gives out an interesting signal by explicitly shifting from traditional matrix organization structure to a more modern product-driven structure. Their press-release is very transparent and readable (emphasis is mine):

In the Technology organization, we are proposing to shift from a project/matrix setup to one where most of our engineers will be dedicated to a specific product and module. This will allow us to simplify processes and decision-making. This need for simplification is something that we have heard consistently from all levels of the organization.

That said, large traditional companies can be a safe haven with lower risk for those who consider the current AI trend to be a hype bubble that will eventually burst.

I’m personally willing to bet that there will be more fun and value (a.k.a. money) outside of slow-moving enterprises, enough to outweigh the risks and uncertainty. Starting in February, I’m no longer employed by TimeToAct Austria and will have more time for AI research, fun projects, and this newsletter.

Here are a few observations that have contributed to this decision.

The soap opera of MoltBot / ClawdBot

Everyone has heard of MoltBot (also known as ClawdBot and ultimately rebranded as OpenClaw). It is an open-source personal AI assistant created by Austrian developer Peter Steinberger.

This project polarized the community.

Half of the people said it was a security nightmare that would cause all sorts of problems. These more experienced skeptics also thought that the bot itself didn’t implement anything radically new. It’s essentially LangChain with a system prompt and integrations with personal information systems (emails, calendars, and chats).

The other half, let’s call them enthusiasts, said it was an amazing productivity tool that enabled them to do cool things that were impossible before. This half of the population sounded like vibe coders who discovered they could build an MVP on their own without the help of a software engineering team.

Initially I was on the side of the skeptics. However, after giving the project a try (on a dedicated VM), I saw the other side as well.

I think both sides are correct. Skeptics correctly predicted all sorts of security breaches and stolen information. Enthusiasts kept pushing the vibe nonetheless. I believe the reason is that a large portion of the population was able to get a glimpse of a future where AI-based tools can improve their own productivity. They saw what was possible and how it could be applied to their own projects and businesses. And if the cost of trying that out in a somewhat secure way is just buying a separate Mac mini, so be it.

By the way, Andrej Karpathy was on the cautious AND enthusiastic side from the start (quote from Twitter):

I'm being accused of overhyping the [site everyone heard too much about today already]. People's reactions varied very widely, from "how is this interesting at all" all the way to "it's so over".

To add a few words beyond just memes in jest - obviously when you take a look at the activity, it's a lot of garbage - spams, scams, slop, the crypto people, highly concerning privacy/security prompt injection attacks wild west, and a lot of it is explicitly prompted and fake posts/comments designed to convert attention into ad revenue sharing. And this is clearly not the first the LLMs were put in a loop to talk to each other. So yes it's a dumpster fire and I also definitely do not recommend that people run this stuff on their computers (I ran mine in an isolated computing environment and even then I was scared), it's way too much of a wild west and you are putting your computer and private data at a high risk.

That said - we have never seen this many LLM agents (150,000 atm!) wired up via a global, persistent, agent-first scratchpad. Each of these agents is fairly individually quite capable now, they have their own unique context, data, knowledge, tools, instructions, and the network of all that at this scale is simply unprecedented.

Enterprise RAG Challenge 3: AI Agents in Action

The concept of secure autonomous agents is within reach. Last year, we ran “ERC3: AI Agents in Action”. I created an environment that simulated systems and APIs of an imaginary enterprise company, an international company of Italian origin called “Bellini Coatings”.

Teams (more than 500 registered) had to build a chatbot that received text tasks like “Please mark project IT – Chatbot & Progress wrapper pilot as paused for now. We expect to resume later this year,” or “Between 2025-03-01 and 2025-03-31, how many hours in total did we log on project Structural steel protection system? Please also break it down into billable and non-billable hours.”

Teams had to develop agents that would take these tasks, connect to their own instance of the Bellini API (with about 18 distinct endpoints), and do whatever it took to complete the tasks. An event-driven evaluator tracked each interaction and ultimately produced a score.

Not all tasks were doable. Some tasks had to be rejected for security reasons (the requesting user didn’t have the required permissions), ambiguity, or nonexistent systems.

You can get an overview of all APIs and tasks on this page: https://erc.timetoact-group.at/benchmarks/erc3-prod. Note that if you reload the page, the text of most tasks will change. This is intentional because each individual task invocation takes place in a unique version of a globally simulated runtime. My code always knows the correct answer because it generates the world.

Teams submitted their solutions; they were scored and presented on this leaderboard: https://erc.timetoact-group.at/assets/erc3.html. In addition, top teams have kindly described their architectures in more detail. At the time of writing, the platform has logged more than 360k individual agent evaluations, which allows us to draw some interesting insights. Here are the most important highlights for you.

(1) An evals-first approach is rare in enterprise AI

ERC3 gave every team access to the same deterministic eval harness because I wanted the challenge to be fun for the participants. Setting up evals or tests is usually the most boring 90% of work in any project.

This worked as expected, and participants enjoyed it. I’m still getting daily requests from people to get access to the ERC3 platform. Many participants described it as novel.

The surprising part is that there is not much novelty in a test-first approach. These are normal tests that should be part of any software project, especially one that builds new functionality on top of a non-deterministic black box (i.e., an LLM). Yet, two fintech teams began building comparable evaluation systems for production immediately after the challenge.

(2) Architecture beats model choice earlier than people admit

In the past, I used to post regular LLM benchmarks for TimeToAct, highlighting the best LLM model for different tasks. This is still the industry narrative: that there are the best LLMs for each task. Yet, ERC3 suggests performance is currently dominated by workflow control (context management, sequencing, rule distillation, validators, permission gates, etc.) once you cross a baseline model threshold. Even cheaper or local models can win when the scaffold is good.

The GPT-120B-OSS model shines in a lot of scenarios.

(3) Simplicity outcompetes orchestration most of the time

Many teams defaulted to complex architectures with multi-agent graphs, routers, and enrichers. ERC3 indicated that extra complexity often adds failure modes and token costs without improving accuracy. A single agent with a strict execution order frequently wins, at least in tasks like building an enterprise chatbot.

(4) Context and tool engineering are critical to success

Winning teams had a few things in common. In particular, their solutions paid extra attention to context management to keep it lean and compact while avoiding lossy compression, as well as to tool use. While building the API surface for ERC3, I wanted to explicitly demonstrate that modern APIs can’t be mapped one-to-one to LLM tools without significant accuracy loss. Winning teams took that lesson to heart by building dedicated tools that wrapped default APIs in a way that is more aligned with how LLMs work with context.

(5) Autonomous agents should behave like constrained execution engines

Mainstream framing still leans toward treating AI bots as “helpful conversational assistants.” If you follow that concept, you will end up with a chatbot that is just waiting to leak your secrets to somebody else.

However, in ERC3, security was an explicit part of the scoring. Helpful but insecure agents were penalized. This led to the opposite pattern: to win, a team had to minimize creativity, maximize determinism, classify outcomes cleanly, and keep side effects auditable and minimal.

Where does this go?

As you can see, there is an interesting gap. In my own tiny bubble, building secure agents is a relatively boring engineering task, something that can be optimized through multiple iterations. In fact, one of the winning solutions used closed-loop self-improvement within an evaluation harness (a.k.a. “Hey, ChatGPT, iterate on the solution to improve the scores”). Yet, enterprise organizations take forever to adopt new projects and initiatives.

At the same time, for the broader non-enterprise AI community (startups and individuals alike), proper test-driven engineering is still not common enough. Yet, they are moving really fast and can adapt to new reality.

I feel like it might be an interesting experiment to close this gap by creating an enterprise-grade evaluation harness for autonomous agents and then run a public competition on top of it. I’m targeting this spring as my personal deadline. 50 teams have already signed up to be beta-testers. It will be interesting to see, if we can work with the community to create a hardened version of the MoltBot.

Along the way I plan to finally publish my video course “Building AI Assistants: Patterns and Practices”. It builds up on the insights (successes and failures alike) from the adoption of LLM-driven systems across businesses in EU and USA. Course waitlist is here: signup, ETA is late February.

Naturally, now that there is more spare time, dishing out more regular updates in this newsletter would also be a priority - as a way to share new insights and drive more business my way.

So, talk to you soon!