Anthropic drew equal parts fear and frustration when it announced it would not release its Mythos model because it could autonomously uncover security vulnerabilities and create exploits.
Cloudflare was one of the companies invited to join Project Glasswing and to access a preview of the new model. They shared their findings after pointing it at more than 50 of their own repositories.
Based on their experience, it is “a different tool doing a different kind of work”, with real progress when compared to general-purpose AI models attempting to do the same thing.
They highlight exploit chain construction and proof generation among the two features that stood out. The former crafts an exploit by leveraging multiple bugs together, while the latter removes the speculation by creating a program from scratch that behaves as predicted.
The article shared an important aspect of any security reporting: the signal-to-noise ratio. Generic models, when tasked with finding security vulnerabilities in a codebase, will often return them, even if they do not exist.
This has led to many bug bounty programs being cancelled or paused until better ways to filter false reports appear. Linus Torvalds reported the overload of the same reports on the Linux kernel.
Mythos preview seemed to do a better job here as well, but still benefited from having a harness to manage the overall execution.
4 lessons out of this activity:
A narrow scope produces better findings
Instead of asking to “find vulnerabilities in a repository”, instruct it to “look for (e.g) command injection in function X, with this trust boundary, given the architecture document and prior coverage in the same area”.
Not so different than what we are used to when asking the model to implement something. Being more specific and providing the right context helps.
Adversarial review reduces noise
Adding a secondary agent with a different prompt and model can help catch much of the noise the first agent generates.
This adversarial mode pits the agents against each other and challenges their findings, as they must resolve the disagreement before converging on an answer.
Splitting the chain across agents produces better reasoning
When having multiple different inquiries, splitting them up and asking them separately is better, as it gives specific focus to each agent.
Parallel narrow tasks beat an exhaustive agent
Instead of expecting a single agent to handle a task exhaustively, break it down into tightly scoped questions, send it to separate agents, and handle any duplication in the results later.
Cloudflare used the following harness
Step 3 (validate) illustrates the use of a second agent that tries to disprove the findings from the hunters.
The consequences of this are more pressure on the security teams, with reduced SLA times from CVE release to patch in production. And that exposes more concerns about the rest of the pipeline.
Imagine finding and patching an issue within a 2h window just to see it wait for a day until a regression test is finished. At the same time, knowing that skipping this step may result in worse issues making their way into production.
That is the conundrum that many high-profile companies find themselves in.
Talks about the AI Bubble and the circular nature of some investments are not new. Last year, even the OpenAI CEO seemed to agree with that when answering whether the industry was overexcited about AI.
Yet we seem to keep pushing its rupture further down. In a recent article at The Atlantic, the author voiced his reasons why, mostly due to the growth rate, especially by Anthropic.
By his account, if its revenue continues to rise at this rate by early 2027, they would be taking more money than any other company in the world.
Unfortunately, there are still many unknowns to confirm this:
What is the cost
Software developers have embraced AI tools, and it is increasingly rare to find those who do not use Claude Code or similar as a part of their development workflow.
But how much does it cost to serve those $20-40 accounts? Rumors are that a single account costs way more, hence the amount of cash being spent, despite the growth.
It is ok to subsidise this cost for a while: market share, fund the evolution of the technology with expectations of cost reduction in the medium term. But it must have an end in sight.
What is the value
AI, even at its current stage, is already powerful enough to replace certain positions where the human-in-the-loop added little value. But that is already in the past, with the productivity gains being promised but yet unrealized by most projects (as of 2025).
There is only so much that can be truly absorbed to speed up software development, and if benefits do not materialize elsewhere, companies will rethink their investments once the AI-native settles
I believe that whatever the outcome is, in the aftermath, AI will still have its way, so the safest bet is to continue to look at it as a tool that will change the way you work, not eliminate it.
In the event-driven world, Sagas are a fairly well-known pattern to provide coordination to workflows and achieve the “distributed transaction” without two-phase commit.
Both orchestration and choreography flavours can help your service to handle unhappy cases when you have a negative outcome and need to compensate or “rollback” changes.
However, while they can tell which step you are in a distributed process, which message you should receive, and what command you should send next, they are not the point of truth of external systems.
This article covers the mistakes of adding additional retries and more conditional code in the saga itself as a way to handle this, when what is really missing is recognizing the role of reconciliation.
The idea is that you first need to know:
What should be true
What is actually true
What corrective actions can you safely apply
In the example provided, imagine that you need to interact with a payment system. You requested a payment to be captured, but the request timed out.
Note that at this point, you do not know what happened. Maybe it was actually processed and the payment captured or declined.
Treating this as a failure and initiating a rollback is not the right call, as it may have been approved.
Adding reconciliation to the saga means that we know we should receive a response, and that currently, we have not, and we should initiate a separate process to contact the payment after X minutes of non-response.
Time-outs are one way of triggering reconciliation processes, but they are not the only one, with polling triggered by users or scheduled jobs being another.
Ultimately, you need to see what makes sense to your context (scale, time sensitivity, etc).
If you want to know more about Sagas, including how to implement them in Cloud-based environments, check the following resources:
Handling Complexity: Using Sagas to Provide Transactional Support for Distributed Systems
Implementing Sagas using AWS Step Functions: Advanced Concepts
Sagas: When a Distributed Transaction Is The Right Thing For You
Looking forward to knowing how to leverage Claude Code with your software engineering projects? Check this workshop. Use code MARIO40 to get 40% off.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.