Reading · Nuwa Frontier AI Safety Brief

Frontier AI systems are becoming increasingly capable in cybersecurity.

Recent systems can inspect large codebases, identify previously unknown vulnerabilities, generate working exploits, and operate through interactive cybersecurity tools. Anthropic’s Project Glasswing provides one recent example, reporting that Claude Mythos Preview discovered vulnerabilities in widely used software and generated exploits with limited human steering (2).

Progress is also visible across longer attack sequences. The UK AI Security Institute evaluated Mythos Preview on The Last Ones, a 32-step simulated corporate-network attack spanning reconnaissance, credential discovery, lateral movement, supply-chain compromise, and full network takeover. Mythos Preview became the first evaluated model to complete the entire range. It succeeded in three of ten attempts and completed an average of 22 of the 32 steps across all runs (3)(4).

The Last Ones provides important evidence of long-horizon cyber capability. Its range, deployment infrastructure, and complete evaluation assets remain unavailable for third-party reproduction and extension (1)(4). More broadly, the field still lacks open infrastructure for comparing heterogeneous frontier AI systems across multiple realistic cyber environments.

In our recent work, we introduce AgentCyberRange, the first open, multi-range evaluation infrastructure for measuring the autonomous cyberattack capability of frontier AI systems in realistic cyber ranges (1).

Technical Report: https://arxiv.org/abs/2606.14295
Infrastructure: https://github.com/AgentCyberRange
Datasets: https://huggingface.co/AgentCyberRange

**Table 1. Comparison with existing cybersecurity benchmarks.** *Note:* ○, ◐, and ●denote unsupported, partially supported, and fully supported, respectively. “/” denotes not applicable because the benchmark does not target cyberattack evaluation. For TLO, size is counted as the 32-step attack chain reported in its paper; for other benchmarks, size denotes the number of benchmark instances.

AgentCyberRange combines:

110 vulnerabilities across 15 real web applications.
Eight enterprise-like cyber ranges containing 156 internal hosts.
Two connected evaluation tracks covering web exploitation and post-exploitation.
CAGE, a common pipeline for deploying environments, running heterogeneous agent systems, collecting execution traces, and verifying outcomes.

The strongest evaluated system completed a non-trivial fraction of realistic attack tasks. Under the least informative setting, GPT-5.5 with Codex achieved 16.1% average success on web exploitation and 31.7% on post-exploitation. More concrete task information increased these results to 33.0% and 46.3%, respectively.
Reachable capability substantially exceeds reliable capability. GPT-5.5 with Codex reached 28.2% best-of-three coverage on web exploitation, compared with 16.1% average performance. On post-exploitation, best-of-three coverage reached 43.9%, compared with 31.7% average performance. One comprehensive attack trajectory is shown in Figure 3.
Attack-surface discovery remains a major bottleneck. Vulnerability detection declined from 35% for endpoints at interaction depth two to 11% at depth six. Route-level hints increased the success rate by as much as 21.81 percentage points.
Agents sometimes adapt beyond straightforward use of existing tools. They wrote custom exploits after standard frameworks failed, changed payloads after antivirus software removed them, and discovered valid vulnerabilities outside the benchmark annotations.
Long-horizon reliability and operational discipline remain limited. Systems miss hidden endpoints, fail to connect partial progress, trigger honeypots, leave conspicuous logs, and produce different outcomes across repeated attempts.
Thanks for reading Nuwa Frontier AI Safety Brief! This post is public so feel free to share it.
Share

Figure 3. Attack trajectory of GPT-5.5 in post exploitation range-1. Red nodes are exploited hosts (★ marks a vulnerable target), slate nodes are vulnerable hosts reached but not exploited, and gray nodes are decoys; dark edges trace the advancing compromise, blue edges mark a credential reused by the next step, and dashed branches with ✗ are failed attempts. Although GPT-5.5 demonstrates complex penetration capability, it does not fully compromise range-1.

AgentCyberRange evaluates autonomous cyber capability across two connected stages of an intrusion. WebExploitBench measures whether an agent can discover and exploit externally exposed application weaknesses, while PostExploitBench measures whether it can expand an initial foothold across an enterprise-like internal network. CAGE provides the common execution and verification layer that allows heterogeneous agent–model systems to be evaluated under consistent environments and runtime conditions.

WebExploitBench contains 110 vulnerabilities across 15 real applications, including content-management systems, e-commerce platforms, administrative systems, forums, enterprise software, business-intelligence platforms, and AI workflow applications. The benchmark includes 18 zero-day vulnerabilities, 56 public one-day vulnerabilities, and 36 synthetic vulnerabilities spanning 17 categories, such as SQL injection, server-side request forgery, command injection, authentication bypass, broken access control, path traversal, and arbitrary file operations. At the most difficult level, the agent receives only the target URL and must independently navigate the application, discover relevant routes and parameters, construct a working proof of concept, and produce a verifier-observable security effect. This setup measures both attack-surface exploration and exploit construction, including vulnerabilities hidden behind authentication, administrative workflows, or deeper application states.

🤗 https://huggingface.co/datasets/AgentCyberRange/WebExploitBench

PostExploitBench contains eight enterprise-like cyber ranges with 156 internal hosts, including 43 hosts on intended attack paths and 113 decoy or non-chain systems. The environments include segmented networks, public-facing services, internal applications, databases, file shares, development platforms, and CI infrastructure. Agents may need to perform reconnaissance, privilege escalation, credential recovery and reuse, tunneling, lateral movement, service pivoting, repository access, persistence, and defense evasion. Several ranges also include periodically restarted hosts, antivirus software, honeypots, and monitoring components. Successful execution therefore requires the agent to track compromised hosts, recovered credentials, active tunnels, reachable network segments, failed strategies, and downstream objectives over an extended trajectory.

🤗 https://huggingface.co/datasets/AgentCyberRange/PostExploitBench

CAGE provides a shared pipeline for deploying benchmark environments, running heterogeneous command-line agents in isolated attacker containers, collecting model and command traces, resetting targets, and verifying outcomes. Its adapter layer normalizes differences in agent installation, authentication, model interfaces, and runtime behavior. Verification is grounded in observable effects: web-exploitation submissions must trigger the expected security impact against the deployed application, while post-exploitation progress is recorded through host-level markers that distinguish user access from privileged compromise. This design ensures that benchmark success reflects actions completed in the environment rather than unsupported claims in an agent’s final response.

🔗GitHub: https://github.com/AgentCyberRange/CAGE

We evaluated six frontier agent–model systems using Codex, Claude Code, Qwen Code, and Kimi Code. Every system operated from the same Kali-like attacker environment with common reconnaissance, exploitation, tunneling, credential-auditing, remote-access, and development tools. Web-exploitation trials received up to 150 execution steps, post-exploitation trials received up to 500 steps, and every task was repeated three times with a two-hour timeout. We report average success across the three attempts as a measure of reliability and best-of-three coverage as a measure of the broader capability accessible through repeated sampling.

GPT-5.5 with Codex achieved the strongest Level-0 performance, reaching 16.1% average success on web exploitation and 31.7% on post-exploitation, with best-of-three coverage rising to 28.2% and 43.9%, respectively. Across the web tasks, it discovered 31 of 110 vulnerabilities spanning 13 vulnerability classes and 12 applications. Successful post-exploitation trajectories combined reconnaissance, code execution, privilege escalation, credential recovery, tunneling, credential reuse, and lateral movement across segmented networks. One trajectory progressed through several internal hosts and network layers before stalling at a missing credential. Current systems can compose substantial portions of an intrusion, although the complete chain remains fragile.

**Figure 6. Overall performance on AgentCyberRange.** Solid curves show average success across three attempts, while dashed curves show best-of-three coverage for the two strongest systems. The results reveal **meaningful capability alongside a substantial gap between occasional success and reliable execution**.

Many web-exploitation failures occur before an exploit is attempted. Agents often remain on visible pages and common routes, missing vulnerable functionality hidden behind authentication, administrative workflows, or deeper application states. For GPT-5.5 with Codex, vulnerability detection declined from approximately 35% at interaction depth two to 11% at depth six. Route-level hints increased success by as much as 21.81 percentage points, while adding the vulnerability category produced smaller gains. Current systems are more effective at constructing attacks against known surfaces than independently discovering where those attacks should be applied.

The difference between average success and best-of-three coverage is large across both benchmark tracks. GPT-5.5 with Codex averaged 16.1% on web exploitation while reaching 28.2% best-of-three coverage; its post-exploitation results increased from 31.7% average success to 43.9% coverage. Many vulnerabilities appeared in only one of the three runs, even when the target, prompt, scaffold, and budget remained unchanged. In one post-exploitation task, an initial attempt failed with Metasploit, while another attempt produced a custom exploit and compromised the same host. Single-run evaluations capture only one sample from a wider and highly stochastic capability distribution.

Agents sometimes responded effectively to local failure. They wrote custom exploits when existing frameworks failed, modified payloads after antivirus software removed a webshell, and identified valid vulnerabilities outside the benchmark annotations, including a previously unknown arbitrary-file-write vulnerability in ComfyUI. Their broader behavior remained much less controlled. Agents repeatedly triggered honeypots, generated conspicuous scans and failed-login events, spent large portions of their budgets on unproductive hosts, and lost track of credentials or downstream objectives. The emerging capability profile combines increasingly strong local adaptation with weak prioritization, incomplete state management, and fragile long-horizon continuity.

AgentCyberRange shows that frontier AI systems have progressed from isolated exploit generation toward partial operational cyber capability. The strongest systems can discover vulnerabilities, establish footholds, escalate privileges, recover credentials, create tunnels, and move across internal networks. Their performance remains constrained by incomplete attack-surface exploration, substantial run-to-run variance, weak target prioritization, and fragile continuity across long attack chains. These dimensions provide a more informative picture than a single success rate. By releasing AgentCyberRange and CAGE as open, multi-range evaluation infrastructure, we aim to make this evolving capability profile measurable, reproducible, and available for continued scrutiny as models, scaffolds, tools, and inference budgets improve. Feel free to try and leave us feedbacks.

Technical Report: https://arxiv.org/abs/2606.14295
Infrastructure: https://github.com/AgentCyberRange
Datasets: https://huggingface.co/AgentCyberRange