Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Design

Presented at DeMeSSAI'25 in Venice, Italy , arxiv version : Large Language Models (LLMs) have emerged as a powerful approach for driving offensive penetration-testing tooling. Due to the opaque nature of LLMs, empirical methods are typically used to analyze their efficacy. The quality of this analysis is highly dependent on the chosen testbed, captured metrics and analysis methods employed. This…