CTFs Are the Only Benchmark You Can't Fake

How Do You Measure an Agent? I&rsquo;ve been tracking the AI field for a while now, and it seems like every week there&rsquo;s a new model that tops the benchmark charts whether it&rsquo;s HLE or SimpleBench, MMLU or GPQA. It seems like these models know everything, but they struggle to apply it their knowledge and fix problems correctly. An agent should be self-aware of its own limits, for…