No Autonomy Without Scalable Oversight
What to expect as we enter the Year of The Judge
Shane Caldwell
Recent content on Shane Caldwell
Latest posts
ProofJudge: Can we align vibe-proving with human taste?
Towards measuring alignment with human taste in autoformalization with judge agents.
The Tests All Pass
METR's SWE-bench analysis shows us taste isn't verifiable.
Intro to GPUs For the Researcher
Getting comfortable with the hardware on a quest for more MFU.
All Reduce Across the Atlantic: Bandwidth in Decentralized Training
The practical realities of devestatingly high communication cost in training.
Twenty Billion Tokens of What, Exactly?
Looking at the data and letting it look back at us.
Pretraining at home: 20B tokens from 222 hours to 12
Optimizing training a Llama 3.2 1B model so we can pretrain in a day without going broke.
Offsec Evals: Growing Up In The Dark Forest
If you contribute a public benchmark, are you giving free capability to your competitors?
DiLoCo: Data Parallelism for the Datacenter Poor
Distributed training sans datacenter.
RL Needed LLMs Because Agency Requires Priors
We tried RL once. It didn't work. I'm confident it will this time.
GPT-5 is Good, Actually: The Agony and Ecstasy of Public Benchmarks
An attempt to explain why benchmarks are either bad or secret, and why the bar charts don't matter so much.
The Religious Devotion of Haskell
An exploration of functional programming through Haskell, motivated by trying to understand the near-religious devotion its practitioners have for the language.
The Input Sanitization Perspective on Prompt Injection
An analysis of prompt injection vulnerabilities in large language models and why they represent a fundamental security challenge.
Infosec's Data Problem
Exploring the fundamental data sharing challenges that limit machine learning progress in information security, and why the field needs its own ImageNet moment.
Deep Reinforcement Learning for Security: Toward an Autonomous Pentesting Agent
An exploration of using deep reinforcement learning to create autonomous penetration testing agents, examining the challenges and potential solutions for automating cybersecurity assessments.
An ML Eng's Review of OSCP
A comprehensive review of the Offensive Security Certified Professional (OSCP) certification from the perspective of a machine learning engineer entering the security field.