[WIP] Benchmarking Local LLMs Against Coding Agent Harnesses

I’ve been running a small benchmark, harness-bench , that pairs local LLMs (served via llama.cpp ’s llama-server ) with agent harnesses (Aider, Claude Code, OpenCode, Pi, Qwen CLI) on 16 software-engineering tasks across Python, PyTorch, JAX, C, C++, Rust, and SQL. Each (model, harness, task) cell is sandboxed: the agent only sees a scratch workspace/ and grading is done by a hidden test.sh that…