What would a fair benchmark for agent architecture look like? [D]

This site does not allow itself to be embedded. You can still read it on the original site — the toolbar below keeps your place in the directory.

I am working on an evaluation design and would appreciate criticism before running it. Most coding-agent benchmarks collapse the model and its harness into one score. If a run fails, it is difficult to tell whether the cause was model capability, context assembly, task decomposition, tool design, retry policy, or the acceptance gate. A model can also look worse because the harness truncated its…

Read on reddit.com ↗