Sixty-Two Percent Correct and Winning by a Hundred and Twenty Points

The standard way to evaluate a question-answering system is to measure how often it gets the right answer. This seems reasonable. It is, in practice, a trap. I ran an experiment to demonstrate why. A Bayesian decision-theoretic agent &mdash; built on the Credence DSL, using Beta-Bernoulli reliability tracking and value-of-information calculations &mdash; competed against several LLM agents on a…