The human benchmark is typically unimpressive

If you ever read a claim of an AI outperforming a human, dig into the performance data to check out the human benchmark. The mediocrity of the human is often more salient than the competence of the AI. The AI has an easy job. Here’s an example from a paper by Ayers and friends (2023) . They sampled 195 question and response exchanges on Reddit’s r/AskDocs and entered those same questions into…