AI Benchmarks: Are We Juking the Stats?

I think I’ve seen this movie before. I have experience in education (teaching prek, K, 4th, 6th, and college). I have got experience in tech (90’s AOL/ICQ script kiddie/webmaster turned founder). Today I’m your average generative AI user. I care about SWE-bench rankings about as much as I cared about standardized test scores as a teacher. Aside from quickly discerning if a model is at least GPT-4…