There’s an iron law of categorization: categories only stabilize where data is dense. Everywhere else, they fracture into opinion. This is why ten thousand IT tickets classify themselves, but five security incidents spark taxonomy wars. If you’re designing LLM schemas without accounting for this law, you’re building for the wrong end of the distribution.
We’re pretty good at giving feedback to traditional software. When something breaks, we know exactly what to do: file a GitHub issue, ping the team on Slack, leave a detailed bug report explaining exactly what went wrong and how to reproduce it. The whole system works because it assumes we can articulate precisely what “wrong” means.
It’s no secret that today’s LLM-based agents are unreliable. This makes them a gamble for most critical tasks, so where can they be safely applied? The answer lies in finding asymmetry: we should use them in domains where the downside of a mistake is low, but the upside of success is huge; a strategy Nassim Taleb calls antifragile.
“Any good classifier knows that in the process of classification, information about variety is lost while information about similarities is gained.” - Joseph Tainter
When thinking of LLM evals especially in the context of RAGs, it occurred to me that there are two kinds of distinct responses people get from LLMs: informational and instructional.
Sarah sighed in frustration as she stared at her laptop screen, the SSH connection to the university server dropping for the third time in the last hour. She was trying to access some important research files for her thesis, but the constant interruptions were driving her mad.