There is no cognitive core

Simon Willison asks : What's the latest research on how much baked-in knowledge an LLM needs in order to be useful? If I want a specialist coding model can I trim the size of that model down by stripping out detailed knowledge of human history, geography etc? Can we even do that? Andrej Karpathy describes work toward such a system The race for LLM "cognitive core" - a few billion param model that…