Random thoughts on LLMs

Last year I said that transformers will be gone in a decade (2035), but didn’t say anything of substance beyond that. I still don’t have much substance, but I’d like to posit a counter architecture/model structure which may serve to be useful. I read some AI papers, but not all, so its most likely that this idea has been implemented, tested, and empirically shown to be terrible but I’ve not read that paper yet.

The core is splitting the objectives of learning and memorization, appealing to my earlier pieces on Memorization and Intelligence: currently, we have models that perform quite well on the latter (that is basically how LLMs work, no? Encoding information into their weights.) and we have approximated the former through RL, i.e. reasoning. I think splitting the two modes into some sort of conjoined models may serve as a means to boost performance.

At first blush, buying into all the subagent hype (RLM, that Nvidia paper about small models being the future), I’d say that a typical transformer model could be used as the memorizer, whereas a new architecture or at least objective must be devised in order to create good learners which haven’t memorized many facts. The learner models should be good at learning, obviously, but perhaps as critically, should have limited knowledge—limited knowledge allows them to be flexible, that they can adapt any structure as required without having “biases” (experience) against this structure. The memorizers would then just memorize a set of information and would be available on demand, moving knowledge into the learners’s context window for it to work with—a sort of “smart” RAG with a full LLM instead of just an embedding model. Moving common or surprising context to files or committing to the learners memory may be helpful, as inspired by the latest Deepseek paper on scalable lookup and Google’s Titans and surprise-based memorization.

Tangent on diffusion models:

DIffusion models obviously can’t reason since they fix the response size and iterate on it—but isn’t a simple (inefficient, like the above idea) solution to tie it to a purely reasoning LLM? I.e., train a bimodal system with a transformer-based LLM to process the user request and then reason about it—then concat those and pass that to the diffusion model which know has essentially “thought” by way of this other LLM? The reasoning model could probably even just be one of these “zero” models which aren’t optioned to chat, so as to reduce some training cost. This seems too obvious, so I’m guessing the latency is too costly to make diffusion models worth it for the generation—you’d just stick with a single transformer based LLM which would reason and answer all at once.