writing as a rehearsal of language. language as a rehearsal of logic. ...
First there were humans, and then there was Abundance. Abundance was a generous deity, under whose watch all that was abundant would grow more abundant. Abundance, in this manner, was the God of Cancer, but also God of that which separates the sea from the puddle. The humans worshipped Abundance, and those who made more abundance were made more abundant. In time, humanity itself became abundant,…
Strong architectures are results of accidental/intentional hardware-software codesign. The dogma, to me, appears to be: how much useful compute/time can we throw at the model? (exemplified by: "the models, they just want to learn" - Ilya Sutskever) In a sense, the principal inductive bias that we adhere to is the fact that we run learning systems on the arbitrary computing substrate that we have…
Per SICP, a computer program is defined as the precise description of a process. Perhaps a bit more generally, a program is a description of a computational process that evolves in time according to some given rules. When the computational process described by the computer program is governed directly by electrons, we call it hardware ; on top, a light abstraction upon the hardware is the firm…
I have always been fascinated by those wizardly folks who seem to know how to talk to their PID controllers, hold it for seconds and tune it nearly perfectly. It's always nearly, though. Of course, the right way is to mathematically find a optimal set of parameters. The physical intuition isn't really necessary. That sounds very true of how we do pre/post-training these days. We seem to be doing…
Do these jokes still work? Anyway, Claude Mythos is coming, and I'm excited. I'll hence allow myself some scientific fiction, for jest. This is all very hand-wavy. Pardon me. I originally wrote this on X, but I'm keeping it here too. I am always excited for larger and larger models. This is a little attempt at saying why. We sometimes look at an LLM's pass@K on a problem: can this model solve the…
This blog probes and develops the idea of distributed training of large models over heterogeneous devices. By distributed training, we will mean two things: one, that the units of compute are located geographically distant from each other. We also present a stronger condition, in that we also want to be able to train large models over the commodity internet. This is (mostly) solved, noticing and…
Towards a Theory for Large Language Models from Across the Internet A concrete theory for large language models must start with the phenomenon that resulted in the notion that such large models could exist . There are small disagreements on which phenomenon that is, but arguably the first observation that hinted towards the existence of such models was what we call the double descent -- the…