There seems to be a lot of recent interest in and excitement about the promise of “agentic AI” (tools like Claude Code or Cursor) for improving productivity in scientific research. The idea, or perhaps more accurately, the hope, seems to be that by automating key steps in scientific workflows, agentic AI tools can massively improve… Continue reading No, agentic AI will not massively boost…
Over the last couple of months, I’ve been continually training the pretrained Llama-3.1-8B model (with a context length of 8192 tokens) for 809B tokens. This was my first truly large-scale distributed training experience and in this post, I’d like to share some of what I’ve learned so far. Why First of all, why am I… Continue reading Continual training of Llama-3.1-8B for 809B tokens →
One of the pitfalls in scientific writing in general is the danger of prematurely describing the basic observations of a field in theory-laden terms. This can be particularly problematic in fields that are in their infancy, such as developmental psychology, where basically all the fundamental questions are wide open, and where we basically don’t know… Continue reading Language development in…
If the doors of perception were cleansed every thing would appear to man as it is, Infinite. For man has closed himself up, till he sees all things thro’ narrow chinks of his cavern. William Blake, The Marriage of Heaven and Hell I’ve recently been reading up on the effects of psychedelics on the brain.… Continue reading On the entropic brain, trapped priors, and machine learning →
An interesting detail in the recently released Llama-3 technical report has caught my eye (p. 8): This has caught my eye, since I had noted the same phenomenon in a previous post about the Chinchilla scaling laws (more than two years ago) to argue for training smaller models (point 4 in that post). I’m glad… Continue reading IsoFLOP curves of large language models are extremely flat →
I’m a bit late to the fray as usual, but I wanted to write a short post about Sora. Sora is OpenAI’s new video generation model. As of this writing, it’s still not open to the public, so all we’ve got so far is some high-level information about the model and some generated samples shared… Continue reading Does Sora understand physics? A few simple observations →
I recently came across this beautiful short blog post by James Betker (who works at OpenAI), arguing that the thing that really determines the capabilities and, more generally, the behavior of a machine learning model is not its architecture, it’s not the particular optimizer used for training the model, or any other details of the… Continue reading The “it” of deep learning and convergent…
One of the recurring themes in Hans Moravec’s prescient book, Robot: Mere Machine to Transcendent Mind (first published in 1999), is how practically important problems (e.g. agile robot navigation in the real world) become tractable more or less automatically, as the amount of widely accessible compute reaches a soft threshold. Before this threshold is reached,… Continue reading Intelligence is a…
While working on some generative video models recently, I had a moment of epiphany about hallucinations in generative models. I wanted to share this tiny bit of insight (if it isn’t too presumptuous to call it an insight) that has occurred to me. It is one of those simple things that has always been right… Continue reading Further thoughts on hallucinations in generative models →
GPT-3.5 seems to be surprisingly bad at answering basic factual questions about famous writers and famous works of literature. This is something I’ve noticed over the last couple of months and here I’d like to share some random examples of this that I encountered recently: Some of these errors are more egregious than others, but… Continue reading GPT-3.5 is surprisingly non-factual about…