Blog about artificial intelligence, machine learning, LLM training, neural networks, and life. Research Engineer at Google DeepMind working on Gemini.
I often get asked what kind of work research engineers in AI labs like Google DeepMind do. Mostly in the context of software engineers considering getting a job in one of the frontier/neo AI labs. System engineering Designing, writing, maintaining systems running on a classical stack: RPC servers, data storages, queues, caching. Storing/updating/serving training and eval data. Storing and querying…
I maintain a set of best explanations1 of how LLMs train and work. “LLMs” here is a broad term for the frontier models that create value in 2026. They have more components than just a language model, but powerful language models are their necessary core. Universal explainers Considering: humans are universal explainers (Deutsch) - they use creativity to explain the world. Conjecture: LLMs are…
Yesterday Iana and I were walking through the evening hills, breathing fresh air, letting our son Robert work up his appetite before dinner, and talking. I brought up our “religions” - sets of beliefs that are hard to prove objectively, which are chosen as the best explanations of the world around us. We started from our Buddhism and Popper/Dawkins/Deutsch1-inspired hypotheses about the world and…
I stumbled upon a post of mine written a year ago, right after the loud DeepSeek R1 release. I would add something about the mission worth fighting for and the research taste today. Reads fresh otherwise. Deep expertise in GPU/TPU programming. Sharded matmuls and communication collectives are the most important building blocks of modern AI systems. Not a lot of engineers have years of experience…
tl;dr. vorushin/pallas_puzzles - JAX’s kernel language puzzles Pallas is JAX’s kernel language for writing custom operations that run on TPU. Pallas for TPUs is what Triton1 is for GPUs. The kernels are written to speed up model training and inference - the kernel languages provide low-level access to the hardware, allowing you to perform optimizations outside of the compiler’s reach. Tri Dao2…
Andrej Karpathy has the nanochat project with the description “The best ChatGPT that $100 can buy”. He evolved a model architecture and training setup that reaches the performance of GPT-2 while costing 600 times less than the original OpenAI run from 2019. This is an inspiring example, showing that pretraining experiments can now be available even to individuals without corporate/university…
Training LLMs on TPUs/GPUs is often constrained not by the speed of the matrix multiplication units (MXUs), but either by data transfer from high bandwidth memory (HBM) to MXU, or by data transfer between accelerators. A large part of optimizing training/inference programs consists of finding ways to overlap data transfer and matrix multiplication operations - transferring data that will be needed…
My name is Roman Vorushin. I am a Research Engineer at Google DeepMind, working on Gemini. I fell in love with ML and AI back in 2011, thanks to the legendary MOOCs ai-class.org (Peter Norvig, Sebastian Thrun) and ml-class.org (Andrew Ng). Before Google I was an early Grammarly employee, had my own startup, and worked in Kazkommertsbank. Here is my resume. I used to have a technical blog in…