The Transformer Had a Great Run. Google DeepMind Just Built Something Better.

For nearly a decade, one architecture has dominated every corner of modern AI. The Transformer — introduced in a 2017 Google Brain paper with the now-famous title “Attention Is All You Need” — became the universal foundation beneath GPT, Llama, Gemini, Claude, and virtually every language model that matters. Its engine, global self-attention, gave models an extraordinary ability to relate any word or token to any other across an entire sequence. It was powerful, elegant, and almost universally adopted.

It also has a mathematical problem that the field has spent years carefully working around rather than actually solving.

Global self-attention computes relationships between every pair of tokens in a sequence. For a sequence of N tokens, that means N² operations — quadratic scaling. At 128,000 tokens, you’re looking at 16 billion operations per layer per forward pass. At one million tokens, one trillion. As context windows keep growing, this cost compounds relentlessly.

Then there’s the inference problem. Every token generated during autoregressive generation must be retained in GPU memory through what’s called a Key-Value cache. For a 9-billion parameter Transformer generating an 8,000-token response, that cache alone consumes 2 to 4 gigabytes of GPU memory. As the cache grows, batch sizes shrink, throughput drops, and generation slows — the longer the output, the worse the performance gets.

The field’s response has been clever engineering: Flash Attention reduces memory movement, ROPE scaling extends context through positional tricks, grouped-query attention cuts cache size with minimal quality loss. These are smart optimizations. But they’re patches on a fundamentally expensive architecture. Google DeepMind chose a different path.

The core insight behind Google DeepMind’s approach is that the Transformer treats all memory as a single monolithic mechanism. Global self-attention simultaneously handles short-range local structure, long-range dependencies, and task-relevant knowledge retrieval — doing everything through one expensive operation, even when most token pairs carry no useful signal for the current step.

The alternative: decompose memory into components, each designed for a specific purpose. A fixed-size recurrent state for efficiently compressing long-range context. A small local attention window for precise short-range lookup. And in the most ambitious version, a neural memory module that updates its own weights during inference. Three mechanisms, each doing one job well, together covering everything global attention does — at dramatically lower cost.

This idea unfolded across three research papers published between February and December 2024: Griffin, Recurrent Gemma, and Titans.

The first paper introduced a new sequence-processing unit called the Real-Gated Linear Recurrent Unit, or RG-LRU. Classical recurrent networks like LSTMs and GRUs maintain a hidden state updated at each step using both the current input and the previous state — creating a sequential chain that prevents parallel computation during training.

The RG-LRU breaks this dependency. Its gates are computed from the current input only, not from the previous hidden state. This means the entire sequence of hidden states can be computed simultaneously using a parallel scan operation. Training speed matches a comparable Transformer. But inference runs at constant cost per step, and the recurrent state size never grows regardless of sequence length.

To understand Griffin’s design, the paper first presents Hawk — a model built entirely from RG-LRU blocks with zero attention of any kind. The result was surprising: Hawk at 3 billion parameters outperformed Mamba at the same scale on downstream language tasks, despite training on roughly half the tokens. A simpler gating mechanism proved more data-efficient than the more complex selective state space approach Mamba uses.

But pure recurrent models have a structural limitation. They compress everything into a fixed-size state, and that compression is lossy. Asking a pure recurrent model to recall a specific piece of information from thousands of tokens back is like asking someone to retrieve a precise detail from a book they read months ago and summarized — the gist might be there, but precision is gone.

Griffin’s solution is precise and minimal: interleave local sliding-window attention blocks with the recurrent blocks in a two-to-one ratio. Two RG-LRU blocks handle long-range context. One local attention block — looking only at the most recent 2,048 tokens through a fixed rolling window — handles precise short-range retrieval. The KV cache for this local attention never grows. The recurrent state never grows. Together they cover the full range of what global attention does, without the quadratic cost.

The second paper translated Griffin into production-scale open-weight models at 2 billion and 9 billion parameters. The benchmark results were illuminating. Recurrent Gemma at 2 billion parameters achieved an average score of 44.6 across standard benchmarks against Gemma’s 45.0 — a gap of 0.4 points — while training on 33% fewer tokens. At 9 billion parameters, Recurrent Gemma scored 56.1 against Gemma-7B’s 56.9, with a 59.3% win rate against Mistral-7B on instruction-following in human evaluation.

The main quality gap appeared on MMLU — a knowledge-intensive benchmark requiring precise factual recall — where the recurrent model trailed by roughly 4 points. This reflects the fundamental tradeoff: a fixed-size state compresses long-range context and loses some precision in the process.

The efficiency gains tell a more dramatic story. RecurrentGemma-9B sustains approximately 6,000 tokens per second in sampling throughput, and that throughput holds constant regardless of how many tokens have been generated. A Transformer’s throughput degrades as its KV cache grows and forces smaller batch sizes. At long sequences, the gap reaches two orders of magnitude — not a modest improvement, but potentially 100 times faster generation in long-form output regimes.

The third paper pushed the concept further than Griffin attempted. Titans ask: what if long-term memory weren’t a fixed-size state passively accumulating signal, but an active learner updating its own weights at every inference step?

The mechanism is a small multi-layer perceptron whose weights are updated via gradient descent at each token, based on how surprising the new input is. A learned forgetting gate controls what gets retained. This is literal test-time learning — a gradient update occurring inside the forward pass, with no separate training phase involved.

Titans come in three variants differing in how this neural memory integrates with local attention. The Memory as Context variant prepends retrieved memories directly to the attention window and delivers the strongest results on recall tasks. Memory as Gate uses the memory output to gate the attention branch for better efficiency. Memory as Layer stacks the components sequentially.

The benchmark results are striking. At 760 million parameters, Titans achieved a WikiText perplexity of 18.61 — compared to 25.21 for an equivalent Transformer baseline, a 26% reduction. On a needle-in-haystack benchmark requiring precise retrieval across 16,000-token sequences, Titans scored 97.4% accuracy against Mamba2’s 5.4%. The paper also reports outperforming GPT-4 and Llama-3.1-70B on the BABILong long-context benchmark, with demonstrated context windows exceeding two million tokens. A formal theoretical result in the paper proves Titans can solve problems beyond the TC0 complexity class — making them more expressively powerful than standard Transformers on state-tracking tasks.

An honest reading of the evidence surfaces real limitations. Pure recurrent models cannot perform multi-query associative recall — retrieving a specific value for a key seen arbitrarily far back — due to their fixed-state compression. Griffin’s local attention window patches this within 2,048 tokens, but beyond that the recurrent state must hold the answer and may not do so with full precision.

A 2025 study found that when context exceeds the effective capacity of Recurrent Gemma’s recurrent state, performance degrades sharply — and a simple chunk-based overflow prevention strategy improved its Long Bench score by 50%. That’s a substantial gain from a straightforward fix, implying meaningful underperformance on long-context tasks out of the box.

The scale question remains open. Recurrent Gemma has been demonstrated at 9 billion parameters. No published results exist at 70 billion or beyond — where most frontier deployment decisions are made. An independent reimplementation of Titans also raised questions about whether the performance gains come primarily from the neural memory module or from the local attention component, an empirical debate that remains unresolved.

Google DeepMind is not alone in exploring alternatives to global attention. Mamba introduced selective state space models. RWKV developed linear attention with recurrent characteristics. AI21 Labs shipped Jamba — a hybrid combining Mamba, Transformer layers, and Mixture of Experts. The sub-quadratic attention space has become one of the most active research fronts in the field.

The Transformer is not disappearing. Global attention will remain dominant at the frontier for the foreseeable future. But for the first time in years, there is a production-validated, open-weight alternative from a major AI lab — grounded in a clear theoretical argument, demonstrated at meaningful scale, and available for anyone to build on right now.

Whether the community picks it up and what happens when these architectures are pushed to 70 billion parameters and beyond will determine whether this is a significant architectural shift or a compelling research detour. The evidence so far suggests it deserves to be taken seriously.

Authored by Deesha Chaware, Founding PMM, Synapt AI.