A practising architect, four weeks into a residential project, asks the model to refine the kitchen detailing. The model returns a confident, well-composed proposal — and quietly drops the constraint, established in week one, that the client has a wheat-allergic child and the kitchen must support strict separation.
The architect reads this as the model failing. It is not. The model is behaving exactly as its architecture says it will, and the architect does not yet know the architecture.
Article 1 of this series established that AI is probabilistic — it samples from a distribution rather than computing a function. This article is one layer down. It shows the specific mechanism: the model does not read your brief the way you wrote it, does not store it the way you imagine, and does not “remember” anything across a long conversation in any sense an architect would recognise. Tokenisation, attention, context windowing—that is the operational machinery deciding whether your project survives iteration or quietly disintegrates over four weeks.
Get this layer right and a discipline appears. We call it the context budget. Once you can see it, the difference between productive briefing and conversational drift stops looking like model behaviour and starts looking like your own allocation choices.
The first correction is the simplest, and it surprises people.
The model does not see words. It sees tokens, subword fragments produced by a tokeniser that runs before the model itself ever touches your text. The English word architecture is not one unit. Depending on the tokeniser, it is roughly [”arch”, “itect”, “ure”]. Vernacular might split into [”ver”, “nac”, “ular”]. A common word like house survives as a single token. A proper noun like Bengaluru may become four or five.
Why this matters operationally, not academically:
Word-counting fails as a length estimate. A 500-word brief can be 700 tokens or 1,200, depending on what’s in it. Hindi, Marathi, or Kannada terms inside a Karnataka brief consume more tokens per word than the English around them—the tokeniser was trained mostly on English and fragments less common scripts more aggressively.
Pricing and limits are quoted in tokens, not words. The “1M context window” frontier models now advertise [1] is measured in tokens, not pages.
Reasoning happens token-by-token, one prediction at a time, each shaped by everything that came before.
You do not need to memorise tokeniser rules. You need to verify the asymmetry once, with your own eyes. Paste a paragraph from your last project brief into a tokeniser playground [2] and watch how the model fragments it. From what we’ve seen in workshops, this five-minute exercise corrects more bad prompting habits than an hour of “prompt engineering” instruction.
Once the brief has been tokenised, the model begins predicting the next token. To do that, it needs a way of deciding which earlier tokens matter most for what comes next. That mechanism is called attention, and it is the load-bearing operation inside every transformer-based language model in production today.
A pragmatic mental model, kept at intuition level. Imagine a junior architect drafting the next sentence of a project brief. Before they write, they glance back across every prior sentence and decide which ones matter most for what comes next. The setback constraint matters for the massing paragraph; less for the materials paragraph. The wheat-allergy note matters for the kitchen; less for the entry portico. They are attending to some, ignoring others, and the quality of the next sentence depends on whether they attended to the right ones.
That is what the attention mechanism does, at scale, for every token the model predicts.[3] One mental model among several the engineering literature uses; it captures the pragmatic behaviour, not the underlying math.
Two consequences fall out immediately. The model is not reading sequentially — it is reaching back across the entire prior context every time it commits to a token. That is what makes long, well-structured briefs powerful: a relevant detail from twenty pages back can be pulled forward if attention selects it. And attention is not free. It is a finite weight, distributed across whatever fits in the window. The more you put in front of the model, the thinner the weight spreads. This is where the discipline begins.
The context window is the maximum number of tokens—your prompt plus the model’s response—the model can hold in active working memory for one exchange. Everything inside the window is potentially attendable. Everything outside the window does not exist for that exchange.
The numbers have moved fast:
GPT-3 (2020): 4,000 tokens
GPT-4 (2023): 8,000
GPT-4 Turbo (2023–2024): 128,000
Claude 3 (2024): 200,000
Gemini 1.5 (2024): 1,000,000
May 2026 frontier — Claude Opus 4.7, Gemini 3.1 Pro, the GPT-5.x family — has converged on roughly 1,000,000 tokens [1]
As of the time of writing, the differentiation between frontier models is no longer in window size. It is in how well attention actually holds across that window — a different question, and the more important one.
A complex residential brief—site, programme, climate, materials, client preferences, references, constraints, codes, prior conversation—can easily exceed 50,000 tokens before the first iteration. A million-token window sounds like infinite headroom. It is the ceiling, not the working zone. The working zone is whatever the model can attend to well inside that ceiling, a much smaller and more uneven space.
The cleanest finding in the long-context literature is also the most useful for an architect to know. Even when the model technically holds a long context inside its window, attention does not weight that context evenly. It tends to favour the very recent tokens—the last few exchanges—and the very early tokens—the original prompt. The middle degrades.
The result has a name: Lost in the Middle.[4] Liu and colleagues documented it in 2023, and it has been confirmed across newer long-context models since. A constraint dropped into the middle of a 60,000-token conversation will be attended to less reliably than the same constraint stated in the first paragraph or the most recent message, even though all three are technically inside the window.
This is structural, not incidental – a property of how attention scales when the input gets long. In our reading of the current data, no frontier model in May 2026 has solved it; they have softened the U-shape, not removed it.
The model does not “forget” your project across a long conversation in any human sense. It attends less reliably to the parts that have drifted into the middle of the window. From the model’s side, a salience problem. From yours, the tool losing the plot.
That is what was happening in the wheat-allergy example.
Three weeks into a residential project. The designer briefed Claude in week one with the full context: Karnataka site, vernacular palette, tight budget, programmatic constraints – and, anchored explicitly, a wheat-allergic child and a kitchen requirement that strict separation be maintained between wheat-handling and non-wheat-handling zones.
By week two the thread is 80,000 tokens deep—massing, structural rationalisation, light wells, courtyard placement, façade studies, material substitutions for budget pressure, three rounds of plan iteration. The wheat-allergy note, anchored 60,000 tokens earlier, has been quiet for the last 50,000.
The designer asks for kitchen-island detailing. The model returns an elegant layout - open on three sides, single continuous worktop, integrated dining edge. Architecturally crisp. Operationally dangerous. The wheat-allergy constraint has fallen below the model’s effective attention threshold for the kitchen-detailing context.
The designer’s first instinct is to blame the model. It forgot the brief.
That is not what happened. The brief is still inside the window. The constraint is still tokenised. It just has not been attended to in fifty thousand tokens of intervening work, and the kitchen-detailing prompt did not surface it. The model behaved exactly as Lost in the Middle predicts.
The literate move is not a longer prompt. It is to treat critical, non-negotiable constraints as re-anchors, restated at the top of every major iteration. Not because the model is stupid. Because attention, even at one million tokens, is finite. The architect’s job is to spend it well.
This is the context budget in operation. Every token in front of the model is a withdrawal against a finite attention bank. The discipline is deciding what to deposit at the top of each iteration so the right things get attended to when it counts.
A few moves follow from the machinery—the floor of the practice, not its ceiling.
Treat critical constraints as re-anchors, not as established facts. Anchored once is not the same as attended to now. Re-state the wheat allergy at the top of the kitchen iteration. Re-state the budget ceiling at the top of the materials iteration. The repetition is not for you. It is for the model’s attention.
Structure the brief so load-bearing constraints sit at the edges of the window, where attention holds most reliably. The opening prompt and the most recent message are the strongest positions. The middle is where things fade.
Treat long conversations as a context cost, not a free archive. Every token of prior chat dilutes the attention available to the current question. Sometimes the right move is a fresh thread with a tightened brief, not another follow-up in a thread that has accumulated 90,000 tokens of background.
Stop expecting the model to remember. Start designing for what it can attend to in this window. Those are not the same skill, and the difference is roughly the entire gap between fluent and frustrated AI use.
The model does not remember your project. It samples from whatever you fit in the window. The question is no longer what does the model know—it is what did you decide to put in front of it this iteration.
That is the architect’s question. It always was.
For why image-AI behaves differently from text-AI even though the same probability principle applies, read Part 3.
The context budget is teachable, but not by reading. It gets installed against your own live projects, in a room where the briefing patterns get rebuilt under deadline pressure. We run a one-day workshop — the AI Design Director Protocol — for exactly this: the room where briefing becomes a discipline of context budget, so your model stops losing your project halfway through it.
Want to know more about the Protocol? Head to protocol.rbdsailab.com
Sources
[1] Frontier-model context windows have converged on approximately 1M tokens as of May 2026 across Anthropic (Claude Opus 4.7, Sonnet 4.6), Google (Gemini 3.1 Pro), and OpenAI (GPT-5.x family). The differentiation between frontier models is now in how well attention holds across the window, not in the window’s nominal size.
[2] OpenAI Tokenizer playground — https://platform.openai.com/tokenizer (OpenAI models). For cross-model tokenisation across GPT, Claude, Gemini, and Grok in one tool: GPT for Work tokenizer — https://gptforwork.com/tools/tokenizer
[3] Cho, Aeree, et al. Transformer Explainer (Georgia Tech / Poloclub). https://poloclub.github.io/transformer-explainer/
interactive in-browser GPT-2 visualisation of next-token prediction. Companion intuition video: 3Blue1Brown, Attention in transformers, step-by-step | Deep Learning Chapter 6.
https://www.youtube.com/watch?v=eMlx5fFNoYc
[4] Liu, Nelson F., Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 2024. https://aclanthology.org/2024.tacl-1.9/ (preprint: https://arxiv.org/abs/2307.03172)
I’m Sahil Tanveer of the RBDS AI Lab, where we explore the evolving intersection of AI and Architecture through design practice, research, and public dialogue. If today’s post sparked your curiosity, here’s where you can dive deeper:
Read my book – Delirious Architecture: Midjourney for Architects, a 330-page exploration of AI’s role in design → Get it here
Join the conversation – Our WhatsApp Channel AI in Architecture shares mind-bending updates on AI’s impact on design → Follow here
Learn with us – Our online course AI Fundamentals for Lighting Designers is power-packed with 17+hrs of video content through 17 lessons → Enroll Here
Explore free resources – Setup guides, tools, and experiments on our Gumroad
Watch & listen – Our YouTube channel blends education with architectural art
Discover RBDS AI Lab – Visit our website
Speaking & events – I speak at conferences and universities across India and beyond. Past talks here
📩 Enquiries: sahil@rbdsailab.com | Instagram
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.