Reading · Products Users Love

This page did not load. You can still read it on the original site — the toolbar below keeps your place in the directory.

The invisible pipeline underneath your AI tools, and why understanding it changes how you build

✨ In this post I’ll cover ✨:

What actually happens when an LLM answers a question
How the invisible layer changes our experience as vibe-coders
Why consistency in AI tools is more complex than any single fix
What happens when over-trust meets an invisible system
Who is responsible for closing that gap

A few weeks ago I was testing the AI scanner I am building. The idea behind it is to replicate the steps I follow in my UX audits and research consulting projects and turn them into an AI-powered tool. Part of what I am working through is figuring out which elements can be confidently evaluated by AI, which can be solved by a classic algorithm, and which still need my judgment and input. One of the things I was testing was an accessibility scanner: give the model a screenshot and the DOM elements of an app, run it against a framework I provided, and get a list of issues with prompt recommendations to fix them.

I ran the exact same app three times. Every time, the scanner showed a different number of issues. This meant the framework I provided wasn’t being consistently followed, which left a lot of margin for randomness in what a user would actually get. So I went deeper to understand if and how I could fix it. The tool suggested setting the LLM temperature to 0.1 and called it resolved. The results were still inconsistent.

One thing was clear: I needed to fully understand how this works before I could make a good decision and build something reliable. And that’s when I started asking a different question. Everyone is using AI to build products right now, but who is researching the vibe-coder’s experience? We are all using a tool that, as I was about to learn, works very differently from the mental model most of us carry around.

Products Users Love is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Vibe-coders are users

There is a lot of conversation in UX and product circles about the experience of people using the products we build. We run usability tests, we map user journeys, we study how people form mental models of systems. We apply all of this to our users.

We almost never apply it to ourselves.

Vibe-coding, the practice of building products through natural language conversations with AI tools, is a user experience. The journey from idea to a working product, navigating prompts, interpreting outputs, debugging unexpected behaviour, deciding when to trust the result and when to push back, is a user journey. And like all user journeys, it is shaped by design decisions that were made before we ever arrived.

The difference is that most of those decisions are invisible. And unlike traditional software, where a confusing interface is at least visible, the decisions shaping our experience as vibe-coders are buried several layers below what we can see.

To understand what happened in my scanner, I had to learn what is actually happening every time an LLM answers a question.

What is actually happening when an LLM answers your prompt

Most of us interact with AI tools through a chat interface. You type something, it responds. It looks simple. Underneath that simplicity, there are at least seven distinct layers operating every single time, none of which are visible, and all of which shape the output you receive.

Training data is where it starts, long before your conversation begins. The model learned from enormous amounts of text. It has no live knowledge, no access to your product, no understanding of your specific context. It knows patterns. Everything it generates is a prediction based on those patterns, not a lookup, not a search, a prediction.

Tokenization is how the model reads your words. Before anything is processed, your text is broken into tokens, which can be whole words, parts of words, or punctuation. These tokens are converted into numbers, because the model does math, not language. One token is roughly 0.75 words. Your carefully written prompt is, to the model, a sequence of numbers.

The system prompt is a set of hidden instructions that conditions the model’s behavior before your message even arrives. When you use any AI tool, there are instructions you never see that are shaping every response you get. This is part of the invisible layer that product teams set once, without testing it with real users.

The context window is the model’s working memory. It holds your prompt, the conversation history, the system prompt, any documents or images you have attached. When it fills up, the model starts losing the beginning of the conversation. Research also shows a U-shaped pattern in how models process information: what is at the beginning and end of the context is more reliably used than what sits in the middle. Where you put information in a long conversation actually matters, and most vibe-coders have no idea.

The attention mechanism is how the model decides what is relevant to what. It calculates relationships between tokens, assigning weights across everything in its context simultaneously. It is not reading your prompt the way you read it. It is calculating relevance scores across the entire input at once, which is why small changes in phrasing can produce surprisingly different outputs.

Generation parameters are the invisible dials. Temperature, top-P, top-K, frequency penalty, presence penalty. Set by the product team or developer, invisible to the user, evaluated against mathematical metrics rather than real user experience. Temperature is the one I stumbled into, and it deserves its own section.

Safety filters are the last gate before a response reaches you. Before showing a reply, safety systems scan for policy-violating content and can block, rewrite, or modify responses. Another invisible layer that can change an output with no signal to you about why.

Seven invisible layers, none of them designed with the vibe-coder’s experience in mind.

And yet here we are, building products through natural conversation, using language so instinctively that we forget we are operating a complex probabilistic system. That is what makes this moment in human-computer interaction so fascinating to me. A natural interaction paradigm is shifting how we relate to technology, and quietly shifting our psychology along with it, without most of us noticing.

Temperature: where it came from and why the fix wasn’t really a fix

When my tool set the temperature to 0.1, it was applying a common engineering convention, not a researched recommendation. To understand why that matters, it helps to know where temperature actually comes from.

Temperature is a concept from 19th century physics. It traces back to Ludwig Boltzmann, who studied how energy distributes among particles in a gas. The same mathematical formula that describes how molecules behave at different heat levels was borrowed by machine learning researchers to control how neural networks make decisions. When Geoffrey Hinton built early neural networks in 1985, he modeled them explicitly on statistical physics and named the parameter accordingly.

So the parameter my tool quietly set to 0.1 started life as a formula for gas molecules. It traveled from 19th century physics into machine learning, and from there into every AI product you have ever used. At no point in that journey did anyone ask: what does this feel like to the person building with it?

The ranges developers use today, low temperature for precision tasks, medium for general use, high for creative work, have no rigorous research behind them. They are patterns people converged on through trial and error, evaluated against mathematical metrics, not against real people doing real tasks. The fact that 0.1 was suggested as a fix is because it follows a convention. Not because anyone tested it with real users, or because it was the right fix for my specific problem (it wasn’t).

The harder truth: non-determinism is not a bug

What happened with my accessibility scanner was not a temperature problem. It was a counting problem. I was asking the model to apply a defined framework to a defined input and return a consistent count of issues. That sounds like exactly the kind of structured, deterministic task a computer should handle reliably. But LLMs work differently.

OpenAI, Anthropic, and Google all state explicitly in their documentation that their APIs can only be “mostly deterministic,” regardless of what temperature is set to. Anthropic’s documentation states directly: results will not be fully deterministic even at temperature zero. This is not a bug in my tool. It is a documented, acknowledged, fundamental property of the systems we are building with.

A useful way to think about it: a calculator gives the same result every time. A language model is more like a complex system that examines context, weights, and usage patterns to decide what comes next. Most of the time the answer is stable. Sometimes it is not.

This problem is serious enough that the former CTO of OpenAI recently launched a lab specifically to address non-determinism in LLMs. The issue I stumbled into while building an accessibility scanner is a primary research focus at the frontier of AI development.

Reframing AI use

The most useful reframe I have found is this: stop asking “will I get the same output every time?” and start asking “will I get outputs that are reliable enough for what I need?”

Those are different questions and they require different approaches. The first is an engineering question with no complete answer. The second is a research question, one that requires understanding your use case, defining what “reliable enough” actually means for your users, and testing against that definition rather than against solution identity.

For my accessibility scanner, this means I may need to reconsider whether an LLM is the right solution for counting issues at all. A classic algorithm built on my framework would likely perform this task more reliably. That is not a failure of the tool, it is a research finding. Understanding how the system actually works is what allowed me to ask the right question: is this the right tool for this specific problem?

This is the reframe I think every vibe-coder needs. Not “lower your expectations for consistency,” but “understand the system well enough to know where LLMs are the right fit and where they are not.” Every tool has tradeoffs. The problem is not the tradeoff itself, it is making that decision without knowing the tradeoff exists.

Building with AI means making implicit decisions about reliability, consistency, and trust at every step. Most vibe-coders are making those decisions without the information they need, because the tools they are building with were never designed to surface it.

And the stakes are not always a slightly inconsistent accessibility issue count.

In late 2025, an AI agent used internally at Amazon caused a 13-hour AWS outage after autonomously deciding to delete and recreate a live production environment. Amazon called it user error, not AI error, attributing the problem to misconfigured access controls. But what is user error, really, when the user was never given an accurate mental model of what the tool could independently decide to do?

Around the same time, Jason Lemkin, founder of SaaStr, documented a nine-day vibe-coding experiment that ended when a Replit AI agent deleted his entire live production database during an active code freeze, wiping records for over 1,200 executives and companies. An analyst reviewing the incident explained that the model does not know what it does not know. It predicts what sounds right. To an LLM, there is no bigger picture.

These are not stories about AI being dangerous. They are stories about over-trust. And over-trust is a direct consequence of an inaccurate mental model. When we do not understand how a system actually works, we trust it in the wrong places, with the wrong tasks, at the wrong moments. This is true whether you are a vibe-coder building an accessibility scanner, an engineer deploying an AI agent, or an executive handing a code freeze to a tool that cannot conceptually understand what a freeze means. The main problem is the gap between what the interface implies and what the system actually is.

Who is responsible for this gap?

Vibe-coders are users. The experience of building with AI is a user experience. And like all user experiences, it is currently being shaped by a set of invisible decisions that were never tested with the people living through them.

Temperature came from physics. Generation parameters were evaluated against mathematical proxies. Non-determinism is acknowledged in documentation that most builders never read. The interface looks like software we already know how to use, while underneath it behaves in ways that none of our existing mental models prepare us for.

🚀 Hi! I’m Andreea, an academic HCI researcher (gesture recognition and virtual reality interfaces) turned UX researcher that helps product teams build products users actually love. I conducted hundreds of research studies for tech companies seeking clarity and I started this newsletter to share real examples and stories from my experience, so teams can do better research and build better products.

If you found this article helpful, I would be super grateful if you shared it with others.

I also love great discussions and debates, so I’m excited to hear your thoughts in the comments.

AI temperatures, mental models, and the UX challenges of vibe-coding