How Real-Time Multimodal AI Actually Works

Your smart speakers (Alexa, Google Home etc) don’t listen while you talk, they are waiting for you to finish. Every voice AI product has this problem and its frustrating. Thinking Machines Lab, the AI research lab founded by former OpenAI CTO Mira Murati, shipped an architecture that throws that assumption out entirely. It’s called an interaction model, and the first version, TML-Interaction-Small, is worth understanding in detail because it changes what “real time” actually means for Voice AI Agents.

When you write a letter to someone, you have to finish your entire thought, sign it, seal it in an envelope, and drop it in the mailbox before anything happens. The envelope goes to a sorting facility where someone reads your handwriting and types up a clean transcript. That transcript lands on a caseworker’s desk, who reads the whole thing, thinks it over, and drafts a complete reply. The reply gets typed up, printed, and mailed back. Days later, it lands in your mailbox.

That’s what a voice assistant does on every single turn. The envelope sealing is the voice activity detector deciding you’re done talking. The sorting facility is speech recognition turning your audio into text. The caseworker is the language model reading the whole transcript and writing a full response. The retyping and mailing is text-to-speech. And just like a letter, you can’t interrupt any of it once it’s sealed. If you think of something to add halfway through writing, too bad, the envelope already went out. If the caseworker misreads what you meant, you don’t find out until the reply lands in your mailbox

Thinking Machines calls this the bitter lesson of interactive AI. The caseworker in the middle, the language model, can get as smart as you want. It won’t matter, because the mail system around it was never built for back and forth. No matter how sharp the reply is, it’s still arriving as a letter, long after the conversation has moved on in your head. As the caseworker gets smarter, the gap between what they could say if you were on the phone and what the mail system lets them say only gets wider. The mail system is the ceiling, not the caseworker.

Here’s what how interaction model works end-to-end visually.

Now instead of envelope, let’s say we are making a phone call to a close friend. You talk over each other a little, you say “mm-hmm” while they’re still mid-thought, you catch yourself and correct mid-sentence and they just adjust without missing a beat. Nobody is running a stopwatch to decide whose turn it is, and nobody is sealing anything before it counts.

That’s the thinking behind TML-Interaction-Small model. Instead of processing a conversation as a stack of sealed envelopes, it slices the entire audio, video, and text stream into rigid 200 millisecond chunks and runs a forward pass on every single one. In each 200 millisecond window, it simultaneously listens to whatever came in and generates whatever it wants to say, all at once, the same way you and a friend on the phone are both listening and half-forming your next sentence at the same moment. The model learns what silence, hesitation, and interruption sound like directly from the call, because time itself is the structural backbone of its attention mechanism.

The number 200 milliseconds roughly matches human reaction time to auditory and visual cues, somewhere between 140 and 250 milliseconds depending on the sense. At that cadence the model reacts on the same clock a person does, which is why it can backchannel, interrupt, and self-correct.

Here is how multi-modal models work end-to-end visually