Every time you open Netflix, a recommendation model decides what you see.
Every time you pay, a fraud model decides whether to approve your transaction. Every time you search, an embedding model decides which results rank highest. Every time a title thumbnail loads, a scoring model decided which artwork to show you specifically.
That’s not one model. That’s hundreds. Running simultaneously. For 250 million users. At over 1 million requests per second.
The engineering challenge isn’t training better models — it’s routing the right request to the right model, on the right server, at the right time, without any of those 250 million users noticing the machinery underneath.
This is the story of Switchboard — Netflix’s custom ML serving router, published on their tech blog in May 2026.
Most ML content focuses on the fun stuff — model architecture, training pipelines, feature engineering. What gets ignored is the serving layer: the infrastructure that takes a trained model and makes it available to production traffic at scale.
At Netflix’s scale, the serving layer has to solve problems that don’t exist in textbooks:
Hundreds of different model types (personalization, payments, studio, search) must share infrastructure
Researchers need to experiment with new model versions without affecting users
A broken model version must roll back in seconds, not minutes
A single client service like the Homepage should not need to know which version of the ranking model it’s talking to
The same user might be in 10 different A/B tests simultaneously, each requiring different model behavior
Off-the-shelf solutions — AWS API Gateway, a standalone service mesh proxy — handle generic HTTP routing. They weren’t built for this.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.