The $195K Monthly Ranker That Only Learns to Mimic Retrieval [Edition #19]

V-Stream is a mid-stage short-form video company that recently crossed the 60 million Daily Active User (DAU) milestone. They have seen aggressive user growth of 15 percent quarter-over-quarter, driven largely by their expansion into South Asian markets.

Their engineering team built a two-stage recommendation engine that powers the primary home feed. Here is their setup.

When a user opens the app or scrolls to the next video, the recommendation service triggers a sequential two-stage pipeline.

Daily Active Users: 60M

Peak Throughput: 145,000 requests per second

Average Throughput: 85,000 requests per second

The ML Pipeline:

The Candidate Generator is a two-tower neural network trained daily on the last 24 hours of engagement and impression logs. It uses a cross-entropy loss where positive samples are items the user interacted with.

The Ranker is a multi-task Deep Learning Recommendation Model (DLRM) that retrains weekly on a rolling 14-day window. It consumes roughly 450 features, including heavy-side embeddings from the retrieval stage.

Current performance:

Average End-to-End Latency: 215ms

System Availability: 99.95 percent

Business impact: Flat watch time per session (32 minutes) over the last 2 quarters.

Costs:

Model Inference (GPU instances): $195,000 per month

Training Pipeline (Data processing/Compute): $65,000 per month

Total: $260,000 per month

Recent incidents:

Incident 1: Retrieval latency spiked to 800ms due to index fragmentation during a daily update, causing a 12 percent drop in DAU for four hours.

Incident 2: Ranker NDCG dropped 5 percent after a feature engineering change failed to account for null values in the follow-count field; recovered via model rollback.

Now let me show you what is actually happening here.

The Candidate Generator is trained on impression logs that have already been filtered by the Ranker. In the Architecture Overview, I noted that the generator trains daily on the previous day’s impressions. Because the generator only sees what the Ranker allowed through, it is no longer learning what the user likes from the entire corpus. It is learning to predict which items the Ranker is likely to score highly. Over six months, the generator has effectively become a distilled proxy of the Ranker. This explains why Recall@200 is increasing while product metrics are flat; the generator is getting better at finding the specific sub-set of videos the Ranker prefers, not the videos the user prefers.

The system trains the Retrieval model daily but the Ranker weekly. In a high-velocity short-form environment, the video distribution shifts hourly. By the time the Ranker is updated on Wednesday, the Generator has already spent three days adapting its embedding space to the old Ranker’s biases. This creates a drift where the two stages are optimizing against different versions of user reality. The Generator is chasing a moving target (the Ranker) that is itself lagging behind the actual content trend by up to seven days.

The teams are looking at local metrics that create a false sense of security. The Generator team celebrates when Recall@200 goes from 0.14 to 0.16. However, if the 200 candidates are all hyper-similar because the model has converged on a narrow slice of the Ranker’s preference, the Ranker has no “difficult” choices to make. The Ranker’s NDCG goes up because the candidate pool is now pre-sorted to its own internal logic. It is an echo chamber. The system is essentially reporting that it is very good at ranking a self-selected, non-diverse pool of content.