LLM Inference Interview Questions #12 - The Top-k Distractor Trap

You’re in a Senior AI Engineer interview at Anthropic and the interviewer asks:

“Your agent has 3,000 registered APIs, so you retrieve top-k tool specs into context per query. It starts confidently calling the wrong tool. Where’s the failure, and why is bumping k the wrong first move?”

Don’t say: “The model is confused, so I’d retrieve more candidates and improve the prompt.”

Wrong instinct. You just described the disease as the cure.

Here’s what’s actually happening.

You have two failure modes stacked on top of each other, and one number hides both:

a) Retrieval recall@k: was the correct tool ever in the context window?

b) Selection precision: given that it was there, did the model pick it?

If the gold tool never made top-k, the model has no abstention path. It doesn’t return “no tool fits.” It picks the nearest semantic neighbor and calls it with confident, well-formed arguments. That’s silent substitution, a schema-valid call to the wrong endpoint. Your validators pass. Your users don’t.

Why increasing k backfires:

Recall climbs logarithmically. Distractors climb linearly. At k=50 you’ve handed the model 40 semantic near-duplicates: search_docs, search_kb, query_index, that differ by one sentence in a docstring a human wrote for other humans, not for a retriever. Selection precision collapses faster than recall improves. You also eat context and position bias for free.

You’re not fixing the funnel. You’re widening the pipe and hoping.

What a senior does instead:

• Split the metric. Measure recall@k and selection-given-gold separately. They have different fixes. One number tells you nothing.

• If recall is broken: your tool descriptions are the bug. They’re written in domain vocabulary; user queries arrive in intent vocabulary. Index synthetic invocation queries per tool (Gorilla-style self-instruct), not the raw spec.

• If selection is broken: go two-stage. Cheap retrieve wide, then rerank narrow into 5-8 finalists. Route hierarchically: namespace first, tool second.

• Give it an exit. An explicit no_suitable_tool option converts a silent wrong call into a visible failure you can actually debug.

The answer that gets you hired: Wrong-tool calls are a retrieval problem masquerading as a reasoning problem, and the real bug is that your agent has no way to say “I don’t have the tool for this.” Fix the abstention path before you touch k.

#AIEngineering #LLM #AIAgents #MachineLearning #MLOps #ToolUse #RAG