When higher eval scores just mean your model learned a copy shortcut. Why utility-under-the-prior is a flawed proxy, and how to mine the hard negatives your agents actually need to survive.
∙ Paid
You’re in a Senior ML Engineer interview at Meta and the interviewer asks:
“You’re building synthetic tool-call training data with Toolformer’s filter, keep the call if the tool output raises the likelihood of the correct continuation. Your eval improves. Production accuracy doesn’t move. What’s wrong with the filter?”
Don’t say: “The tool output could be…
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.