Speculative Tool Calling: Making LLM Agents Faster

This post summarizes work from my project in CSE-291A: Systems for LLMs and AI Agents at UC San Diego, under the guidance of Professor Yiying Zhang . The Problem LLM agents work in a loop: reason about what to do, call an external tool (web search, file reader, calculator, etc.), observe the result, repeat. The bottleneck isn't where you'd expect. Tool calls finish in milliseconds, but the model…