Speculative Decoding Is Not a Heuristic

The Basic Idea Speculative decoding [1] , [2] is an inference technique for increasing the throughput of an LLM. In its most basic form, there is a large/slow target model and a small/fast draft model . The draft model is used to quickly generate a draft sequence of the next tokens 1 . Then, a single pass of the target model is used to obtain next-token distributions 2 . A verification procedure…