Continuous Batching in LLMs

The technique behind vLLM's 23x throughput jump and the default scheduler in every serving engine.