Beyond the ratelimit - hosting the model yourself on a GPU with vLLM

Findings A single A100 with 80GB VRAM running Gemma4 31B with vLLM sustained 138 output tokens per second (serving 8 concurrent client agents). (extrapolates to a ceiling of ~500k tokens per hour) The bottleneck to more parallelism is the amount of VRAM available for KV cache; the model took roughly …