Optimizing vLLM startup time

vLLM has very slow time to first token (TTFT). On AMD Ryzen AI Max+ 395 it can take even 10 minutes depending on exact settings and model size. To make my user experience of using those things myself I need to have comparison of llama.cpp and vLLM startup time and performance. But to do this Continue reading "Optimizing vLLM startup time"