Optimizing reranker inference with vLLM

Rerankers sit in the critical path of every RAG pipeline. You retrieve 100 docs, rerank to top 10, then send to the LLM. If your reranker takes 500ms, that’s 500ms before generation even starts. At 10 requests/sec, you’re spending 5 full seconds just reranking.