vLLM

vLLM is an open-source LLM framework built on PyTorch.

Build and run from source

• Get the code:

$ git clone https://github.com/vllm-project/vllm.git

$ cd vllm

• Create a Python virtual environment:

$ python3 -m venv vllm_venv

$ source vllm_venv/bin/activate

• Install the requirements:

$ python3 -m pip install -r requirements/common.txt

$ python3 -m pip install -r requirements/cuda.txt

• Build vLLM:

$ python3 -m pip install -e .

• Run the vLLM server with a small model:

$ python3 -m vllm.entrypoints.openai.api_server \
    --model HuggingFaceTB/SmolLM2-135M-Instruct \
    --dtype float16 \
    --gpu-memory-utilization 0.5 \
    --max-model-len 128 \
    --enforce-eager \
    --disable-log-stats \
    --kv-cache-dtype fp8

The server is now running and listening on port 8000.

• In a separate shell, send a query to the vLLM server:

$ vllm chat --model HuggingFaceTB/SmolLM2-135M-Instruct \
    --quick "How many letters are in the English alphabet?"

Using model: HuggingFaceTB/SmolLM2-135M-Instruct
The English alphabet has 26 letters.