Inference Python Package

The inference Python package: load models, run inference in your own process, execute Workflows, and stream video.

The inference Python package is the core open source library that powers Roboflow's computer vision deployment stack. It provides model loading, pre/post-processing, GPU/CPU optimization, and Workflows execution, callable directly from Python.

The Inference Server wraps this package and exposes it over HTTP (distributed as a Docker image with all dependencies installed), but you can also use inference directly in your own scripts and applications.

How the pieces fit together

Inference has several components that work together to serve computer vision models:

• inference - core Python package for model loading, inference, and Workflows execution.

• inference-sdk - lightweight Python client for communicating with an Inference Server over HTTP.

• inference-cli - command-line tool for managing the Inference Server and running common tasks.

• Inference Server - HTTP server (Docker) that wraps the inference package as a REST API.

For how these pieces behave at runtime (request routing, parallelization, microservice and appliance patterns), see Inference Architecture.

Multi-backend support

Inference 1.0 supports multiple inference runtime backends: ONNX, TensorRT, Hugging Face, and PyTorch. It automatically selects the fastest available backend for your hardware. For example, if you have an NVIDIA GPU or are running on a Jetson device and a TensorRT engine is available for the model on your platform, Inference uses TensorRT by default.

Installation

We recommend using a Python virtual environment (venv) to isolate the dependencies of Inference.

To install Inference via pip:

pip install inference

If you have an NVIDIA GPU, you can accelerate your inference with:

pip install --extra-index-url https://download.pytorch.org/whl/cu124 inference-gpu
# please adjust the --extra-index-url to the CUDA version installed in your OS
# https://download.pytorch.org/whl/cu<major><minor>, for instance https://download.pytorch.org/whl/cu130 for CUDA 13.0

Quick example

You can run inference either against an Inference Server over HTTP (with inference-sdk) or directly in your process (with the native inference package). The native get_model() call loads the model into your script and returns an object you can call .infer() on; the HTTP client sends the image to a server that does the same.

inference (native)

from inference import get_model

model = get_model(model_id="rfdetr-small")
results = model.infer("https://media.roboflow.com/inference/people-walking.jpg")

inference-sdk (HTTP client)

from inference_sdk import InferenceHTTPClient

client = InferenceHTTPClient(
    # api_url="http://localhost:9001",  # for self-hosted
    api_url="https://serverless.roboflow.com",
    api_key="ROBOFLOW_API_KEY",
)
results = client.infer(
    "https://media.roboflow.com/inference/people-walking.jpg",
    model_id="rfdetr-small",
)

To use models that require an API key, set the ROBOFLOW_API_KEY environment variable or pass it directly:

model = get_model(model_id="my-project/1", api_key="ROBOFLOW_API_KEY")

See the Native Python API page for a more detailed walkthrough with visualization, and Run a Model for the server-based path.

Inference Pipeline

InferencePipeline runs video inside the same Python process as the inference package. Use it only when you choose direct Inference Library execution and need access to its custom logic or sink interfaces. Applications that use an Inference Server or Serverless should use WebRTC Streaming.

from inference import InferencePipeline
from inference.core.interfaces.stream.sinks import render_boxes

pipeline = InferencePipeline.init(
    model_id="rfdetr-large",
    video_reference="https://storage.googleapis.com/com-roboflow-marketing/inference/people-walking.mp4",
    on_prediction=render_boxes,
    api_key="ROBOFLOW_API_KEY",
)

pipeline.start()
pipeline.join()

The code above does object detection annotation directly (through the render_boxes sink). For more information, see the Inference Pipeline page.

Contributing

Inference is open source. The source code, issue tracker, and contribution guide live in the roboflow/inference repository; see CONTRIBUTING.md to get started.

Next steps

• Native Python API - load models and run inference in your own process.

• Inference Pipeline - run models on video streams.

• Model Weights Download - weight caching and persistent storage.

• Benchmarks - measured throughput on common hardware.