Triton Inference Server on Kubernetes: Multi-Model Serving
vLLM is great for LLMs. Triton is what you reach for when you need to serve 20 different models on one GPU cluster.
Kubenatives
Production Kubernetes for ML/AI workloads: GPU infrastructure, control plane internals, and model serving patterns for engineers running inference at scale.
Live Last read · last published · next check
Latest posts
RDMA and InfiniBand: Why GPU Networking Is Different on Kubernetes
A 70B model split across 8 GPUs moves hundreds of gigabytes per second between nodes. Standard Kubernetes pod networking handles zero of that well. Here is what to run instead.
A/B Testing LLM Models in Production with Kubernetes
Swapping Llama 3.1 70B for your old 8B model in one deploy is how you break production. Here is how to do it safely with traffic splitting and real quality metrics.
API Server Latency: What Is Normal and What Is a Red Flag
Your API server p99 is 400ms. Is that fine or is the cluster about to fall over? Here are the numbers that matter and what to do when they go wrong.
GPU Monitoring with DCGM Exporter: The Metrics That Matter
Your GPU nodes are running. nvidia-smi shows green. But are your GPUs healthy, efficient, and about to fail? DCGM tells you.
Resource Requests and Limits for GPU Workloads
Get requests wrong and your pods are Pending. Get limits wrong and they OOM. Here is how to size them correctly for GPU inference.
Autoscaling Inference Workloads: HPA and KEDA for GPU Pods
GPU pods are expensive. Running 4 replicas at 3 AM when traffic is zero wastes thousands per month. Here is how to scale them automatically.
Kubernetes Upgrade Strategy: kubeadm Cluster Upgrades Without Downtime
Kubernetes drops support for old versions every 12 months. Here is how to upgrade without breaking production.
Network Policies in Practice: When Your Pods Cannot Talk to Each Other
You implemented network policies for security. Then DNS broke. Then inter-service communication broke. Here is how to do it without breaking everything.
Architecture Template: GPU Node Pool Setup
Complete YAML for a multi-tier GPU cluster with taints, tolerations, affinity, quotas, and priority classes. Copy, configure, deploy.