Software engineer writing about LLM inference, AI infrastructure, distributed systems, Kubernetes, vLLM, and platform reliability.
Published: 2026-08-15 We run LiteLLM as an internal gateway in front of two kinds of models: local vLLM deployments and paid external models. At the time of the incident, the proxy was running LiteLLM 1.82.x with four replicas, Redis for shared state, and background model health checks enabled. The relevant part of the configuration looked like this: g eneral_settings : b ackground_health_checks :…
Published: 2024-01-01 Updated: 2026-08-01 Hi, I'm Andrey I'm a Staff Software Engineer at Severstal, focused on LLM inference platforms, AI infrastructure, and distributed systems. I am responsible for the architecture and technical development of DaVinci, a shared GenAI platform that supports enterprise AI products and coding agents. Its inference foundation currently runs self-hosted,…
Published: 2026-07-29 A recurring capacity-planning request for an internal LLM platform is: We have selected a model. Do we have enough capacity, and how many GPUs would we need for N users? The question is missing the information needed to answer it. The model and its serving configuration determine the minimum resources required to start one replica. They don't tell us how many requests that…
Published: 2026-07-26 In October 2024, Yuan Tang published AI/ML Innovation in the Kubernetes Ecosystem . The article described three important developments: Kubeflow Model Registry, KServe ModelCars, and TrustyAI. It also pointed toward multi-node serving, inference-aware gateways, speculative decoding, low-rank adaptation (LoRA) adapters, and APIs designed for generative AI. Since then,…
Published: 2026-01-28 Updated: 2026-06-18 Version scope: This guide targets vLLM 0.23.x and its default V1 engine. Metric names and semantics change between releases, so verify the /metrics output of the exact version and serving configuration you run before copying queries or alerts into production. A vLLM server can pass Kubernetes health checks while users experience slow or unstable responses.…
Published: 2026-05-19 I’m starting to learn modern C++, so I wanted a repeatable development setup on Apple Silicon: use a modern LLVM/Clang toolchain build projects with CMake and Ninja get proper language-server support in VS Code and Helix use sanitizers and static analysis from the beginning keep the setup practical This cheatsheet assumes that Homebrew is already installed.
Published: 2026-04-22 I wanted a local environment where I could: run Kubernetes schedule GPU workloads experiment with CUDA / inference / device plugins avoid renting cloud GPUs I used a Lenovo Legion laptop with an RTX GPU. The working stack was: Windows 11 + NVIDIA driver (WSL-enabled) WSL2 (Ubuntu 24.04) K3s (containerd) NVIDIA Container Toolkit NVIDIA device plugin The non-obvious part was…
Published: 2026-03-29 I use Visual Studio Code for work and personal projects. It has been my main editor for years, across several languages and ecosystems. Since I started using coding agents and chat-based interfaces, I've also wanted a more focused environment without the panels, notifications, and status updates. This is especially useful when I'm reading a book and reimplementing its…
Published: 2026-01-27 vLLM is an LLM inference engine designed to improve GPU utilization. Its key mechanism is PagedAttention , which manages the KV-cache without the memory waste common in traditional LLM serving stacks. More efficient memory use supports larger batches and steadier latency under load. KV-cache fragmentation In traditional LLM serving systems, the KV-cache (the keys and values…
Published: 2025-09-10 uv provides a Cargo-like, cross-platform workflow for managing Python projects , command-line tools , single-file scripts , and Python versions . Installation & Updates # Install curl - LsSf https://astral.sh/uv/install.sh | sh # Update uv self update Managing Python Versions uv can manage Python versions instead of pyenv, mise, asdf, or OS-specific installation methods: #…
Published: 2024-05-07 At a previous company, I worked on a decade-old .NET SaaS codebase that had been through several framework upgrades. The upgrades were incomplete, and many NuGet packages in the solution were outdated or deprecated by the time I became the engineering manager. I rely on linting, static analysis, and formatting tools in Python and Go projects, so I reviewed the equivalent…
Published: 2023-10-23 I use Apple Music for most of my music collection. I also buy rare or remastered CD releases that are sometimes distributed as FLAC files, which Apple Music does not support. I convert them to Apple Lossless Audio Codec (ALAC) before importing them into my library. Install FFmpeg with Homebrew: brew install ffmpeg Install FFmpeg with Homebrew Open the directory containing the…
Published: 2023-10-12 When development and deployment use different operating systems or processor architectures, generate the Python lockfile in the deployment environment. GitHub Actions can run this step without requiring the same environment on the developer's machine. I develop on macOS with Apple silicon, while most of my projects run on Linux. Resolved dependencies can differ between these…
Published: 2023-09-23 Keycloak User Federation can synchronize users from an external directory through LDAP or Kerberos. In a large Active Directory structure, the users required by one Keycloak realm may be spread across several organizational units. A custom LDAP filter can select only those accounts.
Published: 2023-09-06 Over the past few years, I have interviewed dozens of software engineers who could not explain how the services they developed ran in production. Infrastructure or platform teams often manage deployment, but that division of responsibility does not remove the need to understand a service's runtime behavior.
Published: 2023-08-29 Apache Kafka is often proposed whenever a project needs to process events, messages, or data streams. The choice is not always deliberate. A conventional message broker such as ActiveMQ may be enough, but Kafka's popularity can influence the decision. If Kafka has been selected deliberately or mandated by the infrastructure team, settle the following points before configuring…
Published: 2023-08-12 Summer of 2019. I'm joining a large retail organization that is undergoing a digital transformation. The role I've been hired for is a technical leadership role. The project I'm taking over doesn't even have a complete team yet, which means I'll be wearing all sorts of hats until I hire someone and delegate work. You could say, I'm the only "developer" on the team. Also, the…
Published: 2023-08-10 At work, I am involved in the development of a machine learning SDK and cloud services for privacy and data protection. Like almost every company in this space, we rely heavily on Python's scientific ecosystem. Because it's quite mature and depends on native library development that started years ago, getting these packages to work on new architectures can be tedious. I am…
Published: 2023-08-05 In January 2022, I joined the community of one of the proof-of-stake blockchains. To play with what the protocol and its ecosystem offered, I created a wallet account on the official website https://wallet.****.org . Apart from general curiosity, I was interested in how they achieved security in a browser, especially in the age of extensions and client-side vulnerabilities.…