How Profitable is LLM Inference? Doing the Math on Kimi K3
A look at LLM inference economics (batch size, GPU count, and the Pareto frontier that sets token prices) applied to Kimi K3 with back-of-the-envelope math.
Moncef Abboud
Moncef Abboud blog and posts.
Latest posts
Distributed LLM Inference with llm-d
An introduction to llm-d, an open-source LLM-aware router that intelligently schedules requests across inference engines like vLLM using KV cache locality and GPU utilization.
Exploring Speculative Decoding: From Concept to Implementation
In this post, we explore speculative decoding through a concrete vLLM-focused implementation, covering draft models, EAGLE, MTP, and the tradeoffs involved.
Exploring Mixture of Experts: From Concept to Inference Engine
In this post, we dabble in Mixture of Experts (MoE) models through a concrete nano-vLLM implementation, exploring Triton kernels, expert parallelism, and other fun things.
Deep Dive into Efficient LLM Inference with nano-vLLM
A look inside a lightweight implementation of vLLM. KV cache, paged attention, tensor parallelism &multi-GPU support, etc.