MLOps, AIOps, AI Observability System Design & Interview Preparation

An end-to-end, multi-tenant Retrieval-Augmented Generation platform built to be watched — wiring Langfuse tracing, DeepEval evaluation, and OpenLLMetry telemetry into a single request path so you can see, judge, and export what happens inside every LLM call.

👉 Companion code: code

👉 Preview: Link

👉 Premium guide: Guide

Most observability tutorials focus on a single tool in isolation. Production teams need three different questions answered at once, on the same request:

1. What happened inside this request? — Every LLM call, agent node, and retrieval step is attributed to a user and session. (Langfuse)

2. Was the answer any good? — faithfulness, answer relevancy, and contextual relevance scored against the retrieved context. (DeepEval)

3. Where do the signals go? — vendor-neutral OpenTelemetry spans are exported to whatever APM the organization already runs. (OpenLLMetry / Traceloop)

The project demonstrates that these three are complementary, not redundant, and that all three can attach to a single execution path with almost no coupling to the business logic.

Langfuse attaches through the standard LangChain callback mechanism, so no retrieval or generation code needs to know it exists. Three small helpers carry the whole integration:

• get_langfuse_callback() — builds the CallbackHandler from settings.

• get_langfuse_metadata() — attaches langfuse_user_id, langfuse_session_id, and feature tags so a flat span stream becomes a navigable, attributable tree.

• flush_langfuse() — forces the async span queue to drain inside a finally block, so a streaming endpoint never drops the tail of its trace.

The callback and metadata are passed into the LangGraph RunnableConfig; any node added later is traced automatically.

Browser (Next.js 15 dashboard, JWT + tenant context)
        │
FastAPI gateway & router
        │
LangGraph agent (StateGraph + Postgres checkpointer)
   ├── Langfuse  → trace every call/node/retrieval
   ├── DeepEval  → score the finished answer
   └── OpenLLMetry → emit OTel spans to any collector
        │
Hybrid retrieval (BM25 tsvector + pgvector, RRF fusion)
        │
PostgreSQL 16 + pgvector  →  Ollama (llama3.2:1b, nomic-embed-text)

• Backend: FastAPI (Python 3.12+), Pydantic v2, Uvicorn

• Frontend: Next.js 15 (App Router, React 19, TypeScript)

• Agent: LangGraph (StateGraph + Postgres checkpointer)

• Data: PostgreSQL 16 + pgvector (hybrid BM25 + vector, RRF rerank)

• Models: Ollama — llama3.2:1b (generation), nomic-embed-text (embeddings, 768-dim)

• Observability: Langfuse v4+, DeepEval, OpenLLMetry (Traceloop)

• Streaming: Server-Sent Events (NDJSON) · Containerization: Docker Compose

• Testing: pytest, DeepEval suite, headed Playwright E2E

# 1) Pull local models
ollama pull llama3.2:1b
ollama pull nomic-embed-text:latest
# 2) Configure .env (placeholders — never commit real keys)
LANGFUSE_PUBLIC_KEY=pk-lf-xxxx
LANGFUSE_SECRET_KEY=sk-lf-xxxx
LANGFUSE_HOST=https://us.cloud.langfuse.com
OPENLLMETRY_ENABLED=true
DEEPEVAL_THRESHOLD=0.7
# 3) Bring up the stack
docker compose up -d --build

• Dashboard: http://localhost:3000

• API docs: http://localhost:8000/api/v1/docs

• Traces: https://us.cloud.langfuse.com

Teaser Video Project

Your model passed every test, shipped at 94% accuracy, and three weeks later, it was quietly wrong — and no dashboard noticed.

That is the failure this book is built to prevent. The model is only 5–10% of a production AI system. The other 90% — data that drifts, models that decay in silence, costs that balloon, and predictions you cannot verify at deploy time — is the part that decides whether AI creates value or quietly destroys it. This is the field guide to that 90%.

Written for senior and staff engineers, platform architects, and anyone preparing for MLOps, LLMOps, or AIOps interviews, this is a system-design-level guide to operating AI in 2026 — classical machine learning, large language models, and the autonomous operations now emerging on top of both.

Inside, you’ll master:

• The 2026 MLOps lifecycle end-to-end: data and feature management, experimentation, CI/CD/CT, deployment strategies, and the four-layer monitoring model.

• AI observability done right — drift detection, data-quality signals, LLM tracing, and evaluation — the discipline of sensing failures before they become outages.

• LLMOps vs MLOps vs AIOps: what actually changes when the artifact is a language model, and how the three disciplines converge onto one control plane.

• FinOps for AI: cost attribution and chargeback, the compute pricing spectrum, spot training, and LLM token economics.

• Edge and federated MLOps: model compression, on-device runtimes, over-the-air fleet rollout, and privacy-preserving training.

• Governance, compliance, and risk — model cards, audit trails, policy as code, and the EU AI Act.

• Autonomous and agentic AIOps — self-healing pipelines and the road ahead.

Built to make you interview-ready. Every chapter ends with five detailed interview questions and worked answers spanning system design, production operations, debugging, and real-world case studies — the exact questions asked at senior and staff level.

Learn by running real code. The final part grounds everything in one open-source reference implementation: a multi-tenant Retrieval-Augmented Generation platform wired for observability with Langfuse tracing, DeepEval evaluation, and OpenLLMetry telemetry — FastAPI, Next.js, PostgreSQL + pgvector, LangGraph, and local Ollama models. Clone it, break it, and instrument it yourself.

Every chapter pairs deep concepts with an architecture diagram, comparison tables, production-grade code, and hard-won field notes.

You cannot operate what you cannot sense. Learn to sense it.