Week 33, 2026 - Traces as Documentation, the Observability Land Grab and Search I Host Myself
Agents read traces where humans read dashboards. Dynatrace bought Arize for $915M, Nini says my posts are too generic and I moved to a self-hosted SearXNG.
Soumendra Kumar Sahoo
Soumendra Kumar Sahoo is an AI Observability Architect who writes about enterprise AI platforms, engineering leadership, observability, evaluation, reliability, and open-source ecosystems.
Latest posts
How Distributed Tracing Works: Scrub It, Poke It, Break It
How does one request keep its identity across a dozen services? I animated the W3C Trace Context spec so you can scrub, poke and break each header yourself.
Week 32, 2026 - The EU AI Act as an Observability Spec, Overruling My Research Agent, and Leaving Conductor
I read the EU AI Act as an observability spec, admitted my research agent picks better stocks than I do, lost two subagent reports and switched to Orca.
Week 31, 2026 - Consolidating Tools, AI Interview Cheating, and Agents That Delete Your Skills
I consolidated my tools, published my AI-interview-cheating research, studied the OpenAI-Hugging Face incident, and lost skills to a silent backup failure.
Stop Trying to Spot AI Cheating. Design Interviews That Expose Shallow Understanding
How do you deal with AI cheating in technical interviews? Eye contact, pauses and pasted code are weak signals. Here is how to collect better evidence of real skill, without confusing ordinary AI use with identity fraud.
Week 30, 2026 - Session Limit Hacks, Human Touch, and AI Worship Cults
A trick to reset Claude Code's 5-hour session window for longer focused runs, why I now write my weekly notes letter by letter, a tour of the emerging cults that worship AI as god, readers in China translating my posts, and a fasting-tracker PWA I built.
Adding Langfuse Observability to Hermes Agent (Self-Hosted, the Hard Parts Included)
How do you add observability to Hermes Agent with self-hosted Langfuse? A walkthrough of enabling the bundled plugin, the SDK gotcha that silently kills tracing, per-model cost tracking on Coolify, and what the traces actually look like.
Week 29, 2026 - Signal vs Noise, Vendor Engineering, and Mapping My Readers
Cutting AI noise down to a few trusted signals, Charity Majors on vendor engineering as high-leverage work, Matt Pocock's guide to Claude skills, Kimi K3's frontier release, resuming my book-reading habit, a Proxmox disk-boot fix, and a world map of where this site's readers live.
Week 27, 2026 - Music-Evoked Chills, Forward Deployed Engineers, and Buddha Vihar
Why an old song can hit the body after a gap (aesthetic chills and the dopamine reward system), what a Forward Deployed Engineer really does, a visit to Mahabodhi Buddha Vihar, basic life support training, an automated weekly-note pipeline, and a VPS disk-growth fix.
Week 26, 2026 - Cheap Claude Tokens, Interview Cheating, and Navayana
Why proxies sell cheap Claude tokens in China, how AI assistants reshape my interviews, comparing Vedanta, Buddhism and Navayana, and a Proxmox disk-hang fix.
Week 25, 2026 - Verbose prompts as a reliability problem, 14 years in IT, and the one-rupee argument for taxes
I found verbose prompts are a reliability problem not just a cost one, marked 14 years in IT, cleaned up my site SEO, and made peace with paying income tax.
The Trends of Engineering in AI: Prompt, Context, Harness, Loop, and Evaluation
What's the difference between prompt engineering, context engineering, harness engineering, loop engineering, and evaluation engineering? A side-by-side comparison with diagrams for AI engineers building production systems.
Week 24, 2026: Second brain infrastructure, export controls, and the AI race leaves the chatbox
A government letter forced Anthropic to disable its top models. Also Project Prometheus, MCP connectors, agent observability pricing, and my Nini research work.
Who Am I?
On rivers, cells, chemistry teachers, and the illusion of a permanent self: a non-technical exploration of identity, impermanence, and the observer within.
Week 23, 2026 — OTel graduation, Langfuse pipeline, memory systems, and the agent infrastructure stack
I stood up a Langfuse tracing pipeline for my agents, wrangled ESM and CJS plugin hell, reworked my memory system, and fell for git worktrees in Conductor IDE.
Week 22 — CoverView API, Satori rendering, and the art of a single curl call
I built a CoverView API that renders cover images with Satori and no browser, turning cover creation into a single curl call, plus a JavaScript-free live clock.
My Subagents Lied to Me: What Happened When I Let AI Research Autonomously
How do you know if an AI agent is telling the truth? When my research agents fabricated 11 model names and 4 fake paper titles, I learned the hard way that autonomous AI research requires independent verification.
Week 21, 2026 · May 18-24: TILs, Clicker Games and Google I/O
I shipped a TIL site, an Odia 2048 game and a Chhena Poda clicker, learned how agent hallucination differs from LLM hallucination, and dug into Google I/O 2026.
Week 20, 2026 · May 11-17: Two Agents, a Workshop and the Leap of Faith
I stood up a second AI agent named Nini, got OpenTelemetry flowing into SigNoz, prototyped a self-healing LLM agent, and documented an NVIDIA Nemotron workshop.
Week 19, 2026 · May 4–10: Digests, Hermes Agent and the Self-Improving Loop
My assistant Lili set up a weekly digest, 44-feed RSS reader, Spotify control and self-improving skills, I launched openodia.com and studied the Hermes Agent.
Subscribe
Get the AI Observability Starter Kit Free guide + weekly insights from someone who actually builds this stuff. Every Friday, I send out: What I learned about AI observability that week One practical tip you can use immediately Links to the tools and patterns I’m testing in production Plus, when you subscribe, you get instant access to the AI Observability Starter Kit — a 10-page guide covering…
Week 15, 2026: Glasswing, Langfuse and the Claude Skills Explosion
Building WideScope pushed me into Langfuse traces, Kafka-buffered OTel, Cleric's self-learning SRE, Anthropic's Glasswing, GLM-5.1, and tokenmaxxing culture.
Week 14, 2026 Claude Skills, Code Internals and Buddhism
The Claude Code source leak, Anthropic Agent Skills as an open standard, and Karpathy's agent talk shaped my week, plus a slides site and a Buddhism deep dive.
Week 13, 2026: Supply Chain Attacks, Wide Events and MuleSoft OTel
The LiteLLM supply chain hack taught me to pin versions, plus a deep dive into wide events, ClickHouse observability, MuleSoft OTel, and Meta's Hyperagents.
Week 12, 2026 · OpenAI Eats the Python Toolchain, Claude Cowork Scheduling, and AI Observability
OpenAI acquiring Astral rattled me, I scheduled a daily AI digest in Claude Cowork, dug into OTTL and OTel error specs, and grounded my SLO and RED thinking.
11: Alerting & Service Level Objectives (SLOs)
How do you define SLOs and configure alerts in Grafana? Learn to build error budget burn rate alerts, symptom-based alerting and SLI definitions for production Python services.
10: Dashboards & the RED Method
What is the RED method for monitoring? Learn how to build actionable Grafana dashboards using Rate, Errors and Duration metrics, extended with Cost and Quality for AI services.
9: Sampling & PII Scrubbing
How do you sample traces and scrub PII with OpenTelemetry? Learn tail sampling to keep 100% of error traces and attribute redaction to remove sensitive data before export.
8.2: Building Your First OTel Collector Pipeline
How do you set up an OpenTelemetry Collector pipeline with Docker Compose? Step-by-step guide to configuring receivers, processors and exporters for Jaeger and Prometheus.
8.3: Operating the OTel Collector in Production
How do you deploy the OpenTelemetry Collector in production? Learn Agent vs Gateway topology, debugging tools, self-observability and production hardening strategies.
8.1: Why the OpenTelemetry Collector Exists
Why do you need the OpenTelemetry Collector? Learn the first-principles case for adding a Collector to your observability pipeline, with architecture diagrams and the 5-Why analysis.
7: Error Handling & Semantic Instrumentation
How do you record errors properly with OpenTelemetry in Python? Learn why try/except blocks hide failures from dashboards and how to make every error visible in traces and metrics.
6: Correlating Telemetry Signals , The Pivot Workflow
How do you correlate logs, traces and metrics in OpenTelemetry? Learn the Pivot Workflow to drill from a Grafana dashboard spike to the exact log line that caused it.
PyConf Hyderabad 2026: A Homecoming I Didn't Know I Needed
After years away from the Python India community, I walked into PyConf Hyderabad 2026 as a speaker. I walked out with a reminder of why I started.
Week 11: Langfuse, ragwatch and the AI Observability Grind
Self-hosting Langfuse on Coolify fought me over passwords and ClickHouse, I shipped ragwatch 0.1.7, added Phoenix to my RAG fork, and prepped my PyConf talk.
Week 10: OpenTelemetry for AI Agents, Context Engineering, and MCP Ecosystem
I studied openllmetry and OTel for AI agents, rethought AGENTS.md context files, explored n8n's MCP server, and started building ragwatch for my PyConf talk.
Week 08: Agentic Evals & Observability, Building in Rust, and Small Web Rabbit Holes
I studied METR's time horizon benchmark and agent evals, built a local retirement planner in Rust with iced, and wandered from Moon maps to the small web.
Miss One Weekend, Fall Behind One Month
I went on a weekend trip. I came back to a new Claude, a new GPT and an existential crisis. The pace of AI is no longer monthly. It's weekly.
Week 07: Realizing Ignorance in Agents & Observability, Ethical AI & Dropping Readwise
I realized I am only a level-two agent user, faced how immature our observability platform is, shipped my first sponsored article, and quit Readwise for good.
Smart, Fast, Easy: How AI Plays An Ethical Part In Modern Business Growth
How AI plays an ethical part in modern business growth.
Week 06: Learning Notes on Content Creation, AI Agent Evals, Prompt Caching & Mental Models
I unpacked prompt caching and KV reuse, explored Galileo agent eval metrics, added mental models like inversion, and shipped two chapters of my o11y series.
5: Custom Metrics with OpenTelemetry
How do you add custom metrics with OpenTelemetry in Python? Learn to use Counters, Histograms and Gauges with Prometheus and Grafana for production monitoring.
4: Distributed Tracing: Following a Request Across Services
How do you trace requests across microservices in Python? Learn distributed tracing with OpenTelemetry, W3C Trace Context and automatic context propagation.
3: Automatic Tracing with OpenTelemetry
How do you set up OpenTelemetry tracing in Python? Learn three levels of instrumentation: zero-code, programmatic and manual, with Jaeger visualization.
Week 05: OTel survey insights, moving to Rybbit, HDD costs, and Meta's Rust rewrite
I dug into the OpenTelemetry Collector survey, moved my analytics from Umami to Rybbit, priced 20TB HDDs, and marveled at Meta rewriting WhatsApp in Rust.
AI in AI: What I Learned Analysing AI in the Adult Industry
I went down the rabbit hole of AI in the adult industry. I expected simple chatbots; I found a sophisticated engineering stack pushing the boundaries of edge computing, privacy and opensource AI.
Week 04: Hello World
I start this journal after finding India's subsidized GPU portal, wrestling with Jellyfin transcoding and a dead mini PC, and linking my server via Tailscale.
2: Context Propagation
How do you propagate request context in Python async applications? Learn how to use ContextVars to keep request IDs consistent across async tasks and threads.
1: Structured Logging in Python using Loguru
What is structured logging in Python? Learn how to replace text logs with JSON-based structured logging using Loguru for production observability.
Resurrecting an Abandoned Project: CoverView
I forked CoverView and added pattern backgrounds, color palettes, platform presets and fixed Unsplash bugs. Here's my journey of improving an abandoned opensource tool.