Build the Eval Pyramid: A Starter Harness for LLM and Agent Testing
Turn the eval pyramid into a working TypeScript harness with deterministic graders, isolated repeated trials, calibrated model judges, human review, and release gates.
Bharadwaj P
Terminal-inspired developer portfolio showcasing full-stack engineering projects, technical expertise, and software development insights. Built with Next.js, TypeScript, and modern web technologies.
Latest posts
The Eval Pyramid: Testing Systems That Do Not Behave the Same Way Twice
LLM and agent systems need more than unit tests, but that does not mean every check should become an LLM judge. The eval pyramid starts with deterministic checks and adds broader judgment only where the product needs it.
Measuring Agent Reliability: pass@k, pass^k, and LLM Judges
A system that succeeds once is capable. A system that succeeds repeatedly is reliable. Here is how to measure both, choose the right repeated-run metric, and use LLM judges without trusting them blindly.
RAG: How an LLM Answers From Documents It Never Saw
A chat model only knows what it saw in training. RAG is the plumbing that lets it answer from your private documents, today's news, or anything it was never trained on: a second kind of model that turns text into coordinates, a database of those coordinates, and a search step that runs before the LLM ever sees your prompt.
From Transformer to ChatGPT: The Part That Isn't the Architecture
A stack of transformer layers is not yet ChatGPT or Claude. This is the rest of the path: how text becomes tokens, how a raw next-word predictor turns into an assistant across three training phases, how LoRA customizes a model on a budget, and why everyone is racing for data and compute.
Transformers and Attention: A Plain-English Guide for Engineers
We use models built on transformers every day, yet the architecture itself usually stays a black box. It doesn't have to, and following it doesn't take heavy math. A visual, ground-up walkthrough: how words become numbers, how attention lets them shape each other, and how it all turns into a next-word prediction.
Notes on Cognitive Surrender
Reading notes from Addy Osmani on the difference between using AI as a tool and letting it quietly replace your thinking. Key research findings and heuristics worth keeping.
Why I Joined the Deep Atlas AI/ML Accelerator
I've been shipping with AI tools for a year. At some point the gap between using them and actually understanding them stopped feeling fine. This is about what I decided to do about it.
Navigating the AI Job Market: What Actually Matters
The AI job market is noisy and competitive. Here are the ideas worth keeping from a talk by Andrew Ng and Laurence Moroney on careers, technical debt, and what it takes to ship real AI work.