# ai evals (blogs) — RSS Amplifier Recent posts from the 3 feeds in the RSS Amplifier directory that cover ai evals. Page: Feed: --- ## [The Normal Distribution, Part 1: Logarithmic Thinking](https://vishalbakshi.github.io/blog/posts/2026-08-17-normal-distribution/) _2026-08-17 · Vishal Bakshi · Vishal Bakshi's Blog_ This is my first of a series of articles on one of the most fascinating concepts: the Normal distribution. You may know it by the more colloquial term “bell curve”. Normal Distribution Many things are (almost) normally distributed: population human height, population blood pressure, birth weights, annual rainfall, standardized test scores, shoes sizes, body temperature…the list goes on. I’ve spent… ## [Integrating flash-kmeans into the ColBERT repo (initial setup)](https://vishalbakshi.github.io/blog/posts/20260814-flash-kmeans/) _2026-08-14 · Vishal Bakshi · Vishal Bakshi's Blog_ I’m dusting off my ColBERT maintenance hat and getting back into the rhythm of things. The first thing I’m working on for the next release is exploring the most likely integration of flash-kmeans ( arxiv , github , pypi ) as a replacement for faiss-gpu. flash-kmeans (2026 Yang, et al) is a ridiculously fast “IO-aware batched K-Means clustering implemented with Triton GPU kernels.” I’ll be digging… ## [AI Product Engineering Notes](https://hamel.dev/) _2026-08-12 · Hamel's Blog_ Notes from 13 sessions on evals, context, and systems. 9.5 hours of talks distilled into about 20 minutes of reading. ## [Thoughts on Supply and Demand from “The Power of PULL”](https://vishalbakshi.github.io/blog/posts/2026-08-07-pull/) _2026-08-04 · Vishal Bakshi · Vishal Bakshi's Blog_ In his book “The Power of PULL,” Rob Snyder articulates a sort of separation-of-concerns between customer/demand and business/supply. Demand is what buyers are trying to accomplish in their lives. Supply is what we build. These are two separate things. There is no such thing as demand for supply. Demand exists as something independent out there in the world, whether or not our supply exists and… ## [The Cavern & An Allegory of How Teams Find Technical Solutions to Business Problems](https://vishalbakshi.github.io/blog/posts/2026-08-04-the-cavern/) _2026-08-04 · Vishal Bakshi · Vishal Bakshi's Blog_ ChatGPT’s illustration of the Cavern The Cavern represents the space of solutions. It’s vast. Not infinite, but large enough that no one person can map it. Even large groups can’t map it in full. The Cavern is an interconnected network of tunnels and open spaces. It’s mostly dark, with some light coming in from the outside. Everyone in your organization has heard that he objects inside the Cavern… ## [Translation Labor](https://vishalbakshi.github.io/blog/posts/2026-08-02-translation-labor/) _2026-08-02 · Vishal Bakshi · Vishal Bakshi's Blog_ It takes translation labor to collaborate well on a cross-functional team. It’s the worthwhile cost of learning and building from other perspectives! What is translation labor? It’s a term I made up for this phenomenon. I like the sound of it. Comment below if there’s a better term! translation labor (noun) the work of mapping between two or more people’s understanding of a problem or solution.… ## [Thoughts on “The Go-Giver” Parable](https://vishalbakshi.github.io/blog/posts/2026-07-30-the-go-giver/) _2026-07-30 · Vishal Bakshi · Vishal Bakshi's Blog_ Snapshot of the book cover I recently came across a LinkedIn post sharing lessons from the book The Go-Giver (A Little Story about a Powerful Business Idea) by Bob Burg and John David Mann. I picked it up at a local bookstore and read it this week. The Go-Giver is a parable about a go-getter named Joe who, through his relationship with a man named Pindar, learns the Five Laws of Stratospheric… ## [“It’s Hard to Eval” Is a Product Smell](https://hamel.dev/blog/posts/eval-smell/) _2026-06-29 · Hamel Husain · Hamel's Blog_ For the past 3 years, AI evals have been my professional focus. 1 The most common objection I hear to evals is “our product is hard to eval”. This objection is a product smell. Artifacts that are hard for you to verify are often hard for users too. In the worst case, users have to redo the work from scratch to verify the output. More importantly, designing your product for ease of verification… ## [The Revenge of the Data Scientist](https://hamel.dev/blog/posts/revenge/) _2026-03-26 · Hamel Husain · Hamel's Blog_ Is the heyday of the data scientist over? The Harvard Business Review once called it “The Sexiest Job of the 21st Century.” 1 In tech, data scientist roles were often among the best paid. 2 The job also demanded an unusual mix of skills: Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician. — JosH100 (… ## [Evals Skills for Coding Agents](https://hamel.dev/blog/posts/evals-skills/) _2026-03-02 · Hamel Husain · Hamel's Blog_ Today, Shreya Shankar and I are publishing evals skills , a set of skills for AI product evals 1 . Eval tools often get in the way. They nudge you toward generic off-the-shelf metrics and fully automated evals before you’ve looked at your data. These skills help you avoid common mistakes we’ve seen helping 50+ companies and teaching students in our AI Evals course . Why skills for evals There are… ## [Ship software peacefully (Sponsored)](https://crawlproof.com/a/8KCdtTvYYsrr) _2026-03-02 · **Sponsored**_ Connect your repo and deploy instantly — Railway handles config, scaling, and monitoring. ## [Why I Stopped Using nbdev](https://hamel.dev/blog/posts/ai-stack/) _2026-01-18 · Hamel Husain · Hamel's Blog_ Programmers love to proclaim they’ve found the best tool. Paul Graham called Lisp his “ secret weapon .” DHH described Ruby as “ a magical glove that just fit my brain perfectly .” Pieter Levels ships million-dollar products with vanilla PHP and jQuery . These declarations aren’t about the languages themselves. They’re about developers finding tools that fit how they think. When the environment… ## [Behind the Taliban’s Iron Curtain](https://thecocoons.substack.com/p/behind-the-talibans-iron-curtain) _2025-11-07 · Stella Liu · The Cocoons by Stella and Amy_ A firsthand look at the real Afghanistan today ## [Yuzheng Sun on AI Evals and the Future of Data Science](https://thecocoons.substack.com/p/yuzheng-sun-on-ai-evals-and-the-future) _2025-10-24 · Stella Liu · The Cocoons by Stella and Amy_ The data scientists who stopped believe in data science? ## [The Missing AI Evals Playbook](https://thecocoons.substack.com/p/the-missing-ai-evals-playbook) _2025-10-09 · Stella Liu · The Cocoons by Stella and Amy_ Join our Lightning Lesson on AI Evals and Analytics next week ## [Selecting The Right AI Evals Tool](https://hamel.dev/blog/posts/eval-tools/) _2025-10-01 · Hamel Husain · Hamel's Blog_ Over the past year, I’ve focused heavily on AI Evals , both in my consulting work and teaching. A question I get constantly is, “What’s the best tool for evals?”. I’ve always resisted answering directly for two reasons. First, people focus too much on tools instead of the process, thinking the tool will be an off-the-shelf solution when it rarely is. Second, the tools change so quickly that… ## [The Planner and the Explorer](https://thecocoons.substack.com/p/the-planner-and-the-explorer) _2025-09-25 · Stella Liu · The Cocoons by Stella and Amy_ Welcome to college! What's your major in the age of AI? ## [Bite-Sized colbert-ai : Building A Maintainer’s Mental Model](https://vishalbakshi.github.io/blog/posts/2025-09-23-bite-sized-colbert/) _2025-09-23 · Vishal Bakshi · Vishal Bakshi's Blog_ Background I recently published a blog post on how I’m excited to start my professional ML career after 8 years of being a data analyst, educator, fast.ai student and community member, and now stanford-futuredata/ColBERT maintainer. In response, I have received messages from folks in the industry wanting to talk about career/project opportunities. As I was preparing for these conversations, I… ## [Exploring My Next Chapter in Machine Learning](https://vishalbakshi.github.io/blog/posts/2025-09-18-career/) _2025-09-18 · Vishal Bakshi · Vishal Bakshi's Blog_ Where I Am Today In 285 days, my current role as a data analyst will come to an end. This is both a closing and an opportunity. I have worked as a data analyst for the last eight years, and it has been an incredibly rewarding experience. Working at all levels of the pipeline (including low-tech and no-tech environments), I’ve navigated constructive technical conversations in the context of complex… ## [Re-evaluating colbert-ai Index Artifacts Between PyTorch Versions with Precision-Based torch.allclose Tolerances](https://vishalbakshi.github.io/blog/posts/2025-09-14-colbert-maintenance-2/) _2025-09-14 · Vishal Bakshi · Vishal Bakshi's Blog_ Background I recently learned that it’s best practice to use different torch.allclose tolerances based on the precision of the floating point value. As a reminder, torch.allclose uses absolute and relative tolerances as follows: ∣input\_i − other\_i∣ ≤ atol + rtol × ∣other\_i∣ bitsandbytes uses the following heuristic: if dtype == torch.float32: atol, rtol = 1e-6 , 1e-5 elif dtype == torch.bfloat16:… ## [Debugging ColBERT Index Differences Between PyTorch 2.7.1 and 2.8.0](https://vishalbakshi.github.io/blog/posts/2025-09-14-colbert-maintenance/) _2025-09-14 · Vishal Bakshi · Vishal Bakshi's Blog_ Background I’ve been redoing my colbert-ai index comparisons between PyTorch versions using bitsandbytes’ torch.allclose tolerances . There are three PyTorch version changes that cause index artifact changes: 2.0.1 to 2.1.0 ( BertModel forward pass outputs diverge for all inputs ), 2.4.1 to 2.5.0 ( certain batch sizes cause BertModel output divergence ), and 2.7.1 to 2.8.0 (detailed in this blog… ## [Crypto Payments, No Custody (Sponsored)](https://crawlproof.com/a/82ZBSJqwyinc) _2025-09-14 · **Sponsored**_ Send crypto to merchant wallets via API with real-time updates and automatic fees. ## [Comparing colbert-ai Artifacts Between PyTorch Versions 2.0.1 and 2.1.0](https://vishalbakshi.github.io/blog/posts/2025-09-13-colbert-maintenance/) _2025-09-13 · Vishal Bakshi · Vishal Bakshi's Blog_ Background I’ve been redoing my colbert-ai index comparisons between PyTorch versions using bitsandbytes’ torch.allclose tolerances . In this blog post I explore colbert-ai index artifact differences between PyTorch versions 2.0.1 and 2.1.0. Comparing Intermediate and Final Index Artifacts Final Index Artifacts Using the more lenient bitsandbytes tolerances, avg\_residual.pt and bucket\_weights.pt… ## [Founder Communication: How Startups Win the Spotlight](https://thecocoons.substack.com/p/founder-communication-how-startups) _2025-09-11 · Stella Liu · The Cocoons by Stella and Amy_ We are Stella and Amy. We share firsthand stories at the crossroads of tech, business, and culture, helping leaders craft actionable cross-cultural strategies. Together, we bridge cultural divides and bring the world a little closer—one step at a time. ## [Batch Size Causes BertModel Forward Pass Divergence Between torch==2.4.1 and torch==2.5.0 for colbert-ai.](https://vishalbakshi.github.io/blog/posts/2025-09-11-colbert-maintenance/) _2025-09-11 · Vishal Bakshi · Vishal Bakshi's Blog_ Background I’ve recently been documenting how PyTorch version changes impact stanford-futuredata/ColBERT ( colbert-ai on PyPI) intermediate and final index artifacts. The index artifact I’ll focus on in this blog post is the very important local\_sample\_embs tensor. This is the sample of token embeddings used to calculate the centroids, which are later on used during search. Instead of loading and… ## [PyTorch .sort Behavior Changes from Version 2.0.1 to 2.1.0](https://vishalbakshi.github.io/blog/posts/2025-09-09-colbert-maintenance/) _2025-09-09 · Vishal Bakshi · Vishal Bakshi's Blog_ Background In this notebook I’m going to explore how (and hopefully why) you can start with different codes.indices but end up with the same ivf and ivf\_lengths when indexing a document collection using colbert-ai . I came across this behavior by accident. I was comparing final and intermediate colbert-ai index artifacts between installs using torch==2.0.1 and torch==2.1.0 and found that even… ## [My Top-5 Blog Posts I’ve Written this Year](https://vishalbakshi.github.io/blog/posts/2025-09-03-top-5-own-blog-posts/) _2025-09-03 · Vishal Bakshi · Vishal Bakshi's Blog_ I recently surpassed my goal of publishing 50 machine learning blog posts in 2025. I shared that on Twitter and got this interesting question from Skylar Payne: Do you have one that is your favorite or one you are most proud of? — Skylar Payne ( @skylar\_b\_payne ) September 3, 2025 I browsed through the blog posts I’ve published this year, and I found it really hard to pick just one because each… ## [PyTorch Version Impact on ColBERT Index Artifacts: 2.7.1 –\> 2.8.0](https://vishalbakshi.github.io/blog/posts/2025-09-02-colbert-maintenance/) _2025-09-02 · Vishal Bakshi · Vishal Bakshi's Blog_ Background In a previous blog post I showed how I traced index artifact differences between colbert-ai installs using torch==1.13.1 (the current pinned version) and torch==2.1.0 (the first version which produces different index artifacts) to a difference in floating point differences in the forward pass of the underlying BertModel . In a subsequent blog post I showed how the index artifact… ## [A Summary of and My Thoughts on the DocWrangler Paper](https://vishalbakshi.github.io/blog/posts/2025-09-01-DocWrangler/) _2025-09-01 · Vishal Bakshi · Vishal Bakshi's Blog_ Background I’ll summarize the main points from the “Steering Semantic Data Processing with DocWrangler” paper by Shreya Shankar et al and share my commentary ( something I’ve been doing more of lately ). This work is inspiring and fascinating. Shreya previewed DocWrangler during the AI Evals course, but reading the paper—especially the user study section—helped me grasp its magnitude. While I lack… ## [The Term “Non-Deterministic” and LLMs](https://vishalbakshi.github.io/blog/posts/2025-08-30-non-deterministic/) _2025-08-30 · Vishal Bakshi · Vishal Bakshi's Blog_ I have recently found myself using the term “non-deterministic” to describe LLM behavior. However, something feels off about using that term and I’m nearly convinced that not only is it (sometimes) incorrect, it is imprecise, as it leaves unexplained a critical charactericistic of LLM behavior that makes LLMs different from deterministic functions. First, defining “deterministic algorithm”… ## [From Chicago's South Side to Shenzhen, China](https://thecocoons.substack.com/p/from-chicagos-south-side-to-shenzhen) _2025-08-28 · Stella Liu · The Cocoons by Stella and Amy_ An American Engineer's Journey Across Borders ## [PyTorch Version Impact on ColBERT Index Artifacts: 2.4.1 –\> 2.5.0](https://vishalbakshi.github.io/blog/posts/2025-08-26-colbert-maintenance/) _2025-08-26 · Vishal Bakshi · Vishal Bakshi's Blog_ Background In a previous blog post I outlined two things: Which two subsequent PyTorch versions caused a divergence in stanford-futuredata/ColBERT index .pt artifacts (ConditionalQA document collection): Version A Version B All .pt Shapes Match? (Matches) All .pt Values Match? (Matches) 1.13.1 2.0.0 Yes (10/10) Yes (10/10) 2.0.0 2.0.1 Yes (10/10) Yes (10/10) 2.0.1 2.1.0 No (9/10) No (0/10) 2.1.0… ## [CLI-first decentralized compute (Sponsored)](https://crawlproof.com/a/DGIO82u5FvQr) _2025-08-26 · **Sponsored**_ Run and pay spare GPUs from the CLI — transcode, coinpay (DID+escrow), infernet (AI). ## [PyTorch Version Impact on ColBERT Index Artifacts](https://vishalbakshi.github.io/blog/posts/2025-08-18-colbert-maintenance/) _2025-08-19 · Vishal Bakshi · Vishal Bakshi's Blog_ Background I recently released colbert-ai==0.2.22 which removed the deprecated transformers.AdamW import among other changes . I’m now turning my attention to upgrading the PyTorch dependency to 2.x, which will not only introduce compatibility with modern version installations of torch but will also allow the integration of the AnswerAI fastkmeans library as a replacement for the faiss-gpu and… ## [TIL: Launching Jupyter with a Custom Modal Image and Volume](https://vishalbakshi.github.io/blog/posts/2025-08-17-modal-jupyter/) _2025-08-17 · Vishal Bakshi · Vishal Bakshi's Blog_ Yesterday I learned of the Modal docs example showing how to start a jupyter server via a Modal tunnel . I was elated to see this because it solved my problem of not being able to specify a custom image when using modal launch jupyter . I have a Dockerfile which installs colbert-ai from the main branch of the stanford-futuredata/ColBERT repo with a specific PyTorch and Transformers version: FROM… ## [Reflections After Completing the AI Evals Course](https://vishalbakshi.github.io/blog/posts/2025-08-17-ai-evals/) _2025-08-17 · Vishal Bakshi · Vishal Bakshi's Blog_ Background This blog post is part journal, part reflection, and part planning around the topics of AI engineering, AI evals, applied AI, and my career path into machine learning. When I decided to take Hamel and Shreya’s AI evals course , I had recently watched a very short talk on LLM reliability by Featherless AI CEO Eugene Cheah. I had also learned about the realevals.xyz benchmark , which… ## [China 2000-2025: A Cultural Consultant's Front-Row View](https://thecocoons.substack.com/p/china-2000-2025-a-cultural-consultants) _2025-08-07 · Stella Liu · The Cocoons by Stella and Amy_ Shaping Global Leaders Through Cultural Intelligence ## [Teaching Independence Day in the Age of AI](https://thecocoons.substack.com/p/teaching-independence-day-in-the) _2025-07-24 · Stella Liu · The Cocoons by Stella and Amy_ This summer, one of the highlights of my trip back to Taiwan was volunteering as an English teacher at a local elementary school’s summer camp. Since our class fell on July 4th, we decided to introduce the Taiwanese kids to a slice of American history—and the meaning behind Independence Day. ## [Farmers, Tech and AI](https://thecocoons.substack.com/p/farmers-tech-and-ai) _2025-07-10 · Stella Liu · The Cocoons by Stella and Amy_ A Conversation on AI during a flight ## [In an Age of Information, Why Are Our Youth Still Lost?](https://thecocoons.substack.com/p/in-an-age-of-information-why-are) _2025-06-26 · Stella Liu · The Cocoons by Stella and Amy_ We are Stella and Amy. We share firsthand stories at the crossroads of tech, business, and culture, helping leaders craft actionable cross-cultural strategies. Together, we bridge cultural divides and bring the world a little closer—one step at a time. ## [LLM Evals: Everything You Need to Know](https://hamel.dev/blog/posts/evals-faq/) _2025-05-28 · Hamel Husain · Hamel's Blog_ This document curates the most common questions Shreya and I received while teaching 700+ engineers & PMs AI Evals. Warning: These are sharp opinions about what works in most cases. They are not universal truths. Use your judgment. For a guided path through the rest of our evals work, use the AI evals topic hub . 👉 Want to learn more about AI Evals? Check out our AI Evals course . It’s a live… ## [A Field Guide to Rapidly Improving AI Products](https://hamel.dev/blog/posts/field-guide/) _2025-03-24 · Hamel Husain · Hamel's Blog_ Most AI teams focus on the wrong things. Here’s a common scene from my consulting work: AI TEAM Here’s our agent architecture – we’ve got RAG here, a router there, and we’re using this new framework for… ME \[Holding up my hand to pause the enthusiastic tech lead.\] “Can you show me how you’re measuring if any of this actually works?” … Room goes quiet This scene has played out dozens of times over… ## [Building an Audience Through Technical Writing: Strategies and Mistakes](https://hamel.dev/blog/posts/audience/) _2024-11-30 · Hamel Husain · Hamel's Blog_ People often find me through my writing on AI and tech. This creates an interesting pattern. Nearly every week, vendors reach out asking me to write about their products. While I appreciate their interest and love learning about new tools, I reserve my writing for topics that I have personal experience with. One conversation last week really stuck with me. A founder confided, “We can write the… ## [Using LLM-as-a-Judge For Evaluation: A Complete Guide](https://hamel.dev/blog/posts/llm-judge/) _2024-10-29 · Hamel Husain · Hamel's Blog_ Earlier this year, I wrote Your AI product needs evals . Many of you asked, “How do I get started with LLM-as-a-judge?” This guide shares what I’ve learned after helping over 30 companies set up their evaluation systems. The Problem: AI Teams Are Drowning in Data Ever spend weeks building an AI system, only to realize you have no idea if it’s actually working? You’re not alone. I’ve noticed teams…