Evaluating LLM Chat Agents with Real World Signals
How human follow-up behavior reveals response quality, and why specific instruction adherence outperforms vague relevance scoring.
Cole Hoffer - AI Engineering Blog
Making AI do the thing.
Latest posts
Mining Reranker Training Data from RAG Citations
Using citation behavior in production RAG systems to generate labeled training data for domain-specific reranking models.
Relevancy Filtering in Retrieval Pipelines
Using LLM-based classification as a second pass to filter retrieval candidates when similarity thresholds fail to generalize.
Applying HyDE to Domain-Specific Search
Using Hypothetical Document Embeddings to bridge vocabulary gaps between user queries and specialized document corpora.
Structured Pre-Filtering for RAG Systems
Why decomposing queries into structured filters before semantic search improves retrieval precision and performance.
Structured Logprobs for Better Classification
Combining chain-of-thought reasoning with logprob extraction improves LLM classification accuracy while giving you real confidence scores.