Before You Upgrade Embeddings, Check Your Query Logs

Note: AI Weekender has moved. New posts are published at ai-weekender.com, and this Substack is now an archive.

To keep receiving weekly issues, please subscribe at ai-weekender.com instead of here.

I presented my RAG system at PyCon Austria in mid-April to a room of engineers. The audience asked great questions like:

“How do you deal with vague questions?”
“What kinds of queries does your AI not do well at?”
“How do you generate synthetic questions to test it?”

These questions all point to the same failure point in many RAG systems: what happens when the input query has low signal?

As engineers, we often assume every technical problem should be solved with more technical sophistication.

That usually means a stronger embeddings model, reranking, or deeper retriever tuning.

When retrieval fails in a RAG system, the instinct is always the same:

• Add reranking

• Optimize the retriever

• Swap to a better embedding model

• Experiment with different similarity metrics

• Maybe even fine-tuning on domain-specific data

But when I looked closely at my own retrieval failures, I saw a different pattern. The worst queries were vague, incomplete, or missing constraints.

I started auditing retrieval failures by query and looked at what users actually asked.

Queries that failed:

• “What is AI” (too vague, 4 words)

• “How do i learn” (incomplete, 4 words)

• “Python” (single word)

Queries that worked:

• “How do i learn python from scratch without a cs degree” (14 words, specific constraint)

• “What’s the difference between machine learning and deep learning” (10 words, clear comparison)

• “How did you transition from analyst to data scientist” (9 words, concrete problem)

The failed queries were not failing because embeddings were weak. They were failing because there was not enough signal in the query itself.

So I added one line to my system prompt:

”If the user’s query is too vague, ask them a clarifying question before searching.”

I made users ask better questions. If I wanted to extend this further, I could add in a step to use LLMs (or SLMs to save on costs) for query augmentation.

I’m basing this on direct query-log audits and repeated failure patterns in production use.

There are two ways to improve retrieval:

1. System side: Better embeddings, reranking, hybrid search, fusing multiple retrievers

2. Input side: Better queries, clearer questions, explicit context, and clarifying turns before retrieval

Engineers default to #1 because it feels measurable and tunable.

But if your input is vague, a better embedding model often just retrieves irrelevant text faster.

I am not running query augmentation in production yet, so treat this as a design note. If I test it, I would compare:

1. Clarify-first: ask one follow-up when the query is vague.

2. Rewrite-first: use an LLM or SLM to expand the query before retrieval.

My current hypothesis is that clarify-first would preserve user intent better, whereas reewrite-first is faster, but riskier if the model guesses wrong.

If I implement this later, I would evaluate three things: retrieval relevance, latency, and extra costs.

I’ve spent the last few months writing about different pieces of RAG. Here’s the order I’d recommend reading in:

1. Build a working baseline: How to build a RAG system that reads your own writing (paid)

2. See the whole system clearly: Visualizing how the full pipeline works, step by step (free)

3. Clean your context inputs: How to turn messy data into clean context (free)

4. Run a quick evaluation loop: How to measure retrieval performance in 5 minutes (paid)

5. Use metrics to prioritize fixes: Why you can’t improve what you don’t measure (free)

This is the exact sequence I use when a RAG answer looks fluent but wrong.

If you’re debugging RAG, what’s the bigger bottleneck for you?

Note: AI Weekender has moved. New posts are published at ai-weekender.com, and this Substack is now an archive.

To keep receiving weekly issues, please subscribe at ai-weekender.com instead of here.