Welcome to the next She Writes AI interview episode covering our second book collaboration stream for the AI Everywhere series. In this new cohort, we’re exploring the personal journeys and professional expertise of 32 authors from 18 countries worldwide who are redefining our relationships with machine intelligence.
Today I (Farida Khalaf) am joined by Dr. Elisa Terumi, a distinguished researcher in Artificial Intelligence and a PhD in Computer Science whose work is at the forefront of training and evaluating Large Language Models (LLMs) for real-world impact. As a LinkedIn Top Voice and the founder of Mulheres Programando (”Women Coding Brazil”), Elisa is not only a technical expert but a vital advocate for increasing the visibility of women in the technology sector. She emphasizes that the true power of AI goes beyond the static knowledge encoded during training and depends on connecting language models to reliable external knowledge. She explains how connecting language models to reliable external knowledge can improve the reliability of AI systems.
Elisa brings a rigorous engineering perspective to the common problem of AI hallucinations and the “knowledge cutoff” that limits even the most advanced models like GPT and Gemini. She challenges developers and leaders to move beyond the excitement of generative chat and instead focus on the underlying “retrieval pipeline”. In this chapter, she provides a technical and strategic roadmap for building systems that don’t just “guess” based on probability, but instead “know” by grounding every response in verifiable, external evidence
In this interview, Dr. Elisa Terumi discusses Retrieval-Augmented Generation (RAG) and its role in building more grounded AI applications, where organizations can move from generic language understanding to domain-specific intelligence. She breaks down the shift from treating an LLM as a standalone brain to treating it as a reasoning engine that must be carefully fed with high-quality, retrieved context. Elisa highlights the hidden risks of “retrieval quality”, the “garbage in, garbage out” reality where even the best model cannot compensate for irrelevant or poorly indexed source material.
She explains a practical architecture for building these systems, focusing on embeddings, vector databases, and hybrid search strategies as the technical requirements for enterprise AI systems. Whether you are a software engineer, a data scientist, or a leader in a high-stakes field like healthcare or finance, Elisa’s work demonstrates how to move from “plausible” outputs to grounded, transparent results. Her insights highlight techniques that can help improve grounding, transparency, and factual consistency in AI applications.
Want to read a preview of Elisa’s chapter? Volunteer as an ARC reader!
Below is Dr. Elisa Terumi’s full interview response to Farida Khalaf’s questions on her chapter: RAG Systems: Bridging Language Models and External Knowledge.
Intended Audience: Software engineers, data scientists, and AI practitioners; researchers and students in NLP, information retrieval, and machine learning who want to build more reliable, grounded AI applications.
The “Static” Knowledge Barrier: You argue that Large Language Models (LLMs) rely on information that is fixed at training time, creating a “knowledge cutoff.” How does Retrieval-Augmented Generation (RAG) fundamentally shift this paradigm from static to dynamic knowledge integration?
Elisa: RAG doesn’t change the model itself; it changes when and how knowledge is introduced into the generation process. Instead of relying solely on knowledge encoded in the model’s parameters, the system retrieves relevant information from external sources at inference time and injects it into the model’s context before generation.
This decouples “what the model knows” (its parameters) from “what the system can answer” (its access to external knowledge). As a result, AI systems become more accurate, adaptable, and capable of answering questions about recent events or proprietary data without expensive retraining.
Mitigating Hallucinations: You state that purely generative models often produce “confident but incorrect statements.” How does grounding a response in retrieved evidence from an external source specifically reduce the likelihood of these hallucinations?
Elisa: Hallucinations often occur when a language model lacks sufficient information and fills the gaps using statistical patterns learned during training.
RAG reduces this problem by grounding the generation process in relevant evidence retrieved from trusted external sources. Instead of relying solely on its internal knowledge, the model synthesizes information from the retrieved documents, making it less likely to fabricate unsupported facts.
While grounding does not eliminate hallucinations entirely (since the model can still misinterpret or overgeneralize retrieved content), it improves factual accuracy, transparency, and traceability because responses can often be linked back to their supporting sources.
“Retrieval-Augmented Generation represents a shift from static to dynamic knowledge integration in language models.” - Elisa Terumi
The “Garbage In, Garbage Out” Principle: You emphasize that the performance of a RAG system is constrained by its retrieval step. What are the most common preprocessing or chunking mistakes that lead to the retrieval of contextually irrelevant information?
Elisa: The effectiveness of a RAG system is often limited by the quality of its retrieval pipeline. One of the most common mistakes is splitting documents without preserving semantic coherence. Chunks that are too small lose important context, while chunks that are too large mix multiple topics, reducing retrieval precision because a single embedding represents heterogeneous content.
Another frequent issue is using no overlap between adjacent chunks, which can break information that spans chunk boundaries. Poor preprocessing (such as leaving boilerplate text, duplicated content, headers, footers, or navigation elements) can also contaminate embeddings and hurt retrieval quality.
Also, ignoring metadata (e.g., document source, date, or section) makes filtering and ranking retrieved documents much less effective. Retrieval quality depends as much on preprocessing and chunking strategy as it does on the embedding model itself.
Semantic Search vs. Keywords: RAG systems typically utilize semantic search within a high-dimensional vector space. Why is this approach superior to traditional keyword matching when dealing with complex user queries like “I forgot my login credentials”?
Elisa: Semantic search understands meaning rather than exact wording. A user asking “I forgot my login credentials” and documentation containing only “reset your password” express the same intent, even though they use different wording.
Embedding-based retrieval maps semantically similar concepts close together in vector space, allowing the system to retrieve relevant information despite vocabulary differences. This allows RAG systems to retrieve relevant information based on meaning rather than string matching, making them much more robust to synonyms, paraphrases, and natural language queries.
The Re-indexing Dilemma: You warn that if an embedding model is changed, the entire vector database must be re-indexed, which is a costly and time-consuming process. What criteria should a team use to select the “right” embedding model from the start?
Elisa: Since replacing an embedding model requires re-indexing the entire vector database, the initial choice should be based on careful evaluation rather than popularity.
Teams should benchmark candidate models using representative queries from their own domain, not just generic benchmarks. They should also consider retrieval quality, multilingual support, latency, cost, and deployment constraints.
A model that performs consistently for the target application is usually a better choice than simply the newest or largest one. Investing in this evaluation upfront avoids the costly process of re-indexing the entire vector database later.
Managing Finite Context Windows: Despite modern models supporting larger context windows, they remain finite. What are the specific trade-offs a developer faces when deciding how many documents (the “top-k”) to include in a prompt versus using summarization or compression?
Elisa: Choosing the right top-k is a trade-off between recall and efficiency. Retrieving more documents increases the chance of including the right evidence, but also may add noise, latency, cost, and may exceed the model’s context window. Retrieving fewer documents keeps prompts concise but risks missing critical information. Techniques such as summarization, compression, deduplication, and hierarchical retrieval help maximize useful information while minimizing token usage. There is no universal value for k; it depends on the application and document characteristics.
"The performance of a RAG system is fundamentally constrained by the quality of its retrieval step... Even the most advanced language models cannot compensate for irrelevant, incomplete, or misleading retrieved content.” - Elisa Terumi
The Power of Hybrid Retrieval: You advocate for combining semantic search with keyword-based methods. In what specific scenarios, such as identifying names, codes, or technical terms, is a purely semantic approach likely to fail?
Elisa: Semantic retrieval may struggle with exact matches such as proper names, product codes, error codes, legal case numbers, software versions, or technical acronyms. For example, an embedding may place “HTTP 403” and “HTTP 404” close together because they are semantically related, even though they represent different conditions. Keyword search ensures exact matching, while semantic search captures conceptual similarity. In practice, combining both methods often yields more robust retrieval, especially in technical, legal, and enterprise applications.
Designing for Honesty: You suggest that prompts should explicitly instruct a model to say “I don’t know” if the retrieved information is insufficient. Why is this “factual consistency” more valuable in an enterprise setting than the model’s inherent creativity?
Elisa: In enterprise applications, the cost of a confident but incorrect answer is often much higher than the cost of saying “I don’t know”. A confident but incorrect answer in healthcare, finance, legal services, or customer support can have serious legal, financial, or safety consequences. Explicitly instructing the model to acknowledge uncertainty when the retrieved evidence is insufficient prioritizes factual consistency over unsupported speculation. Users are more likely to trust a system that admits its limitations than one that confidently invents an answer.
The Failure of Traditional Metrics: You note that standard metrics like BLEU and ROUGE often fail to capture factual correctness in RAG systems. Why are specialized frameworks like RAGAS necessary to assess dimensions like “faithfulness” and “context utilization”?
Elisa: Traditional metrics such as BLEU and ROUGE measure textual similarity, not factual grounding. A RAG system can generate a fluent answer that matches a reference while still contradicting the retrieved evidence. Frameworks such as RAGAS evaluate additional dimensions (including faithfulness, answer relevance, and retrieval quality) to determine whether the response is actually grounded in the retrieved context. This provides a much more meaningful assessment of RAG systems than surface-level text overlap.
The Shift to Domain-Specific Intelligence: Beyond simple chatbots, you see RAG as a way for LLMs to move toward context-aware intelligence. How can large organizations use this to manage knowledge that is currently “distributed across multiple systems and formats”?
Elisa: Large organizations often have valuable knowledge scattered across wikis, PDFs, emails, databases, support tickets, and legacy systems. RAG enables these heterogeneous sources to be indexed into a unified retrieval layer without requiring data migration or system replacement.
By enriching documents with metadata and using re-ranking to prioritize the most reliable sources, organizations can provide synthesized, traceable answers grounded in their existing knowledge. This transforms fragmented information into a context-aware assistant that improves knowledge access and decision-making.
A Word from Dr. Elisa Terumi
Retrieval-Augmented Generation is not the final destination; it is an important milestone toward AI systems that reason over reliable, dynamic knowledge rather than static parameters. As the field evolves, approaches such as GraphRAG and agentic RAG are pushing these capabilities even further, enabling systems to connect information across multiple sources and decide when, where, and how to retrieve evidence. Understanding these emerging architectures is, in my view, one of the most exciting directions for anyone building AI applications today.If you’d like to explore these topics further, I regularly share practical tutorials, research highlights, and discussions about LLMs, RAG, Agents, and AI engineering in my newsletters:
In English:
In Portuguese:
“RAG is more than a technical pattern; it is a paradigm shift toward building AI systems that are not only capable of generating language, but also grounded in knowledge.“ - Elisa Terumi
Dr. Elisa Terumi’s perspective emphasizes:
Overcoming the Knowledge Cutoff: RAG allows models to access recent developments and domain-specific information without the high cost of model retraining.
Factual Consistency: By grounding answers in specific source documents, RAG reduces hallucinations and increases transparency, allowing users to trace the AI’s “logic”.
Enterprise Utility: RAG is a scalable solution for organizations to query their own proprietary data, technical manuals, reports, and clinical records, within a secure, context-aware environment.
Trust and Reliability: In fields like healthcare, finance, and law, the ability to prioritize “right” answers over “plausible” ones is critical for safety and compliance.
Optimizing Context: Effective RAG design involves more than just “more data”; it requires sophisticated chunking, hybrid retrieval, and re-ranking to balance completeness and conciseness
Thank You, Dr. Elisa Terumi, for this incredibly technical and clear blueprint for building the next generation of grounded AI applications. For every developer and researcher trying to bridge the gap between “impressive demos” and “reliable products,” her chapter in AI Everywhere is the definitive guide to Retrieval-Augmented Generation.
Elisa reminds us that building a robust system isn’t just about the generative power of the LLM, but about the discipline of the retrieval pipeline. Her final message is a call to engineering rigor: don’t settle for a model that “guesses”; build a system that “knows” by grounding it in trustworthy, external data. I personally highly recommend following her work on Substack, “Exploring Artificial Intelligence,” and supporting her mission with Mulheres Programando, where she is shaping a more inclusive and technically sound future
And if you’re interested in more conversations like this, consider subscribing to She Writes AI. I’ll continue sharing insights from the voices shaping the future of AI, including contributors from 30 new chapters in AI Everywhere where 32 women from across five continents explore how artificial intelligence is changing the world we live in, and how we can change it for the better.
For more insights on the book, see aiEverywhereBooks.com - our beautiful book series website created by Blessing Okpala, PhD (now updated with previews for book2):
Buy Volume 1 (now in ebook and print on Amazon, Gumroad, Lulu, Ingram Spark, and many reading platforms)
Tips from our publisher on how to buy an Amazon ebook from a country that isn’t a ‘KDP marketplace’, and how to read the ebook without a Kindle device (free)
Look for our new abridged audiobook edition!
Review links (we’re on Amazon, Goodreads, and Storygraph)
To help cover book-related costs (ISBNs, proof copies, etc.) as well as other SheWritesAI community services, we follow the Wikipedia model. We are now accepting donations via paid subscriptions and can also now accept one-time tips through Stripe (‘digital ink’). All donation sizes are welcome and help keep these book authoring opportunities free for She Writes AI Community members worldwide! Patrons and Sponsors are acknowledged on our book site. Read more.
Thanks as always for your support in every form. We appreciate it all. Hearts, shares, restacks, and comments are awesome!
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.