Reading · Abonia Sojasingarayar

In the rapidly evolving landscape of artificial intelligence, the integration of Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs) has opened up new horizons for real-world applications. This chapter delves into the practical use of RAG and LLMs across a wide array of domains, showcasing how these technologies are transforming industries and solving complex problems. By exploring case studies in various domains such as conversational AI, biomedical document understanding, and legal search, we aim to provide a comprehensive overview of how RAG and LLMs are being applied in real-world settings. Additionally, we will examine how these models are utilized to solve real-world problems, including open-domain question answering, long-form text generation, and multi-step reasoning. This exploration will not only highlight the potential of RAG and LLMs but also offer insights into the challenges and future directions of these technologies.

The chapter is structured to provide a detailed look at the applications of RAG and LLMs. It begins with a deep dive into case studies across different domains, providing you with a practical understanding of how these models are being deployed. This is followed by a section on solving real-world problems with RAG, offering examples of open-domain question answering, long-form text generation, and multi-step reasoning. To ensure a broad perspective, we also explore the landscape of vector-capable solutions, including approximate nearest neighbor libraries, vector databases, and cloud offerings. Finally, we delve into specific industries to see how RAG and LLMs are being utilized, providing a comprehensive understanding of their applications and impact.

In this chapter, we will cover the following topics :

Case studies in various domains:
Solving real-world problems with RAG
Landscape of vector-capable solutions
How RAG and LLMs are used in specific industries
Find source code used in this chapter
LLM Engineering Book

In our earlier chapter, we explored the functionality of the RAG pipeline. If you haven't already, we highly suggest revisiting the preceding chapters before delving into this one, as it primarily focuses on the demonstration of implementation details. Below, we provide a simple illustration of how RAG operates in a question-answer format.

This is the most basic form of RAG. It involves three steps:

Retriever: Searches a large corpus of text (like Wikipedia) for passages relevant to a given prompt or question.
Generator: Uses the retrieved passages along with the prompt to generate a response using an LLM.
(Optional) Reranker: In some cases, a reranking step might be included where the retrieved passages are scored and re-ordered based on their relevance to the prompt.
NaiveRAG has limitations. The retrieved passages might not always be the most relevant, and the LLM might struggle to integrate them effectively.

This builds upon NaiveRAG by addressing its limitations. It can involve various techniques like:

Improved Retrieval Strategies: Optimizing how passages are searched and retrieved for better relevance.

Fine-tuning the Retriever: Adapting the retrieval process based on specific tasks or domains.

Advanced Prompt Engineering: Creating more specific prompts for the LLM to leverage the retrieved information effectively.

This is the most flexible approach. It breaks down the RAG process into independent modules:

Search Module: Responsible for retrieving relevant passages.
Memory Module: Manages the retrieved information.
Fusion Module: Combines the retrieved information with the prompt.
Routing Module: Decides which information to use based on the task.
Predict Module: Generates the final output using the LLM.
Task Adapter Module: Adapts the entire process for specific tasks.

ModularRAG allows for more customization and control over each step in the reasoning process. Both NaiveRAG and AdvancedRAG can be seen as special cases of ModularRAG with a fixed set of modules.

Frameworks supporting RAG (Retrieval-Augmented Generation) and LLM (Large Language Models) application development include LangChain, LlamaIndex, Haystack, TinyLLM, Griptape,Embedchain and more.

LangChain is an open-source framework designed to simplify the development of applications powered by large language models. It provides a comprehensive toolkit for building more complex and interactive LLM
applications, going beyond basic search and retrieval. LangChain's components include chains, which allow the chaining of components together, facilitating the use of PromptTemplates and LLMChains for interactive applications.
LlamaIndex is a plug-and-play solution for search-centric applications, focusing on providing quick access to specific information within large datasets. It is more of a specialized framework compared to LangChain, which offers a broader range of applications requiring deeper customization.
Haystack emerges as a comprehensive NLP framework, empowering developers to craft applications infused with cutting-edge NLP models and LLMs. With a diverse range of capabilities spanning question answering, answer generation, and semantic document search, Haystack heralds a new era of NLP-driven application development. Core concepts such as Pipelines and Nodes structure and process data, while Agents, powered by LLMs, navigate complex queries. Specialized tools augment agent capabilities, exemplified by calculators or WebRetrievers, while DocumentStores provide compatibility with various database technologies. Delve into the vast potential of NLP frameworks with Haystack's robust features and functionalities.

Creating a conversational AI chatbot tailored to specific data needs involves several steps, from processing PDF documents to integrating a large language model (LLM) for generating responses. In this section we will walk you through the process, using HuggingFace Embeddings, FAISS for vector storage, and Ollama mistral model. The langchain library is instrumental in managing conversation chains, indexing data, and crafting prompt templates. By the end of this tutorial, you'll have built a RAG system with a conversational UI, capable of detecting hallucinations in the LLM's responses. Before we start we can install Ollma in our local machine for inference as follow:

1.Download Ollama

For linux users-To install Ollma on Linux, you can use the following command in your terminal. This command downloads and executes the installation script directly from the Ollma official site:

curl -fsSL https://ollama.com/install.sh | sh

For Windows users- Windows users can download Ollma by visiting the official Ollma website and following the link to the Windows download page:https://ollama.com/download/windows

For mac users-https://ollama.com/download/mac

2. Pulling the Model

After installing Ollma, you can pull the model of your choice using the following command:

ollama pull mistral

This command downloads the specified model (in this case, "mistral") to your local machine, making it available for inference.

First, install necessary packages

!pip install pypdf langchain langchain-community tiktoken llama-cpp-python panel streamlit

Then, you need to process PDF documents to extract text and metadata. This step involves reading each page of the PDFs, extracting the text, and organizing it into a structured format for indexing.

import PyPDF2
def prepare_docs(pdf_docs):
    docs = []
    metadata = []
    content = []
    for pdf in pdf_docs:
        pdf_reader = PyPDF2.PdfReader(pdf)
        for index, page in enumerate(pdf_reader.pages):
            doc_page = {'title': pdf + " page " + str(index + 1),
                        'content': page.extract_text()}
            docs.append(doc_page)
    for doc in docs:
        content.append(doc["content"])
        metadata.append({"title": doc["title"]})
    print("Content and metadata are extracted from the documents")
    return content, metadata

Next, chunk the extracted content into smaller segments for easier processing and retrieval. This step uses the `RecursiveCharacterTextSplitter` from langchain to split the content based on a specified chunk size and overlap.

from langchain.splitter import RecursiveCharacterTextSplitter
def get_text_chunks(content, metadata):
    text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
        chunk_size=512,
        chunk_overlap=256,
    )
    split_docs = text_splitter.create_documents(content, metadatas=metadata)
    print(f"Documents are split into {len(split_docs)} passages")
    return split_docs
Index the chunked documents into a FAISS-based vector database for efficient similarity search. This step uses HuggingFace Embeddings to generate embeddings for the documents, which are then stored in a FAISS database.
from langchain.vectorstore import HuggingFaceEmbeddings, FAISS
def ingest_into_vectordb(split_docs):
    embeddings = HuggingFaceEmbeddings(model_name='sentence-transformers/all-MiniLM-L6-v2', model_kwargs={'device': 'cpu'})
    db = FAISS.from_documents(split_docs, embeddings)
    DB_FAISS_PATH = 'vectorstore/db_faiss'
    db.save_local(DB_FAISS_PATH)
    return db

Configure a conversational chain for the Ollama model, integrating it with the vector database for information retrieval. This setup enhances the conversational experience by combining language generation with memory and retrieval functionalities.

To use another model replace the model name in Ollama(model="llama").

from langchain.memory import ConversationBufferMemory
from langchain.retrievalchain import ConversationalRetrievalChain
def get_conversation_chain(vectordb):
    llm = Ollama(model="mistral")
    retriever = vectordb.as_retriever()
    memory = ConversationBufferMemory(
        memory_key='chat_history', return_messages=True, output_key='answer'
    )
    conversation_chain = ConversationalRetrievalChain.from_llm(
        llm=llm,
        retriever=retriever,
        memory=memory,
        return_source_documents=True
    )
    print("Conversational Chain created for the LLM using the vector store")
    return conversation_chain

Run below cell to prepare,chunk and vectorize the data.In below example I tested with my CV.Do not hesitate to use any pdf that you wants to work with.

pdf_docs=["./data/CV.pdf"]
content, metadata = prepare_docs(pdf_docs)
split_docs = get_text_chunks(content, metadata)
vectordb=ingest_into_vectordb(split_docs)
Now , ask your Question.We created a conversational chain and now ready to chat with your own data. 
### Question 1
user_question = "who is Abonia Sojasingarayar?"
response=conversation_chain({"question": user_question})
print("Q: ",user_question)
print("A: ",response['answer'])

Output:

Q:  who is Abonia Sojasingarayar?
A:   Abonia Sojasingarayar is a Machine Learning Scientist, Data Scientist, NLP Engineer, Computer Vision Engineer, AI Analyst, and Technical Writer. They have education from the Université Pondicherry in India, IA School in Boulogne-Billancourt, France, and Institut F2I in Paris, France. Abonia has certifications from IBM and deeplearning.IA, and they are proficient in various tools and techniques related to their field such as Python, TensorFlow, GCP professional data engineer Badges, Watson Assistant, and RPA (Robotic Process Automation) among others. They have worked on projects involving API integration, machine learning pipeline development, and research engineering.
### Question 2 
user_question = "where did she graduated?"
response=conversation_chain({"question": user_question})
print("Q: ",user_question)
print("A: ",response['answer'])
print("\nConversation Chain: \n",response)
Output:
Q:  where did she graduated?
A:   Abonia Sojasingarayar graduated from the Université Pondicherry in India with a licence en technologie informatique et Ingénierie degree.

Conversation Chain:

{'question': 'where did she graduated?', 'chat_history': [HumanMessage(content='who is Abonia Sojasingarayar?'), AIMessage(content=' Abonia Sojasingarayar is a Machine Learning Scientist, Data Scientist, NLP Engineer, Computer Vision Engineer, AI Analyst, and Technical Writer. They have education from the Université Pondicherry in India…'), HumanMessage(content='where did she graduated?'), AIMessage(content=' Abonia Sojasingarayar graduated from the Université Pondicherry in India with a licence en technologie informatique et Ingénierie degree.')], 'answer': ' Abonia Sojasingarayar graduated from the Université Pondicherry in India with a licence en technologie informatique et Ingénierie degree.', 'source_documents': [Document(page_content='Abonia Sojasingarayar \n \n \n \nMachine Learning Scientist | Data Scientist | NLP Engineer | Computer Vision Engineer | AI \nAnalyst | Technical Writer \n ……", metadata={'title': './data/CV.pdf page 2'})]}

So, if we observe, when I query again without explicitly specifying the name, the model is now capable of recalling previous interactions, thanks to the integration of a memory buffer. This enhancement allows the model to retain information from past conversations, enabling it to provide more contextually relevant responses in subsequent interactions.

Finally, build a user interface for interacting with the chatbot. This UI allows users to ask questions related to their documents, with the application processing these questions, retrieving relevant information, and generating responses.

import streamlit as st
def handle_userinput(user_question):
    response = st.session_state.conversation({'question': user_question})
    st.session_state.chat_history = response['chat_history']
    for i, message in enumerate(st.session_state.chat_history):
        if i % 2 == 0:
            st.write(user_template.replace("{{MSG}}", message.content), unsafe_allow_html=True)
        else:
            st.write(bot_template.replace("{{MSG}}", message.content), unsafe_allow_html=True)
def main():
    st.set_page_config(page_title="Chat with your PDFs", page_icon=":books:")
    st.header("Chat with multiple PDFs :books:")
    user_question = st.text_input("Ask a question about your documents:")
    if user_question:
        handle_userinput(user_question)
    with st.sidebar:
       st.subheader("Your documents")
        pdf_docs = st.file_uploader("Upload your PDFs here and click on 'Process'", accept_multiple_files=True)
        if st.button("Process"):
            content, metadata = prepare_docs(pdf_docs)
            split_docs = get_text_chunks(content, metadata)
            vectorstore = ingest_into_vectordb(split_docs)
            st.session_state.conversation = get_conversation_chain(vectorstore)

Now, with our RAG conversational chatbot, we have the flexibility to upload any PDF document and immediately initiate a conversation.

Biomedical document understanding through Large Language Models (LLMs) involves leveraging advanced AI technologies to process and comprehend vast amounts of medical text, such as research papers, clinical studies, and patient records. This capability is crucial for healthcare providers to stay updated with the latest medical information and make informed decisions. LLMs, specifically designed for healthcare applications, can analyze complex medical texts, extract meaningful information, and generate insights for healthcare professionals. They are distinguished by the databases they were trained on, with clinical LLMs focusing on medical literature for diagnostic support and biomedical LLMs facilitating fast and accessible biomedical text mining.LLMs in healthcare have numerous use cases, including processing extensive databases of medical literature and patient data, learning from historical cases, and providing insights for accurate and timely diagnosis. For example, a LLM can analyze a patient’s symptoms, medical history, and clinical findings to generate a personalized treatment plan, incorporating the latest research findings and treatment guidelines. This enhances patient care and outcomes by enabling healthcare professionals to make more informed decisions.

Pretrained models like BioBERT, ClinicalBERT, BlueBERT, and BioGPT have shown significant advancements in applying AI in the medical field. BioBERT, trained on large-scale biomedical corpora, excels in understanding complex medical texts and terminology, making it effective for tasks like disease prediction and drug-drug interaction analysis. ClinicalBERT, adapted from BioBERT, is fine-tuned on clinical notes for more accurate patient data analysis and decision support. BlueBERT offers a balanced understanding of both biomedical and clinical texts, making it versatile for various applications. BioGPT, a generative pretrained transformer model, is useful for generating coherent medical text.

Med-PaLM, a large-scale generalist biomedical AI system, stands out as a multimodal generative model designed to handle various types of biomedical data, including clinical language, medical imaging, and genomics. It leverages advances in language and multimodal foundation models, allowing for rapid adaptation to different tasks and settings. Med-PaLM achieves remarkable performance on a wide range of tasks within the MultiMedBench benchmark, often surpassing state-of-the-art specialist models. This model demonstrates promising potential for downstream data-scarce biomedical applications and has the ability to process inputs with multiple images during inference, effectively handling complex medical scenarios.

A Legal Summarizer that simplifies complex legal documents, making them easier to understand for various audiences, including lawyers and non-lawyers. It extracts and condenses the key points, legal terms, and essential information from legal documents, such as court cases, legal briefs, and legislation, into concise summaries. This is particularly useful for:

Lawyers: They can quickly grasp the core arguments, legal issues, and outcomes without sifting through lengthy documents. It helps in preparing for legal cases, advising clients, and researching legal topics more efficiently.

Non-lawyers: For individuals who need to understand legal documents for personal, business, or educational reasons but lack expertise in legal terminology. It provides a clear, accessible summary of legal documents, enabling them to make informed decisions or understand legal implications more effectively.

Legal Education and Research: Students and researchers can use Legal Summarizers to grasp complex legal concepts and cases without the need to read through entire volumes of legal texts. This aids in studying law, conducting legal research, and preparing for exams.

Legal Assistants and Support Staff: They can use summaries to provide clients or colleagues with essential information from legal documents, making it easier to convey complex legal issues in a straightforward manner.

To perform document summarization using LLMs with the LangChain library, you have three main options: Stuff, Map-Reduce, and Refine. In the coming section we will see the hands on guide for each method.

This method involves stuffing all your documents into a single prompt and passing it to an LLM.

Import necessary modules and define the prompt template.

Create an LLM chain with the defined prompt.

Define a StuffDocumentsChain that takes the LLM chain and combines all documents into a single prompt.

Run the summarization.

from langchain.chains.combine_documents.stuff import StuffDocumentsChain
from langchain.chains.llm import LLMChain
from langchain.prompts import PromptTemplate
loader = PyPDFLoader("./data/Raptor-Agreement.pdf")
# Define prompt
prompt_template = """Write a concise summary of the following:
"{text}"
CONCISE SUMMARY:"""
prompt = PromptTemplate.from_template(prompt_template)
# Define LLM chain
llm = ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo-16k")
llm_chain = LLMChain(llm=llm, prompt=prompt)
# Define StuffDocumentsChain
stuff_chain = StuffDocumentsChain(llm_chain=llm_chain, document_variable_name="text")
docs = loader.load()
print(stuff_chain.run(docs))

Due to the limitations of summarizing lengthy content, we may not be able to provide the full summary results here. However, you're welcome to clone the GitHub repository associated with this project. Within the repository, you'll find notebooks corresponding to each chapter. By running these notebooks, you'll be able to explore the complete results and gain a deeper understanding of the concepts discussed. Feel free to experiment.

This method involves summarizing each document individually (map) and then combining these summaries into a final summary (reduce).

-Define the map and reduce prompts.

-Create an LLM chain for mapping each document to an individual summary.

-Use a ReduceDocumentsChain to combine the summaries.

-Optionally, use a MapReduceDocumentsChain to automate the process.

from langchain.chains import MapReduceDocumentsChain, ReduceDocumentsChain
from langchain_text_splitters import CharacterTextSplitter
llm = ChatOpenAI(temperature=0)
# Map
map_template = """The following is a set of documents
{docs}
Based on this list of docs, please identify the main themes 
Helpful Answer:"""
map_prompt = PromptTemplate.from_template(map_template)
map_chain = LLMChain(llm=llm, prompt=map_prompt)
# Reduce
reduce_template = """The following is set of summaries:
{docs}
Take these and distill it into a final, consolidated summary of the main themes. 
Helpful Answer:"""
reduce_prompt = PromptTemplate.from_template(reduce_template)
# Run chain
reduce_chain = LLMChain(llm=llm, prompt=reduce_prompt)
combine_documents_chain = StuffDocumentsChain(
    llm_chain=reduce_chain, document_variable_name="docs"
)
reduce_documents_chain = ReduceDocumentsChain(
    combine_documents_chain=combine_documents_chain,
    collapse_documents_chain=combine_documents_chain,
    token_max=4000,
)
map_reduce_chain = MapReduceDocumentsChain(
    llm_chain=map_chain,
    reduce_documents_chain=reduce_documents_chain,
    document_variable_name="docs",
    return_intermediate_steps=False,
)
text_splitter = CharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=1000, chunk_overlap=0
)
split_docs = text_splitter.split_documents(docs)
print(map_reduce_chain.run(split_docs))

Here is the output:
This document outlines the terms and conditions of an Agreement between Raptor Technologies, LLC (Raptor) and a Subscriber organization for access to Raptor's Subscription Services. The key themes include:

1. License and Terms: Raptor grants a limited, non-exclusive license to the Subscriber to use its Subscription Services subject to certain terms and conditions. The Subscriber is responsible for providing their own Internet access and equipment to use the Subscription Services.

2. Confidentiality: The Subscriber agrees to keep confidential any information related to the Subscription Services and Equipment provided by Raptor, except as expressly permitted.

3. Data Collection and Distribution: The Subscriber is prohibited from disclosing or making public individual's personally identifying information obtained through the Subscription Services except as required in the ordinary course of business or by applicable law.

4. Fees and Term: The Agreement has an initial term of one year, during which the Subscriber must pay the Annual Software Access Fee for each Campus that will utilize the Subscription Services. Upon termination, all amounts due to Raptor remain payable and all licenses granted under the Agreement terminate at the end of the pre-paid annual term.

5. Termination: The Subscriber may terminate the Agreement with 60 days' written notice prior to the end of the then-current term. Sections 1, 2, 3, 6, and 7 survive termination.

6. Disclaimers: Raptor does not guarantee or warrant any information made available within the Subscription Services, including determinations of an individual's registered sex offender status or custom alert status. The Subscriber is responsible for ensuring compliance with applicable laws and regulations related to data collection and distribution.

7. Other Provisions: The Agreement includes provisions related to acts beyond Raptor's control, lack of creation of partnership or agency relationship, and non-assignment by the Subscriber without consent. Contact information for written notices and effective date are also included.

This method involves iteratively refining a summary based on new context.

-Define the prompt template for refining.

-Load the summarize chain with the refine chain type.

-Run the summarization with the input documents.

from langchain import load_summarize_chain, PromptTemplate
prompt_template = """Write a concise summary of the following:
"{text}"
CONCISE SUMMARY:"""
prompt = PromptTemplate.from_template(prompt_template)
chain = load_summarize_chain(llm, chain_type="refine")
chain.run(split_docs)

Output:

" This document outlines the terms of a subscription agreement between Subscriber (district/school or organization) and Raptor Technologies LLC (Raptor) for access to Raptor's Subscription Services. The agreement grants Subscriber a limited, non-exclusive license to use the services in accordance with the agreement and applicable laws. Confidential information provided by Raptor must be kept confidential and not disclosed to third parties without prior written consent. Individual's personally identifying information obtained through the services must not be disclosed except as required by law or in the ordinary course of business. Subscriber is responsible for providing its own Internet access and equipment to use the services, and fees are payable annually in advance. The agreement has an initial term of one year, with automatic renewal unless written notice of non-renewal is given.\n\nRaptor disclaims all responsibility for determinations of an individual’s registered sex offender status or custom alert status based on the information conveyed in connection with the Subscription Services. Subscriber is solely responsible for such determinations and understands that information provided by Raptor is not intended to substitute for the determinations made by Subscriber and its employees and contractors.\n\nThe agreement may be amended only pursuant to a written agreement between the Parties. All terms and conditions of this Agreement shall be binding upon, inure to the benefit of, and be enforceable by, the Parties and their respective successors and permitted assigns. Raptor will not be in default of this Agreement for any performance failure caused by occurrences beyond Raptor’s reasonable control (including, but not limited to, acts of God). This Agreement does not create any right enforceable by any person not a party. Nothing in this Agreement shall create the relationship of partners or principal-agent between the parties. Subscriber may not assign this Agreement without the prior written consent of Raptor. The waiver or failure of Raptor to exercise in any respect any right provided for under this Agreement shall not be deemed a waiver of any further right under this Agreement."

Each of these methods has its use cases depending on the specific requirements of your document summarization task. The Stuff method is simpler but may not capture all nuances. Map-Reduce provides a more detailed approach but requires more setup. Refine offers a way to iteratively improve a summary based on additional context.

Retrieval-Augmented Generation (RAG) approach addresses the challenge of generating factually accurate and coherent long-form text for open-domain question answering (QA), long-form text generation, and multi-step reasoning tasks. By retrieving relevant documents and incorporating their information into the model's responses, RAG improves the relevance, factual correctness, and attribution of the generated text, making it particularly useful for domains requiring external knowledge such as science, medicine, and technical support.

Let's dive directly to the demo.LangChain provides multiple built-in document loaders, that work with PDF files, JSON files, or a Python file in your file directory. We can use LangChain’s PyPDFLoader to import your PDF seamlessly. Here we will be using the data directly from the website to ask questionsHere we will be using medium articles.

Install necessary packages

!pip install nest_asyncio langchain_community langchain playwright html2text sentence-transformers faiss-cpu
from langchain_community.document_transformers import Html2TextTransformer
from langchain.text_splitter import CharacterTextSplitter
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import FAISS
import nest_asyncio
nest_asyncio.apply()
# Articles to index
articles = ["https://medium.com/@abonia/bertscore-explained-in-5-minutes-0b98553bfb71",
            "https://medium.com/@abonia/document-based-llm-powered-chatbot-bb316009de93/",]
# Scrapes the blogs above
loader = AsyncChromiumLoader(articles)
docs = loader.load()

When our document is long, it’s necessary to split up our document text into chunks. There are various ways to split your text. Let’s just use the simplest method CharacterTextSplitter to split based on characters and measure chunk length by the number of characters.

# Converts HTML to plain text 
html2text = Html2TextTransformer()
docs_transformed = html2text.transform_documents(docs)
# Chunk text
text_splitter = CharacterTextSplitter(chunk_size=100, 
                                      chunk_overlap=0)
chunked_documents = text_splitter.split_documents(docs_transformed)
# Load chunked documents into the FAISS index
db = FAISS.from_documents(chunked_documents, 
                          HuggingFaceEmbeddings(model_name='sentence-transformers/all-mpnet-base-v2'))
retriever = db.as_retriever()

The text chunks are then translated into numerical vectors through embeddings, allowing us to work with text data like semantic search in a computationally efficient manner. We can choose an embedding model provider like OpenAI, HuggingFaceEmbedding, Jina etc for this task.We then need to store our embedding vectors in a vector store, which allows us to search and retrieve the relevant vectors at query time.

We can expose the vector store in a retriever interface. To retrieve text, we can choose a search type like “similarity” to use similarity search in the retriever object where it selects text chunk vectors that are most similar to the question vector. k=2 lets us find the top 2 most relevant text chunk vectors. A RetrievalQA chain chains a large language model with our retriever interface. You can also define the chain type as one of the four options: “stuff,” “map reduce,” “refine,” “map_rerank.” The default chain_type=”stuff” incorporates ALL text from the documents into the prompt. The “map_reduce” type breaks texts into groups, poses the question to the LLM for each batch separately, and derives the ultimate answer based on the replies from each batch.

The “refine” type partitions texts into batches, presents the first batch to the LLM, and then submits the answer along with the second batch to the LLM. It progressively refines the answer by processing through all the batches. The “map-rerank” type divides texts into batches, submits each one to the LLM, returns a score indicating how comprehensively it answers the question, and determines the final answer based on the highest-scoring replies from each batch.

from langchain import PromptTemplate
from langchain_community.llms import Ollama
from langchain.chains import RetrievalQA
prompt_template = """
### [INST] Instruction: Answer the question based on the medium article knowledge. Here is context to help:
{context}
### QUESTION:
{question} [/INST]
 """
# Create prompt from prompt template
prompt = PromptTemplate(
    input_variables=["context", "question"],
    template=prompt_template,
)
llm = Ollama(model="mistral")
qa = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever, return_source_documents=True, chain_type_kwargs={"prompt": prompt_template},)
answer = qa.invoke("What is cosine similarity?")

Output

{'query': 'What is cosine similarity?',

'result': ' Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space. It is computed as the cosine of the angle between them, which indicates how similar they are in direction. The result ranges from -1 to 1, with 1 indicating perfect similarity and 0 indicating orthogonal (perpendicular) vectors.',

'source_documents': [Document(page_content='The formula for cosine similarity is:\n\n> similarity(A, B) = (A . B) / (||A|| ||B||)', metadata={'source': 'https://medium.com/@abonia/document-based-llm-powered-chatbot-bb316009de93/'}),

Document(page_content='Cosine similarity — This method measures the cosine of the angle between two\nvectors, which indicates how similar they are in direction. Cosine similarity\nranges from -1 to 1, with 1 indicating perfect similarity.', metadata={'source': 'https://medium.com/@abonia/document-based-llm-powered-chatbot-bb316009de93/'}),

Document(page_content='Cosine Similarity and Cosine Distance — credit', metadata={'source': 'https://medium.com/@abonia/document-based-llm-powered-chatbot-bb316009de93/'}),

Document(page_content='Where A and B are the two vectors being compared, . is the dot product of the\nvectors, and || || represents the Euclidean norm (magnitude) of the vectors.', metadata={'source': 'https://medium.com/@abonia/document-based-llm-powered-chatbot-bb316009de93/'})]}

Sample code to build RAG Chain

llm_chain = LLMChain(llm=mistral_llm, prompt=prompt)
rag_chain = ( 
 {"context": retriever, "question": RunnablePassthrough()}
    | llm_chain
)
result = rag_chain.invoke("What is cosine similarity??")
print(result['text'])

To implement multi-step reasoning in a Retrieval-Augmented Generation (RAG) application, we can follow a multi-stage retrieval process that combines different retrieval methods for improved overall quality. Augmentation involves the process of effectively integrating context from retrieved passages with the current generation task. Before discussing more on the augmentation process, augmentation stages, and augmentation data, here is a taxonomy of RAG's core components:

Retrieval augmentation can be applied in many different stages such as pre-training, fine-tuning, and inference.

Augmentation Stages: RETRO is an example of a system that leverages retrieval augmentation for large-scale pre-training from scratch; it uses an additional encoder built on top of external knowledge. Fine-tuning can also be combined with RAG to help develop and improve the effectiveness of RAG systems. At the inference stage, many techniques are applied to effectively incorporate retrieved content to meet specific task demands and further refine the RAG process.

Augmentation Source: A RAG model's effectiveness is heavily impacted by the choice of augmentation data source. Data can be categorized into unstructured, structured, and LLM-generated data.

Augmentation Process: For many problems (e.g., multi-step reasoning), a single retrieval isn't enough so a few methods have been proposed:

Iterative retrieval enables the model to perform multiple retrieval cycles to enhance the depth and relevance of information. Notable approaches that leverage this method include RETRO and GAR-meets-RAG.

Recursive retrieval recursively iterates on the output of one retrieval step as the input to another retrieval step; this enables delving deeper into relevant information for complex and multi-step queries (e.g., academic research and legal case analysis). Notable approaches that leverage this method include IRCoT and Tree of Clarifications.

Adaptive retrieval tailors the retrieval process to specific demands by determining optimal moments and content for retrieval. Notable approaches that leverage this method include FLARE and Self-RAG.

The figure below depicts a detailed representation of RAG research with different augmentation aspects, including the augmentation stages, source, and process.

In the rapidly evolving landscape of technology, the advent of vector-capable solutions has marked a significant shift in how we process, store, and retrieve information. These solutions, rooted in the realm of Generative AI and Large Language Models (LLMs), have transformed the way we interact with data, offering unprecedented capabilities in areas such as computer vision, recommendation systems, and natural language processing tasks. This introduction to the landscape of vector-capable solutions aims to explore the essence of vector databases, their role in augmented generation (RAG), and their potential to revolutionize the future of data management and AI applications.

Vector databases, at the heart of this landscape, are purpose-built to efficiently manage high-dimensional data represented as vectors. These vectors serve as mathematical representations of various data types, such as text, images, and videos, capturing their features and relationships in a way that traditional databases struggle to achieve. By leveraging vector embeddings, vector databases enable sophisticated search and retrieval mechanisms, turning raw data into a format that AI models can comprehend and utilize effectively. This capability is particularly crucial in the era of RAG, where the integration of vector databases plays a pivotal role in enriching LLMs with additional data and context, enhancing their performance and capabilities.

The landscape of vector-capable solutions is not just limited to vector databases. It encompasses a range of tools and technologies designed to support the efficient creation, storage, and querying of vector embeddings. This includes open-source models like Google's 'text2vec' and 'BERT', as well as proprietary models developed by leading AI research institutions. These models are instrumental in generating vector embeddings, which are then stored in vector databases for subsequent retrieval and analysis.Moreover, the landscape is continually evolving, with new solutions emerging to address the growing demands of applications that require high-performance similarity search, real-time querying, and scalable deployment. Services like Pinecone, designed for scalable, high-performance similarity search, and SingleStore, known for its high performance and scalability, represent just a fraction of the innovative offerings in this space. These solutions not only cater to the specific needs of vector-intensive applications but also integrate seamlessly with broader data management and AI infrastructures, setting new standards for efficiency, scalability, and performance.

These libraries play a crucial role in optimizing the search for nearest neighbors in high-dimensional spaces, where traditional methods often fall short due to computational constraints. The importance of ANN libraries cannot be overstated, as they enable efficient similarity searches in scenarios ranging from recommendation systems to image and document search, natural language processing, and fraud detection.

Annoy, developed by Spotify, stands out as a notable ANN library. It is available in both C++ and Python, optimized for memory usage and facilitating the loading and saving of large datasets to disk. Annoy is designed to create large read-only file-based data structures that can be memory-mapped into memory, allowing multiple processes to share the same data efficiently. This makes it particularly suitable for applications that require high performance and scalability, such as Spotify's personalization and recommendation systems.

The ANN Library, created by David M. Mount and Sunil Arya, is another significant contribution to the ANN domain. Written in C++, this library supports both exact and approximate nearest neighbor searching in high dimensions. It implements various data structures based on kd-trees and box-decomposition trees and employs different search strategies. The library is designed to handle datasets ranging in size from thousands to hundreds of thousands points and dimensions up to 20. It allows users to specify a maximum approximation error bound, enabling a trade-off between accuracy and running time.

In addition to Annoy and the ANN Library, Python users have access to FLANN and NMSLIB. FLANN is a versatile library that implements a variety of ANN algorithms, including ball trees, KD trees, and LSH. NMSLIB, on the other hand, offers implementations of different ANN algorithms, including HNSW, catering to a broad range of applications.

Faiss, developed by Facebook AI, is a library designed to provide efficient similarity search and clustering of dense vectors. It supports a wide range of index types, including those based on hierarchical navigable small world (HNSW) graphs, which are particularly effective for high-dimensional data. The HNSW algorithm is known for its speed and memory efficiency, making it an excellent choice for applications requiring real-time vector search capabilities.

The landscape of vector databases is vast and growing, with several notable options available for different use cases. Some of the top vector databases in 2023 and 2024 include:

Pinecone: A fully managed cloud service that simplifies the deployment and scaling of vector search systems.
Milvus: An open-source vector database that supports various data types and integrates with machine learning models for automatic vectorization.
Chroma: A vector database designed for high-performance similarity search and analytics.
Weaviate: An open-source, graph-based vector database that offers both cloud and self-hosted deployment options.
Deep Lake: A vector database focused on deep learning applications.
Qdrant: A vector database that emphasizes high-performance similarity search and analytics.
Elasticsearch: A widely used search and analytics engine that also supports vector search capabilities.
Vespa: A real-time big data processing and serving engine that includes vector search capabilities.
Vald: An open-source vector database designed for high-speed similarity search.
ScaNN: A library developed by Google Research for efficient vector similarity search.
Pgvector: An extension for PostgreSQL that adds support for vector data types and functions.
Faiss: A library developed by Facebook AI Research for efficient similarity search and clustering of dense vectors.
ClickHouse: An open-source column-oriented database management system that supports vector search.
OpenSearch: A community-driven, open-source search and analytics suite that includes vector search capabilities.
Apache Cassandra: A highly scalable, distributed NoSQL database that can be extended to support vector data types

pip install chromadb

We use OpenAIEmbeddings so we have to get the OpenAI API Key.

import os
import getpass
os.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key:')
from langchain_community.document_loaders import TextLoader
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import CharacterTextSplitter
from langchain_community.vectorstores import Chroma
# Load the document, split it into chunks, embed each chunk and load it into the vector store.
raw_documents = TextLoader('press_conference.txt').load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
documents = text_splitter.split_documents(raw_documents)
db = Chroma.from_documents(documents, OpenAIEmbeddings())
Similarity search
query = "What did the president say about Ketanji Brown Jackson"
docs = db.similarity_search(query)
print(docs[0].page_content)

Output:

Today, I urge the Senate to prioritize critical legislation for the American people. Let's move forward with passing the Climate Action Plan, investing in renewable energy, and protecting our planet for future generations.

I also want to take a moment to recognize the dedication and service of our frontline healthcare workers. From doctors to nurses to medical staff, your tireless efforts during these challenging times have not gone unnoticed. Thank you for your unwavering commitment to keeping our communities safe and healthy

It is also possible to do a search for documents similar to a given embedding vector using similarity_search_by_vector which accepts an embedding vector as a parameter instead of a string.

embedding_vector = OpenAIEmbeddings().embed_query(query)
docs = db.similarity_search_by_vector(embedding_vector)
print(docs[0].page_content)

The query is the same, and so the result is also the same.

Vector stores are usually run as a separate service that requires some IO operations, and therefore they might be called asynchronously. That gives performance benefits as you don't waste time waiting for responses from external services. That might also be important if you work with an asynchronous framework, such as FastAPI.

LangChain supports async operation on vector stores. All the methods might be called using their async counterparts, with the prefix a, meaning async.

Qdrant is a vector store, which supports all the async operations, thus it will be used in this walkthrough.

pip install qdrant-client


from langchain_community.vectorstores import Qdrant
Create a vector store asynchronously
db = await Qdrant.afrom_documents(documents, embeddings, "http://localhost:6333")
Similarity search
query = "What statements did the president make regarding the Supreme Court nominee during the recent press conference?"
docs = await db.asimilarity_search(query)
print(docs[0].page_content)

Output:

Similarity search by vector

embedding_vector = embeddings.embed_query(query)
docs = await db.asimilarity_search_by_vector(embedding_vector)

Azure Cosmos DB, Azure Cognitive Search, Azure SQL, Azure Cache for Redis (Enterprise), and Azure Data Explorer (ADX) provide a robust suite of services for vector database requirements. These services cater to various applications, from MongoDB and PostgreSQL compatible services to AI-oriented applications with Azure AI Search. Azure also offers the option to host popular "vector native" databases like Pinecone, Qdrant, FAISS, Milvus, and Elastic Search on Azure, providing a flexible and scalable infrastructure for managing vector embeddings and enhancing AI capabilities through vector search and retrieval-augmented generation (RAG).

Amazon Web Services (AWS) offers a comprehensive suite of services for vector databases, including Amazon Aurora PostgreSQL-Compatible Edition, Amazon RDS for PostgreSQL, Amazon Neptune ML, Vector Search for Amazon MemoryDB for Redis, Amazon DocumentDB (with MongoDB compatibility), and Amazon OpenSearch Service. These services support the storage, indexing, and searching of high-dimensional vector data, making them ideal for machine learning applications and complex graph analysis.

Google Cloud Platform has integrated LangChain with all of its database offerings, including CloudSQL, Spanner, Firestore, Bigtable, and Memorystore for Redis, to enhance generative AI applications. These integrations support vector search and retrieval-augmented generation, enabling the development of applications that leverage Large Language Models (LLMs) with enterprise data.

Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs) are increasingly being used across various industries to enhance productivity, accuracy, and efficiency in tasks that require up-to-date, domain-specific knowledge. Here's how they are applied in specific industries:

In healthcare, RAG can be used to develop AI systems that provide accurate, up-to-date medical information to patients and healthcare providers. For example, RAG can be integrated into patient management systems to offer personalized medical advice based on the latest research and patient records, ensuring that healthcare providers have access to the most relevant information at their fingertips.

For the legal industry, RAG can significantly enhance the efficiency of legal research and case preparation. By integrating RAG with legal databases, AI systems can provide accurate citations, legal precedents, and case summaries, aiding lawyers in preparing arguments and ensuring the legality and accuracy of their cases. This not only improves the quality of legal services but also enhances auditability and transparency in legal proceedings.

In finance and banking, RAG can be used to develop AI-powered financial advisors that offer personalized financial advice based on real-time market data and customer financial information. This can include providing insights on investment opportunities, financial planning, and risk management, helping clients make informed financial decisions.

In the education sector, RAG can be utilized to create AI-powered tutoring systems that offer personalized learning experiences. These systems can provide students with accurate, up-to-date information on various subjects, helping them stay ahead in their studies. RAG can also be used to develop AI-powered grading systems that provide detailed feedback on assignments, enhancing the learning experience for students.

For e-commerce businesses, RAG can be integrated into customer service chatbots to provide customers with accurate, real-time product information, shipping updates, and personalized recommendations. This can significantly improve customer satisfaction and increase sales by providing personalized shopping experiences.

In manufacturing, RAG can be used to develop AI systems that monitor equipment and processes in real-time, providing operators with accurate, up-to-date information to ensure optimal performance and efficiency. This can help in predictive maintenance, reducing downtime and improving product quality.

For environmental monitoring, RAG can be integrated into AI systems that analyze satellite data and sensor readings to provide real-time information on environmental conditions. This can help in monitoring pollution levels, wildlife populations, and weather patterns, aiding in environmental conservation efforts.

Retrieval-Augmented Generation (RAG) systems, leveraging Large Language Models (LLMs), revolutionize various industries by enhancing AI capabilities beyond static training data. RAG facilitates real-time data integration, reducing costs and enhancing security by keeping sensitive data outside the model and allowing for real-time access restrictions. It offers greater explainability, reduces the likelihood of generating false information (hallucination), and overcomes context size limitations by dynamically retrieving relevant documents. This technology is instrumental in compliance checks, B2B sales, customer feedback analysis, product recommendations, financial consultation, insurance claims processing, financial reporting, and enhanced portfolio management, ensuring accuracy, efficiency, and security in these critical business processes. RAG's ability to dynamically pull relevant information from comprehensive databases ensures up-to-date, accurate, and personalized responses, making it a transformative tool in the evolving landscape of AI and business automation.

Here is a quiz to assess understanding of this chapter :

1. Which of the following is NOT a common application of Conversational AI?

- A. Personalized customer service

- B. Voice-activated smart home devices

- C. Social media bots

- D. Real-time translation services

- **Correct Answer: C. Social media bots

2. What is a significant challenge in Biomedical Document Understanding?

- A. Lack of standardization in document formats

- B. Difficulty in understanding complex medical terminologies

- C. High computational cost

- D. All of the above

3. In the context of Legal Search, what does AI primarily help with?

- A. Streamlining the legal research process

- B. Analyzing case law

- C. Predicting legal outcomes

- D. Writing legal documents

4. Which of the following is a benefit of using RAG (Retrieval-Augmented Generation) for Long-form text generation?

- A. Improved accuracy in text generation

- B. Reduced need for extensive training data

- C. Increased speed in generating long texts

- D. All of the above

5. Which of the following is NOT a type of Approximate Nearest Neighbor library?

- A. Faiss

- B. Annoy

- C. Euclidean

- D. PCA

6. Which industry does RAG (Retrieval-Augmented Generation) most commonly benefit?

- A. Healthcare

- B. Finance

- C. Manufacturing

- D. Retail

7. What is a key advantage of Conversational AI in customer service?

- A. It allows for personalized interactions

- B. It reduces the need for human agents

- C. It can handle multiple queries simultaneously

- D. It improves the speed of customer service

8. Which of the following is a primary goal of Biomedical Document Understanding?

- A. To replace human medical professionals

- B. To understand complex medical terminologies and documents

- C. To automate medical procedures

- D. To develop new drugs

9. Which of the following is a key benefit of using RAG for Long-form text generation?

- A. It can only generate short texts.

- B. It can generate long, coherent texts based on a given prompt.

- C. It is faster than traditional methods.

- D. It requires no training data.

10. Which of the following is a major use case for vector databases?

- A. Storing and retrieving large datasets

- B. Real-time data analysis

- C. Performing similarity searches

- D. All of the above

Correct Answers:

C. Social media bots
D. All of the above
A. Streamlining the legal research process
D. All of the above
C. Euclidean
A. Healthcare
A. It allows for personalized interactions
B. To understand complex medical terminologies and documents
B. It can generate long, coherent texts based on a given prompt.
C. Performing similarity searches
Connect with Me
If you have any inquiries, feel free to reach out via message or email.
Website/Newletter
Connect with me on Linkedin
Find me on Github
Visit my technical channel on Youtube

No posts

Chapter 6 - Real-World Applications of RAG and LLMs

Find source code used in this chapter

Connect with Me

Comments