Large Language Models have transformed how applications interact with users, but they come with a major limitation: they do not know your private data. If you want an AI assistant that can answer questions about your internal documents, PDFs, or knowledge base, you need a way to connect those documents to the model. This is where Retrieval-Augmented Generation (RAG) comes in.
RAG enables an AI system to retrieve relevant information from your documents and use that context to generate accurate, grounded answers. Instead of relying only on pre-trained knowledge, the model becomes a dynamic system that reasons over your data.
In this article, you will learn how to build a RAG Q&A AI agent using LangChain4j. The solution will load documents, convert them into embeddings, store them in a vector database, retrieve relevant content at runtime, and generate answers using a language model.
1. What is RAG (Retrieval-Augmented Generation)?
Retrieval-Augmented Generation is an architecture that combines two key steps:
- Retrieval: Find relevant pieces of information from a knowledge base.
- Generation: Use a language model to generate a response using that information.
Instead of asking a model a question directly, the system first retrieves relevant context from documents and adds it to the prompt. This helps the model generate answers that are grounded in real data rather than guesses. By working this way, the system ensures that responses are based on actual information instead of relying only on what the model learned during training.
This approach improves factual accuracy by reducing hallucinations and allowing the model to reference trusted sources at runtime. It also makes it possible to use private, domain-specific, and constantly changing data without retraining the model. As a result, it becomes a practical solution for real-world applications, with key benefits such as reduced hallucinations, access to private data, up-to-date responses and better accuracy.
2. Why use LangChain4j?
LangChain4j makes it easier for Java developers to build AI applications that use large language models together with real-world data. Instead of working with complex low-level APIs, it provides simple and well-structured tools that help us build features like Retrieval-Augmented Generation with less effort and less code.
It also makes it easier to connect language models to external data sources, create and manage embeddings, and handle how information is retrieved and used to generate answers. LangChain4j works well with Java frameworks, supports embeddings and vector stores out of the box, and makes it simple to build complete RAG pipelines. It also supports tools and agent-based designs, and allows us to work with different model providers, giving us flexibility when building AI applications.
3. Architecture Overview
A RAG Q&A agent built with LangChain4j follows a simple flow. The system is designed to combine your documents with a language model so that answers are based on real data. The architecture has the following parts:
- Document Ingestion: Your documents (PDFs, text files, web pages, or database records) are loaded and split into smaller chunks. These chunks make it easier to search and retrieve only the most relevant information later.
- Embeddings and Vector Storage: Each document chunk is converted into an embedding, which is a numerical representation of its meaning. These embeddings are stored in a vector database. This allows the system to quickly find similar content when a question is asked.
- Retriever: When a user asks a question, the system converts the question into an embedding and searches the vector store. It retrieves the most relevant document chunks based on similarity.
- LLM (Answer Generation): The retrieved content is added to the prompt sent to the language model. The model then generates an answer using both the user’s question and the retrieved context.
Setting Up Ollama
To run your RAG agent locally, you need a model provider, and Ollama offers a simple way to run models directly on your machine without relying on external APIs. Start by installing Ollama from its official website at Ollama official site, then confirm the installation was successful by running a version check command in your terminal.
ollama --version
Next, you need to pull a model that supports chat and embeddings so your RAG agent can process queries and retrieve relevant information. You can download a suitable model using the Ollama CLI:
ollama pull qwen3:8b
4. Project Setup
To build the RAG Q&A agent, this article uses Quarkus as the development framework. Quarkus provides excellent support for building AI-powered services with LangChain4j. You can quickly bootstrap the project using the Quarkus Maven plugin:
mvn io.quarkus:quarkus-maven-plugin:create \
-DprojectGroupId=com.example.jcg \
-DprojectArtifactId=rag-document-assistant \
-Dextensions="quarkus-langchain4j-ollama,rest-jackson" \
-DclassName="com.example.jcg.RagResource" \
-Dpath="/chat"
Add Additional Dependencies
Update your pom.xml to include embeddings support. To support local embeddings, we add a dependency for the BGE small embedding model
<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-embeddings-bge-small-en-q</artifactId>
</dependency>
This dependency provides the bge-small-en-q embedding model. The model generates 384-dimensional vectors and is efficient and well-suited for this RAG use case.
Configuration
Add the following to application.properties:
quarkus.langchain4j.ollama.base-url=http://localhost:11434 quarkus.langchain4j.ollama.chat-model.model-name=qwen3:8b quarkus.langchain4j.ollama.log-requests=true quarkus.langchain4j.ollama.log-responses=true quarkus.langchain4j.ollama.timeout=120s quarkus.langchain4j.embedding-model.provider=dev.langchain4j.model.embedding.onnx.bgesmallenq.BgeSmallEnQuantizedEmbeddingModel rag.path=src/main/resources/docs
This configuration defines how your RAG application connects to models and where it loads documents from.
quarkus.langchain4j.ollama.base-url=http://localhost:11434: This sets the base URL for Ollama, which runs locally on your machine.quarkus.langchain4j.ollama.chat-model=qwen3:8b: This specifies the chat model used to generate answers.quarkus.langchain4j.embedding-model.provider=dev.langchain4j.model.embedding.onnx.bgesmallenq.BgeSmallEnQuantizedEmbeddingModel: This overrides the default embedding provider and uses a local ONNX-based BGE small (quantized) model instead.rag.path=src/main/resources/doc: This is a custom property that defines where your documents are stored. Your ingestion component will read files from this directory and load them into the embedding store.
Create Your Own Document (Knowledge Base)
Create a file: docs/remote-work-guide.md
# Remote Work Productivity Guide Remote work can improve productivity when done correctly. Key practices include: 1. Create a dedicated workspace to avoid distractions. 2. Follow a consistent daily schedule. 3. Take regular breaks to avoid burnout. 4. Use communication tools effectively to stay aligned with your team. 5. Set clear goals for each day. Benefits of remote work: - Increased flexibility - Reduced commuting time - Better work-life balance Challenges: - Isolation - Communication gaps - Difficulty separating work and personal life
Document Ingestion
This class loads and processes the document into embeddings.
@ApplicationScoped
public class KnowledgeBaseLoader {
public void ingestDocuments(@Observes StartupEvent ev, EmbeddingStore store, EmbeddingModel model, @ConfigProperty(name = "rag.path") Path documents) {
List<Document> list = FileSystemDocumentLoader.loadDocumentsRecursively(documents);
EmbeddingStoreIngestor ingestor = EmbeddingStoreIngestor.builder()
.embeddingStore(store)
.embeddingModel(model)
.documentSplitter(recursive(100, 25, new HuggingFaceTokenCountEstimator()))
.build();
ingestor.ingest(list);
Log.info("Documents ingested successfully");
}
}
This class is responsible for the document ingestion phase of the RAG pipeline and is automatically triggered when the application starts. By observing the StartupEvent, it ensures that all documents are loaded, processed, and stored before any user queries are handled. This guarantees that your AI assistant has a fully prepared knowledge base ready for retrieval.
The key components injected into the method are the EmbeddingStore, EmbeddingModel, and the document path (rag.path). The EmbeddingStore is where all vectorized document chunks are stored, while the EmbeddingModel is responsible for converting raw text into numerical vector representations that can be searched semantically. The documents themselves are loaded recursively from the configured directory using FileSystemDocumentLoader.
A critical part of this process is the document splitting strategy. The recursive(100, 25, new HuggingFaceTokenCountEstimator()) configuration breaks documents into chunks of 100 tokens with an overlap of 25 tokens.
Finally, ingestor.ingest(list) performs the core ingestion process: it splits the documents, generates embeddings for each chunk, and stores them in memory.
Retrieval Setup
This retrieves relevant content based on user queries.
public class ContentRetrievalProvider {
@Produces
@ApplicationScoped
public RetrievalAugmentor buildRetriever(EmbeddingStore store, EmbeddingModel model) {
var contentRetriever = EmbeddingStoreContentRetriever.builder()
.embeddingModel(model)
.embeddingStore(store)
.maxResults(3)
.build();
return DefaultRetrievalAugmentor.builder()
.contentRetriever(contentRetriever)
.build();
}
}
This class implements the retrieval step of the RAG pipeline. It is responsible for finding the most relevant pieces of information from the embedded documents when a user asks a question.
The core component here is the EmbeddingStoreContentRetriever. It takes the user’s query, converts it into an embedding using the same EmbeddingModel, and searches the EmbeddingStore for similar vectors. The maxResults(3) setting ensures that only the top 2 most relevant document chunks are retrieved, keeping the context focused.
The DefaultRetrievalAugmentor then wraps this retriever and prepares the retrieved content to be added to the prompt. When the augment() method is called, it returns the request containing both the user’s question and the retrieved context. This augmented input is what the language model uses to generate accurate, grounded answers.
In-Memory Vector Store
@ApplicationScoped
public class EmbeddingStoreProvider {
@Produces
@ApplicationScoped
EmbeddingStore embeddingStore() {
return new InMemoryEmbeddingStore<>();
}
}
This class defines the vector storage mechanism used by the application. It produces an InMemoryEmbeddingStore, which is a lightweight implementation that keeps all embeddings in memory.
Using an in-memory store is ideal for development and testing because it requires no external infrastructure and provides fast access to embeddings. However, since the data is not persisted, all embeddings will be lost when the application restarts.
This component works closely with both the ingestion and retrieval steps: during ingestion, embeddings are stored here, and during querying, they are searched to find the most relevant content. In production scenarios, this can be replaced with more robust vector databases such as Infinispan, PostgreSQL with pgvector, or other specialised vector stores.
AI Service Interface
This is the interface that connects your application to the language model.
@RegisterAiService
@ApplicationScoped
@SystemMessage("You are a helpful assistant that answers questions based on a remote work productivity guide.")
public interface ProductivityAssistant {
String ask(@UserMessage String question);
}
This interface defines the AI layer of the RAG system. The @RegisterAiService annotation lets LangChain4j automatically generate the implementation, so you don’t need to manually call the language model. The @SystemMessage sets the assistant’s behavior, guiding it to answer questions based on the provided document context. The ask method takes the user’s question (@UserMessage), which is combined with retrieved content and sent to the model.
In summary, this is the entry point where the user query and retrieved data come together to generate a final, context-aware answer.
REST Endpoint (Testing the RAG Agent)
The RAG pipeline is exposed through a REST API endpoint.
@Path("/chat")
public class RagResource {
@Inject
ProductivityAssistant assistant;
@POST
@Consumes(MediaType.TEXT_PLAIN)
@Produces(MediaType.TEXT_PLAIN)
public String askQuestion(String question) {
return assistant.ask(question);
}
}
Run the Application
To run the application, start Quarkus in development mode using the command mvn quarkus:dev. To test the RAG API, send a request containing your question to the exposed endpoint:
curl -X POST http://localhost:8080/ask \
-H "Content-Type: text/plain" \
-d "What are the benefits of remote work?"
Sample Response
The benefits of remote work include: - **Increased flexibility**: Remote work allows you to manage your schedule and tasks according to your personal preferences and peak productivity times. - **Reduced commuting time**: Eliminating the need to travel to an office saves time and reduces stress, giving you more hours in your day for work, rest, or personal activities. - **Better work-life balance**: With the boundaries between work and personal life more defined, remote work can help you prioritize self-care, family time, and hobbies without the disruptions of a traditional office environment. While challenges like isolation or communication gaps may arise, these benefits make remote work an attractive option for many professionals
6. Using WebSocket (Real-Time Streaming)
Instead of a REST endpoint, you can use WebSockets to stream responses token-by-token in real time. First, add WebSocket support using Quarkus WebSockets Next:
./mvnw quarkus:add-extension -Dextension=websockets-next
WebSocket Endpoint
@WebSocket(path = "/chat")
public class RagWebSocket {
@Inject
ProductivityAssistant assistant;
@OnOpen
Multi<String> onOpen() {
return Multi.createFrom().item("Connected to RAG assistant. Ask questions about your documents.");
}
@OnTextMessage
public Multi<String> onMessage(String userQuery) {
return assistant.ask(userQuery);
}
}
This WebSocket endpoint replaces the REST API by keeping a persistent connection open between the client and server. When a user sends a message, the assistant.ask() method returns a Multi<String>, which streams the response tokens progressively instead of waiting for a full response. This allows the UI (or even a terminal client) to display the answer as it is being generated, making the interaction feel faster and more natural.
Web UI (HTML + JavaScript)
Place the following file in: src/main/resources/META-INF/resources/index.html
<!DOCTYPE html>
<html>
<head>
<title>RAG WebSocket Chat</title>
<style>
body {
font-family: Arial, sans-serif;
background: #f5f5f5;
margin: 20px;
}
h2 {
text-align: center;
}
#chat {
border: 1px solid #ccc;
background: white;
padding: 10px;
height: 350px;
overflow-y: auto;
display: flex;
flex-direction: column;
gap: 8px;
}
.user, .bot {
padding: 10px;
border-radius: 8px;
max-width: 75%;
line-height: 1.4;
}
.user {
align-self: flex-end;
background: #d1e7dd;
}
.bot {
align-self: flex-start;
background: #e2e3e5;
}
#inputArea {
margin-top: 10px;
display: flex;
gap: 10px;
}
#input {
flex: 1;
padding: 10px;
}
button {
padding: 10px 15px;
cursor: pointer;
}
</style>
</head>
<body>
<h2>RAG Chat Assistant</h2>
<div id="chat"></div>
<div id="inputArea">
<input type="text" id="input" placeholder="Ask something about your documents..." />
<button onclick="sendMessage()">Send</button>
</div>
<script>
const chatBox = document.getElementById("chat");
const input = document.getElementById("input");
let socket;
let buffer = "";
function connect() {
socket = new WebSocket("ws://localhost:8080/chat");
socket.onopen = () => {
console.log("Connected");
};
socket.onmessage = (event) => {
try {
// accumulate streamed chunks
buffer += event.data;
// normalize spacing
buffer = buffer.replace(/\s+/g, " ").trim();
// convert newlines into paragraphs
const formatted = buffer
.split(/\n+/)
.map(p => `<p>${p}</p>`)
.join("");
let lastMessage = chatBox.lastElementChild;
// update existing bot message instead of creating new lines
if (lastMessage && lastMessage.className === "bot") {
lastMessage.innerHTML = formatted;
} else {
const botDiv = document.createElement("div");
botDiv.className = "bot";
botDiv.innerHTML = formatted;
chatBox.appendChild(botDiv);
}
scrollToBottom();
} catch (e) {
console.error("Error handling message:", e);
}
};
socket.onerror = () => {
appendSystemMessage("Connection error");
};
}
function sendMessage() {
const message = input.value.trim();
if (message === "") return;
// reset buffer for new response
buffer = "";
const userDiv = document.createElement("div");
userDiv.className = "user";
userDiv.innerText = message;
chatBox.appendChild(userDiv);
socket.send(message);
input.value = "";
scrollToBottom();
}
function appendSystemMessage(msg) {
const div = document.createElement("div");
div.className = "bot";
div.innerText = msg;
chatBox.appendChild(div);
}
function scrollToBottom() {
chatBox.scrollTop = chatBox.scrollHeight;
}
window.onload = connect;
</script>
</body>
</html>
To run and test the application, start it in development mode using mvn quarkus:dev, then open your browser and navigate to http://localhost:8080, where you can interact with the application. At this point, you should see a simple chat UI.
When you type a question and click Send, your message appears instantly in the interface, followed by the AI-generated response streaming in gradually rather than appearing all at once. The output creates a smooth, real-time conversational experience similar to modern AI chat applications.
7. Conclusion
In this article, we explored how to build a Retrieval-Augmented Generation (RAG) application using Quarkus and LangChain4j, starting from document ingestion and embedding generation to storing vectors and retrieving relevant context for answering user queries. We first implemented a traditional REST-based approach and then enhanced the experience by introducing WebSocket communication to stream responses in real time.
8. Download the Source Code
This article explored how to build a RAG QA AI agent for your documents using LangChain.
You can download the full source code of this example here: build a RAG QA AI agent for your documents using langchain
