RAG. Simplified.

12 min read
Artificial IntelligenceRAGCAGLLMsEngineering

Diving into Retrieval Augmented Generation, how it works, the main RAG pipeline variants, and how it compares to semantic search and Cache-Augmented Generation.

RAG banner

I've been building AI based web applications for the past two years. One thing remains clear: without RAG, none of it is going to work as it's supposed to.

RAG?

If you're looking for a bookish definition, it stands for Retrieval Augmented Generation. It's a technique used to fix a common flaw in LLMs like ChatGPT or Gemini, their tendency to make things up because they can only rely on old training data.

To keep it short and concise, RAG gives AI the ability to "look things up" in an open book exam, pulling verified facts from external sources.

How RAG works

In a split second, a typical RAG system completes three major tasks. Simple RAG Retrieval is the initial stage. When you write a question, the system uses vector embeddings to search your own database or an external database for the most pertinent material based on meaning. The AI adds the retrieved, factual documents to the original query in the second stage known as Augmentation. Lastly, the language model analyses the augmented prompt, uses the retrieved documents as its source of truth and produces a precise, context-aware response in Generation.

Why RAG is important and useful, now more than ever

LLMs are a key technology powering intelligent chatbots and other natural language processing applications. The goal is to create bots that can answer user questions in various contexts by cross referencing authoritative knowledge sources. Unfortunately, the nature of LLM technology introduces unpredictability in responses. On top of that, LLM training data is static, which introduces a cutoff date on the knowledge it has.

Known challenges include:

I consider LLMs to be a hermit to civilization after years off the grid, speaking with total authority about a world that has quietly progressed without them. That kind of misplaced confidence can swiftly undermine user trust, so I don't want my chatbots to mimic it.

RAG is one way to solve some of these problems. It redirects the LLM to retrieve relevant information from authoritative, pre-determined knowledge sources. This gives organisations more control over the generated output, and gives users some insight into how the LLM actually arrived at its answer.

A few more benefits worth mentioning:

How does it work?

Without RAG, an LLM is stateless by default. It takes the user input and creates a response based purely on what it learned during training.

With RAG, an information retrieval component is introduced. It uses the user's input to first pull information from a new data source. Both the user's query and the relevant retrieved information are then passed to the LLM, which uses its training data along with this new context to generate a better response.

Create External Data

External Data refers to information that is not part of the LLM's initial training set. It can exist in various formats, including files, database records, and long form text, and it can originate from a variety of sources, including document repositories, vector databases and APIs.

This data is transformed into numerical representations and stored in vector database using a different method known as embedding models. In essence, this builds a knowledge base that generative AI models peruse and comprehend.

Retrieving relevant information

A relevancy search comes next. After being transformed into a vector representation, the user's query is compared against the database's vectors.

For example, think of a smart chatbot that can answer questions related to some codebase. If a developer asks, "Where is the logic for data ingestion?", the system retrieves the files related to ingestion along with the relevant lines of code. These specific results come back because they're the closest match to the query, mathematically speaking, using vector calculations to determine relevance. This is something I achieved myself while making Git Intel, a project that could tell you about your codebase before Github Copilot was a thing.

Augment the LLM prompt

Next, the system augments the user's input by adding the relevant retrieved data as context. For this step to work, prompt engineering should be used in order to communicate effectively with the LLM. This augments the prompt for the generation of an accurate and more grounded answer.

Down below, is a simple example of prompt engineering / augmentation.

const buildAugmentedPrompt = (query: string, chunks: { text: string; source: string }[]) => `You are a helpful assistant. Use the context below to answer the question. If the answer isn't in the context, say you don't know.

Context: ${chunks.map((c, i) => `[${i + 1}] (${c.source}): ${c.text}`).join("\n\n")}

Question: ${query}
Answer:`;

Once this prompt is built, you send it off to your LLM, and it generates the final response using both its own knowledge and the context you just handed it on a plate.

Types of RAG

Once I started diving deep into more RAG pipelines, I realised that RAG isn't really one single technique. There are more than 10 techniques but I have listed the ones which are used dominantly in the industry.

1. Naive RAG

Naive RAG

Most tutorials begin with this setup as it is the fundamental approach when building a RAG pipeline. The user's query is transformed into an embedding, the best matched chunks are extracted from your database, and are immediately inserted into the prompt. Nothing fancy, no rewriting, no reranking. Early on, I tried this for a basic FAQ bot and it was successful because the dataset was tiny and queries were predictable.

2. Advanced RAG

Advanced RAG This is where things get more interesting. Advanced RAG adds steps before and after retrieval to improve the results. Before retrieval, you might rewrite or expand the query so it matches your data better. After retrieval, you add a reranking step that sorts the chunks by how relevant they actually are, so the model isn't reading through noise. I moved to this approach once my naive RAG bot started pulling in irrelevant chunks for slightly ambiguous questions.

3. Modular RAG

Modular RAG Modular RAG allows you to mix and match the retrieval ranking, routing and generation components of the pipeline according to the task at hand. Consider it similar to creating using Lego pieces rather than a single, fixed pipelines. This method enables me to efficiently route some queries to a SQL database and others to a vector store.

4. Graph RAG

Graph RAG Instead of retrieving flat chunks of text, Graph RAG pulls from a knowledge graph where entities and their relationships are mapped out. This is great for questions that need multi hop reasoning, like "which suppliers does Company A depend on that are also based in the region affected by the new regulation?" A flat vector search struggles with that kind of question, but a graph can walk through the connections to get there.

5. Agentic RAG

Agentic RAG This type is now receiving the most attention and is the most active. An agentic RAG system organises its own research rather than doing a single retrieval and declaring it finished. It can determine whether it needs to look at more than one source, assess whether the initial result truly answers the issue, and look again if it does not. I see it more as a research assistant who continues to work until they are certain of the solution than as a librarian giving you a book, and that behaviour makes it the most expensive RAG types out there.

In my opinion, both Self-RAG and Corrective RAG (CRAG) allow the model to evaluate its own retrieved results and determine whether to repeat the search again.

RAG vs Semantic Search

This one confused me for long time because RAG and semantic search are based on the same basic technology; embeddings and vector databases. Therefore, it is simple to presume that they are the same; except, they are NOT the same.

Semantic search is just the retrieval part. You type a query, it gets converted into a vector, and the system comes back with the pages or chunks closest to it in meaning, not just by matching keywords. That's the whole job, a ranked list of results. Think of a docs site search bar that somehow understands "how do I log a user out?" even when the docs only ever say "ending a session.".

RAG adds a second layer; generation, to the same retrieval procedure. Once the right sections are found, they get handed to an LLM, which reads through them and puts together a clear answer. So instead of being handed a pile of five documents to dig through yourself, you get something direct; like "to log a user out, call /api/v1/session/logout endpoint. Here's how ...".

The way I've come to think about it; semantic search is the librarian who points you to the right shelf. RAG is the librarian who actually pulls the books, finds the relevant pages, and just tells you what you need to know.

RAG vs CAG

CAG, or cache augmented generation, is a newer idea that's been picking up attention, and I think it's worth knowing about because it solves a similar problem to RAG, just in a very different way.

With RAG, every time a user asks something, the system goes off and retrieves relevant documents in real time, then feeds them to the model. That's great for data that's large or changes often, but it adds latency, since there's a search happening on every query, and it adds complexity, since you're maintaining a whole retrieval pipeline with vector databases and indexes.

CAG flips this around. Instead of retrieving at query time, you preload all the relevant knowledge into the model's context window ahead of time, and the model caches its internal state for that data, often called the key value cache. When a question comes in, the model just answers from what's already loaded, with no retrieval step in the way.

The tradeoff becomes pretty obvious once you see it laid out. CAG works really well when your knowledge base is small and doesn't change much, think a company policy document or a product manual. Since there is nothing to look up, it is quick and easy. However, CAG begins to fail if your data is big or updated regularly because you would have to constantly reload and recalculate that cache each time something changed.

I've come to think of it like this; CAG is like someone who read the entire handbook last night and is now responding from memory, but RAG is like a researcher who checks something up every time you ask a question. If the manual changes every week, you want the researcher. If it barely ever changes, the person who already memorized it is faster.

For most of what I build, the knowledge base keeps growing and changing, so RAG still makes more sense for me. But for smaller, stable use cases, CAG is genuinely worth considering instead of reaching for a vector database by default.

Why Not Just Paste Text Into a Normal Prompt?

Let's assume you pasted a massive 1,00,000-word document into a standard prompt every time a user asked a question. You'd face two massive problems.

CAG solves this by using Prompt Caching. The AI would keep the processed "mega-prompt" warm in its server memory. And when you ask a question, the model will skip reading the documentation and goes straight to generating the answer. This cuts response times down to milliseconds, and will not empty your pockets as it also reduces token costs by 90%

Where this leaves me

Looking back at everything above, RAG isn't a single pattern you implement once and move on from. It is a spectrum; where you land on it depends entirely on what you are building. A small FAQ bot does not need agentic self-correction anymore than a multi-hop research assistant can survive on naive retrieve-and-stuff. The trick is matching the complexity of the pipeline to the complexity of the problem at hand, instead of reaching for the fanciest technique because it is the one everyone's talking about.

If I had to sum up everything I've learned while building these systems into one single rule, it would be start naive, and let the failures tell you what to add. Naive RAG breaks when retrieval gets noisy, that is when you add reranking and query rewriting (Advanced RAG). Advanced RAG breaks when one retrieval strategy cannot serve every kind of query, that's when you start routing and mixing component (Modular RAG). And so on, up through Graph and Agentic RAG. Each layer exists because something simpler broke first.

I do not think RAG is fading away anytime soon, at least not while LLMs are still stateless and knowledge keeps changing faster than any model's training cutoff. What is truly changing is how we build it.