Building Production-Ready RAG Applications with Python

A retrieval-augmented generation (RAG) demo takes an afternoon: load some documents, embed them, ask a question. A RAG system that people depend on takes longer, and almost none of the extra work is about the language model. This article walks through the parts that matter.

The five stages

Every RAG application has the same shape:

1. Ingest — parse source documents into clean text.
2. Index — split text into chunks, embed them, store the vectors.
3. Retrieve — find the chunks most relevant to a question.
4. Generate — ask the model to answer using only those chunks.
5. Evaluate — measure whether the answers are right.

Most quality problems come from stages 1 to 3. If the right passage never reaches the model, no prompt will fix the answer.

Ingestion: garbage in, confident garbage out

PDF extraction is where many pipelines quietly fail. Tables become word soup, headers repeat on every page, and two-column layouts interleave. Before tuning anything else, read your extracted text. If you would struggle to answer the question from it, so will the model.

Keep metadata with every chunk — source file, page, section heading. You will need it for filtering, for citations and for debugging.

Chunking and indexing

Fixed-size chunks are a reasonable start, but splitting on document structure (headings, clauses, table boundaries) usually retrieves better because each chunk is about one thing.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
    separators=["\n## ", "\n\n", "\n", ". "],
)
chunks = splitter.split_documents(documents)

One rule is non-negotiable: use the same embedding model and the same text normalization when indexing and when querying. A mismatch does not raise an error — it just returns worse results.

Retrieval: do not rely on vectors alone

Embedding search is good at meaning and poor at exact terms such as product codes, names and numbers. Keyword search is the opposite. Combining them (hybrid retrieval) and then re-ranking the merged candidates is the single most reliable improvement for real document sets.

Use metadata filters wherever you can. Restricting the search to the right document type or date range removes whole categories of wrong answers.

Generation: constrain the model

The prompt should do three things: tell the model to answer only from the supplied context, tell it what to say when the context does not contain the answer, and ask for citations.

SYSTEM = """Answer using only the context below.
If the context does not contain the answer, reply: "Not found in the provided documents."
Cite the source id for every claim, like [doc-3]."""

An honest "not found" is a feature. Users forgive a system that admits a gap far more readily than one that invents an answer.

Evaluation: the part that gets skipped

Build a set of 50 to 200 real questions with known correct answers and the passages that support them. Then measure two things separately:

• Retrieval — was the supporting passage in the top results?
• Answer — given the retrieved context, was the answer correct and grounded?

Separating them tells you where to work. Run the set on every change to chunking, prompts or models, exactly as you would run unit tests.

Operating it

In production, add what any service needs: request logging with the retrieved chunk ids, latency and token cost per query, caching for repeated questions, and a re-indexing job that runs when documents change. Version your prompts alongside your code.

Summary

Stage	Most common failure	First fix
Ingest	Broken text from PDFs	Inspect extracted text manually
Index	Chunks mix unrelated topics	Split on document structure
Retrieve	Exact terms missed	Hybrid search plus re-ranking
Generate	Invented answers	Strict grounding prompt with citations
Evaluate	No measurement	A labelled question set

RAG is a data and search problem with a language model at the end. Treat it that way and it becomes dependable.