RAG in Production: Retrieval Is an Engineering System

Retrieval-augmented generation (RAG) is often introduced as "search for context, then ask an LLM." That description is useful for learning, but production systems need a much broader engineering model.

The pipeline

Every arrow is a potential failure point. Poor parsing damages chunks; poor chunks damage retrieval; poor retrieval produces plausible but unsupported answers.

Chunking

There is no universally correct chunk size. The right strategy depends on document structure, user questions, retrieval model and the amount of context the generation step can use effectively.

A useful experiment compares multiple chunk strategies against a fixed evaluation set rather than choosing a size by intuition.

Retrieval

Measure retrieval separately from generation. If the relevant evidence never reaches the context window, prompt engineering cannot reliably repair the problem.

Useful retrieval measures include recall at k, ranking quality and whether the returned passage actually supports the expected answer.

Evaluation

Build a small, reviewed dataset of representative questions and expected evidence. Track changes to ingestion, embeddings, retrievers, rerankers, prompts and models against the same dataset.

Question → expected source → retrieved source → generated answer → reviewer outcome

Security

Retrieved content is input to the model and must be treated as untrusted. Consider authorization boundaries, sensitive data, prompt injection, malicious documents and source provenance.

Observability

Capture request latency, retrieval latency, retrieved document identifiers, token usage, model errors and evaluation outcomes. Avoid logging sensitive content without a deliberate privacy and retention policy.

Production RAG is therefore an application, data pipeline, retrieval system and model integration problem at the same time—which is exactly why connected engineering matters.