Skip to content
← Technical Diagram Library
AI Engineering

Production RAG System

See retrieval, reranking, generation and evaluation as one production path rather than a single LLM call.

LLMsEmbeddingsVector DBHybrid SearchRerankingOpenTelemetryEvaluation

Production RAG System

Reliable RAG separates retrieval quality, context assembly, model generation and evaluation. The architecture should make each stage observable so relevance, latency and hallucination risk can be measured independently.

01

Retrieval quality and generation quality are separate variables.

02

Reranking is a distinct control point for relevance.

03

Production systems need evaluation and telemetry in the loop.

Production RAG SystemSee retrieval, reranking, generation and evaluation as one production path rather than a single LLM call.01User / AppQuestion02AI GatewayAuth + policy03RetrieverHybrid search04Vector / SearchIndexed knowledge05RerankerTop-k refinement06LLMGrounded generation07Eval + TelemetryQuality • cost • latency
How to read it

Follow the handoffs, then ask where evidence exists.

01

User / App

Question

02

AI Gateway

Auth + policy

03

Retriever

Hybrid search

04

Vector / Search

Indexed knowledge

05

Reranker

Top-k refinement

06

LLM

Grounded generation

07

Eval + Telemetry

Quality • cost • latency

Architecture questions
Retrieval quality and generation quality are separate variables.
Reranking is a distinct control point for relevance.
Production systems need evaluation and telemetry in the loop.
Technology context
LLMsEmbeddingsVector DBHybrid SearchRerankingOpenTelemetryEvaluation

The diagram is intentionally architectural rather than vendor-specific. Use it as a mental model, then map the components to the actual environment.

Need the architecture applied?

Use the visual model as the starting point for a workshop or technical review.