RAG Architecture: Ground AI in Your Data

Build retrieval-augmented generation pipelines that give models access to your documents.

Why RAG exists

LLMs are frozen at their training cutoff. They don't know about your internal documents, recent events, or proprietary data. RAG (Retrieval-Augmented Generation) solves this by fetching relevant documents at query time and inserting them into the model's context. The model then answers based on retrieved content rather than training memory — giving you accurate, citable, up-to-date answers.

The five-step pipeline

Every RAG pipeline has five stages: (1) Ingest — load your documents. (2) Chunk — split them into passages of 200–800 tokens. (3) Embed — convert each chunk to a dense vector using an embedding model. (4) Retrieve — when a query arrives, embed it and find the nearest chunks by cosine similarity. (5) Generate — insert the top-k chunks into the model's context and let it answer.

Chunk size is the most impactful parameter. Too small: chunks lose context. Too large: retrieval becomes imprecise. 400–600 tokens is a good starting point.

Choosing a vector database

For prototyping: Chroma (local, zero-setup) or Pinecone (managed, generous free tier). For production: Weaviate, Qdrant, or pgvector (if you're already on PostgreSQL). For enterprise scale: Pinecone, Zilliz (managed Milvus), or Vertex AI Vector Search. If you're on Azure or AWS, their native vector services (AI Search, OpenSearch) reduce ops overhead. Don't over-engineer: a local FAISS index handles millions of vectors fine for most applications.

Choosing an embedding model

OpenAI's text-embedding-3-large is best-in-class quality but costs money per token. For open-source, nomic-embed-text-v1.5 and BGE-M3 are state-of-the-art and run locally. For multilingual content (Portuguese, Swedish, etc.), E5-multilingual or LaBSE outperform English-first models significantly. Always embed queries and documents with the same model — mixing models breaks the similarity scores.

Benchmark on your actual data. MTEB scores are useful signals but your domain may behave differently.

Improving retrieval quality

Basic cosine similarity retrieval degrades on complex queries. To improve it: (1) Hybrid search — combine dense vectors with BM25 keyword matching and re-rank. (2) Query expansion — use the model to rewrite the user query before embedding. (3) Parent-child chunking — retrieve small chunks for precision, then return their parent paragraph for context. (4) Metadata filters — store document date, author, section as metadata and filter before vector search. Each technique adds complexity; add them only when you can measure the improvement.

Concepts: RAG Concepts: Embeddings Resources: APIs & Frameworks