Why Most RAG Demos Fail in Production
The tutorial works: chunk documents, embed, retrieve, generate. But production adds constraints: latency budgets, cost per query, hallucination rates, stale data, permission-aware retrieval.
Pattern 1: Hybrid Search (BM25 + Dense)
Pure vector search misses exact matches (error codes, product SKUs, acronyms). Combine BM25 for keyword precision with dense vectors for semantic recall. Weight them 60/40 or learn the blend.
Pattern 2: Two-Stage Retrieval with Reranking
Retrieve 50-100 candidates with fast ANN search, then rerank with a cross-encoder. The latency hit is ~50ms but precision@k jumps 15-30%.
Pattern 3: Query Rewriting & Decomposition
User queries are often ambiguous or multi-part. Use an LLM to rewrite into standalone sub-queries, retrieve for each, then fuse results. Handles "Compare X and Y" or "What changed since last quarter?" naturally.
Pattern 4: Evaluation-Driven Development
You can't improve what you don't measure. Build a golden eval set (50-100 Q&A pairs) before you ship. Run it on every config change: chunk size, embedding model, reranker, prompt template.
Pattern 5: Cost-Aware Retrieval
Not every query needs the full pipeline. Route simple factual queries to a smaller model + fewer chunks. Reserve the expensive reranker + large context for complex reasoning queries.