RAG Pipeline Development Guide 2026: Architecture, Tools & Implementation

RAG pipelines have moved from research concept to production staple in 18 months. Almost every enterprise AI project we run at Ortem now includes a RAG component — whether it's a customer-facing chatbot, an internal knowledge assistant, or a document analysis tool. This guide covers what we've learned building them in production.

What Is a RAG Pipeline?

RAG (Retrieval-Augmented Generation) addresses the core limitation of large language models: they know everything their training data contained, and nothing else.

A RAG pipeline solves this by:

1. Taking your proprietary documents (PDFs, databases, wikis, emails)
2. Breaking them into chunks and converting them to vector embeddings
3. Storing those embeddings in a vector database
4. At query time, retrieving the most semantically similar chunks to the user's question
5. Passing those chunks as context to the LLM alongside the query
6. Returning a grounded, cited answer

The result: the LLM can answer questions about your specific data without knowing about it during training, without fine-tuning, and without hallucinating facts it doesn't have.

---

The Four Core Components

1. Document Ingestion

The ingestion pipeline handles loading documents from your source systems:

• File types: PDF, DOCX, HTML, Markdown, plain text, spreadsheets
• Sources: S3/GCS buckets, SharePoint, Confluence, Notion, Slack, databases
• Loaders: LangChain document loaders cover most formats. For PDFs: PyMuPDF for text, Unstructured for complex layouts.

from langchain.document_loaders import PyMuPDFLoader, UnstructuredFileLoader

# Simple PDF
loader = PyMuPDFLoader("policy-document.pdf")
documents = loader.load()

# Mixed file types (recursive)
from langchain.document_loaders import DirectoryLoader
loader = DirectoryLoader("./docs/", glob="**/*.pdf", loader_cls=PyMuPDFLoader)
documents = loader.load()

2. Chunking

How you split documents into chunks is the single biggest determinant of RAG quality. Poor chunking is the most common reason RAG systems give incoherent answers.

Chunking strategies:

Strategy	Best for	Chunk size
Fixed-size (character)	Simple, fast, decent baseline	500–1000 chars with 100 char overlap
Recursive text splitter	Most document types	500–1000 tokens with 10% overlap
Semantic chunking	High-quality retrieval	Variable — splits on semantic boundaries
Document-aware	PDFs with tables/headers	Requires layout-aware parser

Overlap is important. Without overlap, a chunk may contain half a sentence from the relevant passage. We typically use 10–15% overlap.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
    separators=["

", "
", ". ", " ", ""]
)
chunks = splitter.split_documents(documents)

3. Embedding + Vector Storage

Embeddings convert text chunks into numerical vectors that capture semantic meaning. Similar text produces similar vectors.

Embedding model options:

Model	Provider	Dimensions	Notes
text-embedding-3-small	OpenAI	1536	Best cost/quality for English
text-embedding-3-large	OpenAI	3072	Higher quality, 5× cost
embed-english-v3.0	Cohere	1024	Good multilingual support
all-MiniLM-L6-v2	HuggingFace (local)	384	Free, good for self-hosted
nomic-embed-text	Ollama (local)	768	Best open-source option

Vector database selection:

Database	Hosting	Best for
Pinecone	Managed	Easiest production setup
Weaviate	Managed or self-hosted	GraphQL API, good filtering
pgvector	Self-hosted (Postgres)	Already on Postgres, simple
Qdrant	Managed or self-hosted	High performance, production-ready
Chroma	Local only	Development and prototyping

from langchain_openai import OpenAIEmbeddings
from langchain_pinecone import PineconeVectorStore

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = PineconeVectorStore.from_documents(
    chunks,
    embeddings,
    index_name="my-knowledge-base"
)

4. Retrieval + Generation

At query time, the RAG pipeline:

1. Embeds the user query using the same embedding model
2. Searches the vector store for the top-k most similar chunks
3. Constructs a prompt with retrieved context
4. Calls the LLM for a grounded response

from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o", temperature=0)
retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=retriever,
    return_source_documents=True
)

result = qa_chain.invoke({"query": "What is the refund policy?"})
print(result["result"])
print(result["source_documents"])  # Always return sources

---

Advanced Patterns for Production

Hybrid Search (BM25 + Vector)

Pure vector search misses exact keyword matches. Pure BM25 misses semantic variants. Hybrid search combines both using Reciprocal Rank Fusion (RRF).

Most production RAG pipelines at Ortem use hybrid search — it consistently outperforms either method alone by 15–25% on recall metrics.

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

bm25_retriever = BM25Retriever.from_documents(chunks)
bm25_retriever.k = 5

vector_retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

hybrid_retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, vector_retriever],
    weights=[0.3, 0.7]
)

Re-ranking

After initial retrieval (top-20), a cross-encoder re-ranker scores each chunk against the query and re-orders them. The LLM then only sees the top-5 after re-ranking.

This significantly reduces context noise without reducing recall. Cohere Rerank and Jina Reranker are the two best options.

Metadata Filtering

Always store metadata with your chunks: document title, section, date, author, access level. Use metadata filters at retrieval time to scope queries to relevant subsets.

This is essential for multi-tenant deployments where users should only see their own data.

---

Evaluation: How to Know If Your RAG Is Working

The most common mistake in RAG development is shipping without an evaluation framework. You cannot improve what you cannot measure.

Key metrics:

Metric	What it measures	Tool
Faithfulness	Does the answer match the retrieved context?	Ragas, TruLens
Answer relevance	Does the answer address the question?	Ragas
Context precision	Is the retrieved context relevant?	Ragas
Context recall	Does retrieval find all relevant passages?	Ragas
Latency	End-to-end response time	Custom logging

Build a golden test set of 50–100 query/answer pairs from your domain experts before launch. Run evaluation against this set on every pipeline change.

---

Private RAG: On-Premise Deployment

For HIPAA, financial, or defence clients, no data can leave the organisation's infrastructure. A fully private RAG stack:

• LLM: Llama 3.1 70B or Mistral 7B via Ollama or vLLM
• Embeddings: nomic-embed-text or all-MiniLM-L6-v2 running locally
• Vector database: Qdrant or pgvector on-premise
• Orchestration: LangChain or LlamaIndex (no external API calls)

Performance is 80–90% of GPT-4o for most knowledge retrieval tasks at zero marginal cost per query after infrastructure.

---

Getting Help with RAG Pipeline Development

Building a production RAG pipeline — with hybrid search, evaluation, private deployment, and HIPAA/GDPR compliance — is a 6–10 week engagement. Ortem Technologies' LLM integration team has shipped RAG pipelines across healthcare, fintech, legal, and enterprise SaaS.

Book a free RAG scope call → | LLM Integration Services → | AI Agent Development →