How to Build a Production RAG Pipeline (2026 Guide)

To build a production RAG pipeline in 2026, ingest documents with structure-aware chunking, preserve access and citation metadata, embed each chunk with one versioned model, and store vectors alongside searchable text. At query time, combine semantic and keyword retrieval, rerank the candidates, and give the language model only authorised context with stable citations. Release the system only after a representative question set checks retrieval, answer support, permissions, fallback behaviour, latency, and cost. If the source set is tiny, stable, or already modelled as structured records, ordinary search or a direct query may be simpler than RAG.

The production RAG framework: seven decisions

A RAG pipeline is a chain of contracts rather than a single model call. The useful question at each stage is not “which tool is fashionable?” but “what can this stage prove before the next stage depends on it?” Use this sequence:

1. Ingestion and structure-aware chunking turn documents into bounded passages.
2. Embedding gives each passage a vector, with a model and dimension that must remain identifiable.
3. Storage and indexing keep content, vectors, versions, and access metadata together.
4. Hybrid retrieval combines semantic similarity with exact-term search.
5. Reranking narrows a broad candidate set to context worth sending onward.
6. Grounding, citations, access control, and fallback constrain the answer to evidence the requester may use.
7. Evaluation and observability show which contract failed and whether a change is safe to release.

This is the RAG Production Decision Matrix and Readiness Worksheet. It is deliberately a decision aid, not a claim that any default works for every corpus. Fill the evidence column with your own checks before marking a row ready.

A blank evidence cell is not a pass. If a team cannot say what it will inspect, the design is still a proposal.

Architecture contracts that survive change

Start with a small data contract. The answer layer should not have to guess whether a passage is current, which tenant owns it, or how a reader may cite it.

type ChunkMetadata = {
  sourceId: string;
  documentVersion: string;
  title: string;
  heading?: string;
  position: number;
  tenantId: string;
  accessTags: string[];
  embeddingModel: string;
  embeddingDimension: number;
};

type Chunk = {
  id: string;
  text: string;
  metadata: ChunkMetadata;
};

Treat metadata as part of the retrieval result: provenance supports citations and re-indexing, tenant and access fields drive filters, and embedding fields make migrations auditable.

Structure-aware chunking

Split on document meaning first: headings, paragraphs, list items, and table rows should be boundaries where the source permits them. Fixed slices can separate a qualification from its rule. Preserve heading and position so a small passage can regain nearby context without placing the whole document in the prompt.

This compact TypeScript example shows the contract, not a universal parser:

function chunkMarkdown(
  doc: { id: string; version: string; title: string; tenantId: string; body: string },
  accessTags: string[],
  { maxChars = 1600, overlap = 200 } = {},
): Chunk[] {
  const blocks = doc.body.split(/\n(?=#{1,6}\s)|\n\s*\n/);
  const chunks: Chunk[] = [];
  let buffer = "";
  let heading: string | undefined;
  let position = 0;

  const flush = () => {
    const text = buffer.trim();
    if (!text) return;
    chunks.push({
      id: `${doc.id}:${doc.version}:${position}`,
      text,
      metadata: {
        sourceId: doc.id,
        documentVersion: doc.version,
        title: doc.title,
        heading,
        position,
        tenantId: doc.tenantId,
        accessTags,
        embeddingModel: "pending",
        embeddingDimension: 0,
      },
    });
    position += 1;
    buffer = buffer.slice(-overlap);
  };

  for (const block of blocks) {
    const match = block.match(/^#{1,6}\s+(.*)$/m);
    if (match) heading = match[1];
    if (buffer && buffer.length + block.length > maxChars) flush();
    buffer += `${buffer ? "\n" : ""}${block}`;
  }
  flush();
  return chunks;
}

The example uses character limits for readability. In production, measure tokens for the selected model and add handlers for tables, code, scans, and lists. Replace pending before indexing; exposing it makes migration state ambiguous. The 1,600-character and 200-character values are starting inputs, not measured outcomes.

Embeddings and storage

Choose one embedding model per index and record its identity on every vector. If the model or dimensions change, create a versioned migration: generate new vectors, validate source coverage and filters, switch reads deliberately, and retain rollback until the new path is accepted.

A minimal Postgres shape can keep vector and lexical retrieval close to the metadata filter:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id BIGSERIAL PRIMARY KEY,
  source_id TEXT NOT NULL,
  document_version TEXT NOT NULL,
  title TEXT NOT NULL,
  heading TEXT,
  content TEXT NOT NULL,
  tenant_id TEXT NOT NULL,
  access_tags TEXT[] NOT NULL,
  embedding_model TEXT NOT NULL,
  embedding VECTOR(1024) NOT NULL,
  tsv TSVECTOR GENERATED ALWAYS AS
    (to_tsvector('english', content)) STORED,
  created_at TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX chunks_embedding_hnsw
  ON chunks USING hnsw (embedding vector_cosine_ops);

CREATE INDEX chunks_tsv_gin
  ON chunks USING gin (tsv);

VECTOR(1024) is illustrative; replace it with the chosen model's dimension and keep that choice in deployment configuration. Compare these index definitions with the system's filtering, update, restore, and latency requirements. Choose a specialised store only for an explicit operating need, not to hide missing metadata discipline.

Retrieval and reranking

Semantic and lexical search answer different parts of a query. The semantic branch connects paraphrases; the lexical branch preserves identifiers, error codes, names, and version strings. Run both with the same scope predicates, then fuse their ordered results. RRF avoids comparing incompatible raw score scales:

type Hit = { id: string; position: number };

function reciprocalRankFusion(lists: Hit[][], k = 60) {
  const scores = new Map<string, number>();
  for (const list of lists) {
    for (const { id, position } of list) {
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + position));
    }
  }
  return [...scores.entries()]
    .sort((a, b) => b[1] - a[1])
    .map(([id]) => id);
}

The k = 60 value is an explicit starting parameter, not a universal setting. Retrieve broadly—for example, 40 candidates—then rerank and pass a smaller set, such as 5–8 chunks, only when it fits the prompt and latency budget. Log discarded candidates too; otherwise an early retrieval failure looks like a generation problem.

A reranker receives the query and each candidate as a pair, never unauthorised candidates. After reranking, apply a support check: if selected context lacks evidence for the claim, return “insufficient context”. Similarity is not permission to answer.

Example: an access-aware policy question

The following worksheet is intentionally illustrative. It is not a customer dataset, a completed test, or a report of an observed outcome. Assume a single-tenant employee portal contains a versioned travel policy and an older archived policy. A user asks: “Can a contractor expense an overnight hotel during domestic travel?”

For an authorised user, a grounded response could say: “The current Accommodation section permits overnight accommodation only where its stated travel and approval conditions are met. The available passage does not establish whether contractors are covered; confirm that point through the approved policy workflow.” The citation must point to the exact illustrative source span. If the retrieved context explicitly states the contractor rule, the answer may quote or summarise that rule instead. If it does not, the system should not fill the gap from general knowledge.

This example exposes a useful design test: change only the requester, not the question. If the answer changes because the user's access changes, the difference should be visible in the access decision and retrieved source set—not hidden in model behaviour.

Implementation steps from corpus to release

1. Define the answer contract and authority. State what needs a citation, what counts as insufficient evidence, which questions require a human or structured workflow, and which source is authoritative for each document family.

1. Inventory, permission, and parse. Record owner, version, effective date, tenant, access policy, format, and update/delete events. Preserve headings, lists, tables, page or section references, and stable IDs. Quarantine files the parser cannot represent safely.

1. Chunk and embed with versions. Inspect short, dense, tabular, and narrative documents before embedding. Store model name, version, dimension, creation time, and index version. A model change is a migration with a rollback path, not a prompt tweak.

1. Index content and metadata together. Add vector and lexical indexes, with tenant and access predicates as mandatory query inputs. Exercise updates and deletions so stale passages cannot remain answerable by accident.

1. Retrieve and fuse. Run semantic and keyword branches with identical scope filters, fuse their ordered results with RRF or another documented method, and log both branch results and fusion output.

1. Rerank, ground, and cite. Score query–chunk pairs, cap context by a token budget, retain stable source references, and reject context that is related but not evidential. Tell the model to use only supplied context and to report missing support.

1. Enforce fallbacks and trace operations. Define no-answer, search-only, human-escalation, provider-failure, and stale-index states. Record access decisions, latency, token usage, cost inputs, model versions, prompt versions, and response identifiers under the debugging policy.

1. Evaluate in two layers. Retrieval checks ask whether the right source is present and ordered usefully; answer checks ask whether the response is supported, relevant, and cited correctly. Keep approved questions with expected sources and agree thresholds before changes.

1. Release reversibly. Compare index and prompt versions, review failures by stage, and keep rollback available. Release only when the stated gates have evidence and the remaining limitations are understood.

For the separate release-gate protocol that goes deeper on answer support, citations, fallbacks, permissions, latency, and cost, use the RAG-answer testing guide in the CodeAustral journal. This build guide keeps the testing reference distinct so the architecture worksheet remains usable during design.

RAG, fine-tuning, or neither?

RAG supplies private, changing, large, or citation-sensitive facts. Fine-tuning shapes stable response behaviour; it cannot replace a current source of truth. Use both only when both needs are evidenced. Use neither for a small curated corpus, deterministic calculation, or direct database lookup. Choose by whether the gap is knowledge, behaviour, or computation.

Limitations

• The matrix has no universal thresholds; chunk size, candidate count, context budget, reranker, latency, and cost depend on the corpus, model, traffic, and permissions.
• The snippets omit parser coverage, retries, secrets, pooling, migrations, rate limits, and provider-specific handling. They show boundaries, not a complete service.
• RAG cannot repair an incomplete, contradictory, stale, or wrongly authorised source. Expose the gap or use an approved escalation path.
• Component versions can change retrieval and answer behaviour. Revalidate upgrades and retain version data in traces.
• A citation is useful only when its span supports the claim; an existing identifier is not evidence by itself.
• High-consequence decisions may need human review, rules, or an authoritative transaction system. This guide does not establish legal, medical, financial, privacy, or regulatory suitability.

FAQ

What is a sensible chunk size for RAG?

There is no single correct size. The existing implementation pattern gives 200–500 tokens as a starting range for ordinary prose, with roughly 10–15% overlap; these are tuning inputs, not measured outcomes. Start from document structure, measure tokens for the selected model, and adjust by document type.

Do I need a dedicated vector database?

Not necessarily. If Postgres already owns the application data and you need metadata and access filters beside vector search, pgvector is a practical first option. Consider a specialised store only when explicit sharding, throughput, isolation, or latency requirements exceed the current system's capabilities.

Why combine keyword and vector retrieval?

They fail differently. Vector retrieval connects paraphrases; lexical retrieval preserves identifiers, error codes, names, and version strings. Run both with the same scope filters and fuse their ordered results before reranking.

How should a RAG system handle an unanswered question?

Retrieve only from the authorised source set, instruct the answer layer to use supplied context, require citations, and let it state when evidence is insufficient. Add a deliberate fallback such as search-only results or human escalation. Fluency without support is a failure state.

Should I use RAG or fine-tuning?

Use RAG for current or private facts that need traceability; use fine-tuning for stable response behaviour or a narrow task pattern. Use neither when a direct query, rules engine, or curated answer is a better fit.

How do I prevent cross-tenant disclosure?

Make scope a mandatory part of both semantic and lexical retrieval. Store the access decision in the trace and filter before reranking and prompt assembly. Include denied and cross-tenant cases in the question set; if exclusion cannot be shown, stop the release.

A practical next step

CodeAustral's supplied public service description says its applied-AI work includes RAG over a team's content, evaluation, cost and latency controls, guardrails, fallbacks, and Next.js/Postgres integration. A useful brief states the corpus, owners, user groups, access rules, source policy, fallback, and release evidence needed.