Skip to main content
AI & ML OpsLast reviewed 2026-09-20

RAGOps

Operating the retrieval side of grounded AI systems: corpus ingestion and freshness, chunking and embedding versions, index rebuilds, retrieval quality, permission-aware search, and citations that actually support the answer.

Overview

Read the full RAGOps guidance
WHAT IT IS RAGOps is the operational discipline for retrieval-augmented systems: ingesting and refreshing a corpus, versioning the chunking and embedding strategy, rebuilding and reindexing without downtime, measuring whether the retrieved context actually contains the answer, enforcing the user's permissions at retrieval time, and verifying that citations support the claims made. **Not this:** RAGOps is not LLMOps: it owns the corpus, the index and what gets retrieved — the failures that look like model failures but are actually retrieval failures. WHY IT EXISTS Most bad answers from a grounded system are retrieval failures wearing a model's clothes. The generation step can only work with the context it was given; when the right document was never retrieved, no amount of prompt tuning fixes it, and teams routinely spend weeks on the wrong half of the system. The second reason is sharper: retrieval is where an authorization bug becomes a data breach, because an index that ignores permissions will happily ground a fluent, well-cited answer in a document the user was never allowed to read. WHEN TO USE IT Adopt it whenever a system answers from your own content. The permission question applies from day one if the corpus contains anything not already public — retrofitting access control onto an index built without it is substantially harder than building it in. WHO OWNS IT The engineering team owns the ingestion pipeline, the index and the retrieval path. The content owners own corpus quality and currency — retrieval over stale documents produces confidently outdated answers, and no engineering control fixes a wrong source. Security owns the access model enforced at query time, overlapping directly with /secops. Whoever owns the feature owns the retrieval quality bar. WHAT GOOD LOOKS LIKE Corpus freshness is measured, and documents have a known ingestion lag. Chunking and embedding configuration is versioned, and changing either triggers a full reindex rather than a silently mixed index — mixed embedding versions are a subtle and common defect. Retrieval quality is measured on a labelled set, so you know how often the right passage is actually retrieved. Permissions are enforced at retrieval time against the requesting user, not filtered afterwards. Citations are checked to support what was said, and reindexing happens without serving stale or partial results. HOW TO IMPLEMENT Start by measuring retrieval in isolation: take real questions, label which documents should be found, and measure how often they are. That number tells you whether to work on retrieval or generation, and most teams are surprised. Version the chunking and embedding configuration explicitly and record it alongside the index. Enforce permissions in the query path before any optimisation work. Establish a documented reindex procedure and rehearse it, treating it like any other production data migration — the runbook discipline in /runbooks applies. Add citation verification and corpus freshness monitoring once the basics hold. HOW TO ASSESS YOURSELF Ask how often the correct document is retrieved for a real question, and whether anyone has measured it. Ask what happens if a user asks about a document they cannot access. Ask when the corpus was last refreshed and how you would know if ingestion stopped. Take the scored self-assessment at /assess and turn the findings into a plan with /roadmaps.

Capabilities

  • Citation verification

    Checking that the cited sources actually support the claims made, rather than trusting that a fluent answer with links is grounded.

  • Corpus freshness monitoring

    Knowing how current the indexed content is, so the system does not answer confidently from documents that were superseded months ago.

  • Corpus ingestion

    A monitored pipeline that brings source documents into the index with a known lag and a known failure signal.

  • Embedding and chunking versions

    Explicit versioning of chunking and embedding configuration, because a silently mixed index is a subtle and common defect.

  • Permission-aware retrieval

    Enforcing the requesting user's access rights inside the query path rather than filtering afterwards — the control that keeps a retrieval bug from becoming a data breach.

  • Reindex procedure

    A rehearsed way to rebuild the index without serving partial or stale results, treated like any other production data migration.

  • Retrieval quality measurement

    Measuring on a labelled set how often the passage that answers a question is actually retrieved, which tells you whether to work on retrieval or generation.

Practices

  • Evaluation set curation

    Growing a fixed set of real cases — especially the ones that have already caused complaints — and keeping it small enough that it actually runs.

  • Reindex rehearsal

    Practising a full index rebuild on a schedule so the procedure is known to work before a corpus or embedding change forces one.

Metrics

  • Citation support rate

    Share of sampled answers whose cited sources actually support the claims made.

    Unit: percent. Good direction: up.

  • Corpus freshness lag

    Time between a source document changing and the index reflecting that change.

    Unit: hours. Good direction: down.

  • Retrieval permission findings

    Occasions where retrieval surfaced content the requesting user was not entitled to see — a count that should be zero and be checked.

    Unit: findings per quarter. Good direction: down.

  • Retrieval recall@k

    Share of labelled questions where a passage containing the answer appears in the top k retrieved results.

    Unit: percent. Good direction: up.

Tools

  • Langfuse

    Open-source observability, prompt management and evaluation for LLM applications, including retrieval traces.

    Visit site ↗
  • LlamaIndex

    Framework for building ingestion, indexing and retrieval pipelines over private content.

    Visit site ↗
  • Ragas

    Evaluation library for retrieval-augmented systems, scoring retrieval and answer quality against reference sets.

    Visit site ↗
  • pgvector

    Vector similarity search inside PostgreSQL, keeping retrieval in an existing database rather than adding a separate store.

    Visit site ↗

Architecture patterns

Reference architectures and their trade-offs live in the blueprints library.

Maturity

  1. L1 ReactiveDocuments were indexed once and forgottenrequired stage

    The corpus was loaded at build time with no refresh, no freshness signal and no measurement of whether retrieval finds the right passage.

  2. L2 RepeatableIngestion is monitored with a known lagrequired stage

    A pipeline refreshes the corpus on a schedule, its failures are visible, and how far behind the index runs is a known number.

  3. L3 StandardizedPermissions are enforced inside the query path

    Retrieval filters by the requesting user's entitlements before results are returned, rather than removing them after generation.

  4. L4 MeasuredRetrieval quality is measured on a labelled set

    Real questions with labelled answer passages give a recall figure, so effort goes to retrieval or generation on evidence rather than instinct.

  5. L5 OptimizedReindexing is rehearsed and citations are verified

    Full rebuilds are practised without serving partial results, embedding versions are explicit, and sampled answers are checked against their citations.

Maturity for ragops is measured, not guessed — every score traces to your answers. See how maturity is scored.

Put it to work

Sources