RAGOps
Operating the retrieval side of grounded AI systems: corpus ingestion and freshness, chunking and embedding versions, index rebuilds, retrieval quality, permission-aware search, and citations that actually support the answer.
Overview
Read the full RAGOps guidance
Capabilities
Citation verification
Checking that the cited sources actually support the claims made, rather than trusting that a fluent answer with links is grounded.
Corpus freshness monitoring
Knowing how current the indexed content is, so the system does not answer confidently from documents that were superseded months ago.
Corpus ingestion
A monitored pipeline that brings source documents into the index with a known lag and a known failure signal.
Embedding and chunking versions
Explicit versioning of chunking and embedding configuration, because a silently mixed index is a subtle and common defect.
Permission-aware retrieval
Enforcing the requesting user's access rights inside the query path rather than filtering afterwards — the control that keeps a retrieval bug from becoming a data breach.
Reindex procedure
A rehearsed way to rebuild the index without serving partial or stale results, treated like any other production data migration.
Retrieval quality measurement
Measuring on a labelled set how often the passage that answers a question is actually retrieved, which tells you whether to work on retrieval or generation.
Practices
Evaluation set curation
Growing a fixed set of real cases — especially the ones that have already caused complaints — and keeping it small enough that it actually runs.
Reindex rehearsal
Practising a full index rebuild on a schedule so the procedure is known to work before a corpus or embedding change forces one.
Metrics
Citation support rate
Share of sampled answers whose cited sources actually support the claims made.
Unit: percent. Good direction: up.
Corpus freshness lag
Time between a source document changing and the index reflecting that change.
Unit: hours. Good direction: down.
Retrieval permission findings
Occasions where retrieval surfaced content the requesting user was not entitled to see — a count that should be zero and be checked.
Unit: findings per quarter. Good direction: down.
Retrieval recall@k
Share of labelled questions where a passage containing the answer appears in the top k retrieved results.
Unit: percent. Good direction: up.
Tools
Langfuse
Open-source observability, prompt management and evaluation for LLM applications, including retrieval traces.
Visit site ↗LlamaIndex
Framework for building ingestion, indexing and retrieval pipelines over private content.
Visit site ↗Ragas
Evaluation library for retrieval-augmented systems, scoring retrieval and answer quality against reference sets.
Visit site ↗pgvector
Vector similarity search inside PostgreSQL, keeping retrieval in an existing database rather than adding a separate store.
Visit site ↗
Architecture patterns
Reference architectures and their trade-offs live in the blueprints library.
Maturity
- L1 ReactiveDocuments were indexed once and forgottenrequired stage
The corpus was loaded at build time with no refresh, no freshness signal and no measurement of whether retrieval finds the right passage.
- L2 RepeatableIngestion is monitored with a known lagrequired stage
A pipeline refreshes the corpus on a schedule, its failures are visible, and how far behind the index runs is a known number.
- L3 StandardizedPermissions are enforced inside the query path
Retrieval filters by the requesting user's entitlements before results are returned, rather than removing them after generation.
- L4 MeasuredRetrieval quality is measured on a labelled set
Real questions with labelled answer passages give a recall figure, so effort goes to retrieval or generation on evidence rather than instinct.
- L5 OptimizedReindexing is rehearsed and citations are verified
Full rebuilds are practised without serving partial results, embedding versions are explicit, and sampled answers are checked against their citations.
Maturity for ragops is measured, not guessed — every score traces to your answers. See how maturity is scored.
Put it to work
Related Ops disciplines
Sources
- Tier 1 — PrimaryRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks ↗— Lewis et al. (arXiv)(verified 2026-09-20)
- Tier 2 — AuthoritativeOWASP Top 10 for Large Language Model Applications ↗— OWASP(verified 2026-09-20)
- Tier 2 — AuthoritativeRagas — Evaluation Metrics Documentation ↗— Ragas(verified 2026-09-20)