Definition
RAG system monitoring is the practice of tracking the health and quality of retrieval-augmented generation pipelines in production. Unlike monitoring a traditional API (where success means a 200 response), RAG monitoring must capture whether the system is retrieving relevant context, whether the retrieved context is current, whether the embedding model is producing consistent vectors, and whether the generated answers are actually correct. A RAG system can return a 200 response with a confidently wrong answer — making quality monitoring essential.
Architecture
A production RAG monitoring stack operates at four layers, each with distinct metrics and failure signals.
Layer 1: Embedding Pipeline Health
The embedding pipeline converts source documents into vectors. When it fails or degrades, the knowledge base becomes stale.
Source Documents → Chunking → Embedding → Vector Store
│ │ │ │
Metrics: Metrics: Metrics: Metrics:
- doc count - chunk - latency - index size
- staleness size - error - query
- change dist rate latency
rate - count - model ver - capacity
Key metrics to track:
Index freshness. The time between a source document changing and its embeddings being updated in the vector store. If your knowledge base updates daily but the embedding pipeline runs weekly, users get stale answers for up to six days.
# Track embedding pipeline lag
pipeline_lag = {
"last_source_change": "2026-02-28T10:30:00Z",
"last_index_update": "2026-02-28T10:35:00Z",
"lag_seconds": 300,
"documents_pending": 12,
"chunks_pending": 47,
}Embedding errors. Track the error rate of the embedding model — timeouts, malformed inputs, dimension mismatches. A spike in embedding errors means new documents are not being indexed.
Model version tracking. When you change embedding models, vectors generated by the old model are incompatible with the new model. Track which model version generated each vector and alert when a query compares vectors from different models.
Layer 2: Retrieval Quality
Retrieval quality is the most important metric and the hardest to measure automatically.
Retrieval relevance scores. Most vector databases return a similarity score with each result. Track the distribution of top-K scores over time. A gradual decline in average relevance scores indicates that the query patterns are drifting away from the indexed content — or that the content is drifting away from what users ask about.
Empty retrieval rate. How often does the retrieval step return no results above the relevance threshold? A rising empty retrieval rate means either the threshold is too high, the index is missing content, or users are asking about topics outside the knowledge base.
Chunk utilization. Which chunks are retrieved most frequently? Which are never retrieved? Frequently retrieved chunks may need to be split for precision. Never-retrieved chunks may indicate content that is not aligned with how users phrase their queries.
-- Track retrieval patterns over time
SELECT
date_trunc('hour', queried_at) AS hour,
avg(top_score) AS avg_relevance,
count(*) FILTER (WHERE top_score < 0.5) AS low_relevance_count,
count(*) FILTER (WHERE result_count = 0) AS empty_retrieval_count,
count(*) AS total_queries
FROM rag_queries
WHERE queried_at > now() - interval '7 days'
GROUP BY 1
ORDER BY 1;Layer 3: Generation Quality
The generation layer takes retrieved context and produces an answer. Monitoring here is harder because "correct" is subjective.
Answer groundedness. Is the generated answer supported by the retrieved context? This can be partially automated by using a separate LLM call to check whether the answer contains claims not found in the context (hallucination detection).
Answer length distribution. A sudden change in average answer length can indicate a prompt regression, a model behavior change, or a context quality issue. If answers suddenly become much shorter, the model may be receiving less useful context.
User feedback signals. If your system has thumbs-up/thumbs-down or similar feedback, track the ratio over time by query category. A declining satisfaction ratio in a specific category points to content gaps or retrieval issues in that domain.
Layer 4: System Health
Standard infrastructure monitoring still applies:
Query latency breakdown. Decompose end-to-end latency into embedding time, retrieval time, and generation time. A latency spike in one component is easier to diagnose than an overall latency increase.
Token usage. Track input tokens (retrieved context) and output tokens (generated answer) per query. Rising input tokens may indicate that the retrieval step is returning too many chunks. Rising output tokens may indicate verbose prompts.
Error rates by component. Separate error rates for the embedding service, vector store, and LLM provider. An error in any component affects the end-to-end experience differently.
Example
A production RAG system monitoring dashboard might track these daily:
RAG Health Dashboard — Last 24 Hours
Embedding Pipeline
Documents processed: 1,247
Processing errors: 3 (0.2%)
Avg processing time: 1.2s
Index freshness lag: 5m
Retrieval Quality
Total queries: 8,432
Avg top-3 relevance: 0.78 (baseline: 0.76)
Empty retrievals: 124 (1.5%)
Low relevance (<0.5): 312 (3.7%)
Generation Quality
Avg answer length: 185 tokens
Groundedness score: 0.91 (LLM-judge)
User satisfaction: 87% positive
System Performance
P50 latency: 420ms
P99 latency: 2,100ms
LLM errors: 7 (0.08%)
Vector store errors: 0
The actionable signals here are comparative. An empty retrieval rate of 1.5% might be fine, but if it was 0.8% last week, something changed. The monitoring value is in trends, not absolute numbers.
Failure Modes
Embedding Drift
When you update your embedding model, all new vectors are in a different embedding space than the existing vectors. Queries embed with the new model but search against old vectors. Relevance scores drop because the cosine similarity between vectors from different models is meaningless.
Prevention: when changing embedding models, re-embed the entire corpus before switching the query path. Use model version tags on every vector and alert if a query attempts cross-model comparison.
Stale Index Syndrome
The embedding pipeline stops processing updates due to a silent failure — a credential expiration, a source system change, a disk space issue. The index becomes progressively stale. Users continue to get answers, but the answers are based on outdated information. No errors are thrown because the system is functioning correctly with old data.
Prevention: track the timestamp of the most recent document processed by the embedding pipeline. Alert if the pipeline has not processed any documents in a period that exceeds your expected update frequency. This is a liveness check for data freshness, not just system health.
Relevance Threshold Miscalibration
Setting the relevance threshold too low returns irrelevant chunks that confuse the LLM. Setting it too high returns nothing, forcing the LLM to generate without context. The optimal threshold depends on your data, your embedding model, and your query patterns — and it changes as any of these change.
Prevention: track the distribution of relevance scores, not just the average. If the score distribution becomes bimodal (clusters at high and low relevance with nothing in between), your chunking strategy may be producing chunks that are either perfect matches or completely irrelevant. A healthy distribution has a smooth curve.
Context Window Pollution
When the retrieval step returns chunks that are technically relevant but not useful — boilerplate text, repeated headers, table-of-contents fragments — the LLM's context window fills with noise. The answer quality degrades even though the retrieval metrics look acceptable.
Prevention: monitor not just whether chunks are retrieved, but what they contain. Chunks with very high retrieval frequency and low information density are likely boilerplate. Consider filtering them at the retrieval layer or improving the chunking strategy to exclude them.
Summary
What is RAG system monitoring?
RAG system monitoring tracks the health of retrieval-augmented generation pipelines across four layers: embedding pipeline health (freshness, errors, model versions), retrieval quality (relevance scores, empty retrieval rates, chunk utilization), generation quality (groundedness, answer length, user satisfaction), and system performance (latency, token usage, error rates). The goal is detecting degradation before users notice — which requires tracking trends over time, not just current values.
Key Idea
The most important RAG monitoring metric is one most teams do not track: index freshness. A RAG system can be functionally healthy — low latency, no errors, high availability — while serving answers based on content that is days or weeks out of date. Monitor the lag between source document changes and index updates. If you track nothing else, track freshness.