rag-auditor

RAG Auditor

Systematic RAG pipeline evaluation across the full retrieval-generation chain: designs evaluation query sets, measures retrieval metrics (Precision@K, Recall@K, MRR), evaluates generation quality (groundedness, completeness, hallucination rate), diagnoses component-level failures, and recommends targeted improvements.

Reference Files

Prerequisites

• Access to the RAG pipeline (or its outputs for post-hoc evaluation)
• A set of test queries with known-correct answers
• Understanding of the pipeline components (embedding model, retriever, generator)

Workflow

Phase 1: Pipeline Inventory

Document the RAG pipeline configuration:

1. Document source — What documents are indexed? Format, count, size.
2. Chunking — Strategy (fixed-size, semantic, paragraph), chunk size, overlap.
3. Embedding — Model name and version, dimensionality.
4. Vector store — Type (FAISS, Pinecone, Chroma, pgvector), index type.
5. Retrieval — Method (similarity, hybrid, reranking), top-K parameter.
6. Generation — Model, prompt template, context window usage.

Phase 2: Design Evaluation Queries

Create a diverse set of test queries:

For each query, document the expected answer and the source chunk(s).

Phase 3: Evaluate Retrieval

For each test query, measure:

1. Precision@K — Of the K retrieved chunks, how many are relevant?
2. Recall@K — Of all relevant chunks in the corpus, how many were retrieved?
3. MRR (Mean Reciprocal Rank) — How high is the first relevant chunk ranked?
4. Chunk relevance — Score each retrieved chunk: Relevant, Partially Relevant, Irrelevant.

Phase 4: Evaluate Generation

For each test query with retrieved context:

1. Groundedness — Is every claim in the response supported by the retrieved context? Score: 0 (hallucinated) to 1 (fully grounded).
2. Completeness — Does the response use all relevant information from the context? Score: 0 (ignored context) to 1 (complete).
3. Hallucination detection — Identify specific claims not supported by context.
4. Abstention — For unanswerable queries, does the model correctly say "I don't know"?

Phase 5: Diagnose Failures

For every incorrect or low-quality response, classify the root cause:

Phase 6: Recommendations

Based on failure analysis, recommend specific improvements:

Output Format

## RAG Audit Report

### Pipeline Configuration
| Component | Value |
|-----------|-------|
| Documents | {N} ({format}) |
| Chunking | {strategy}, {size} tokens, {overlap}% overlap |
| Embedding | {model} ({dimensions}d) |
| Retrieval | {method}, K={N} |
| Generation | {model}, temperature={T} |

### Evaluation Dataset
- **Total queries:** {N}
- **Known-answer:** {N}
- **Multi-hop:** {N}
- **Unanswerable:** {N}

### Retrieval Quality

| Metric | Score | Target | Status |
|--------|-------|--------|--------|
| Precision@{K} | {score} | {target} | {Pass/Fail} |
| Recall@{K} | {score} | {target} | {Pass/Fail} |
| MRR | {score} | {target} | {Pass/Fail} |

### Generation Quality

| Metric | Score | Target | Status |
|--------|-------|--------|--------|
| Groundedness | {score} | {target} | {Pass/Fail} |
| Completeness | {score} | {target} | {Pass/Fail} |
| Hallucination rate | {score} | {target} | {Pass/Fail} |
| Abstention accuracy | {score} | {target} | {Pass/Fail} |

### Failure Analysis

| # | Query | Failure Type | Root Cause | Recommendation |
|---|-------|-------------|------------|----------------|
| 1 | {query} | {type} | {cause} | {fix} |

### Recommendations (Priority Order)
1. **{Recommendation}** — addresses {N} failures, expected impact: {description}
2. **{Recommendation}** — addresses {N} failures, expected impact: {description}

### Sample Failures

#### Query: "{query}"
- **Expected:** {answer}
- **Retrieved chunks:** {chunk summaries with relevance scores}
- **Generated:** {response}
- **Issue:** {diagnosis}

Calibration Rules

1. Component isolation. Evaluate retrieval and generation independently. A great retriever with a bad generator looks like retrieval failure if you only check end output.
2. Known answers first. Start with factoid questions where the correct answer is unambiguous. Multi-hop and ambiguous queries are harder to evaluate.
3. Quantify, don't qualify. "Retrieval is bad" is not a finding. "Precision@5 is 0.3 (target: 0.8) with 70% of failures due to chunk boundary splits" is actionable.
4. Sample failures deeply. Aggregate metrics identify WHERE the problem is. Individual failure analysis identifies WHY.

Error Handling

When NOT to Audit

Push back if:

• The pipeline hasn't been built yet — design it first, audit after
• The corpus has fewer than 10 documents — too small for meaningful retrieval evaluation
• The user wants to compare embedding models — that's a benchmark task, not an audit