# Evidence: rag-eval-pipeline-closure

## Changed Assets

- `scripts/eval_rag_retrieval.py`
- `eval/rag-retrieval/cases/golden-cases.json`
- `eval/rag-retrieval/fixtures/*.json`
- `eval/rag-retrieval/reports/baseline.json`
- `eval/rag-retrieval/reports/baseline.md`
- `eval/rag-retrieval/README.md`
- `mvp/architecture/rag-eval-closure.md`
- `src/test/java/com/superbiz/agent/eval/RagLookupSnapshotGeneratorTest.java`
- `src/test/java/com/superbiz/agent/eval/RagEvalSeedImporterTest.java`
- `scripts/generate_rag_lookup_snapshots.ps1`
- `scripts/prepare_rag_eval_seed.ps1`
- `eval/rag-retrieval/seed-docs/*.md`
- `src/main/java/com/superbiz/agent/dto/Frontmatter.java`
- `src/main/java/com/superbiz/agent/dto/KnowledgeEntry.java`
- `src/main/java/com/superbiz/agent/service/FrontmatterParser.java`
- `src/main/java/com/superbiz/agent/service/KnowledgeIndexService.java`
- `src/main/java/com/superbiz/agent/service/DocumentManagementService.java`
- `src/main/java/com/superbiz/agent/service/VectorIndexService.java`
- `src/main/java/com/superbiz/agent/service/VectorSearchService.java`
- `src/main/java/com/superbiz/agent/service/SpringAiVectorStoreSidecarService.java`
- `src/main/resources/application.yml`

## Baseline Result

```text
Evaluated 7 cases: passRate=1.0, recall@5=1.0, failed=0
```

## Regression Signals

The evaluator now fails on:

- missing expected source
- missing breadcrumb or evidence keyword
- non-`lookupResult` fixture
- selected attempt mismatch
- fallback reason mismatch
- fallback reason outside accepted values
- evidence status mismatch
- missing context source
- rerank top source mismatch

The snapshot generator now provides:

- real `LookupKnowledgeTool` invocation
- one fixture per golden case
- explicit opt-in through `rag.snapshot.enabled=true`
- optional post-generation baseline evaluation
- scoped retrieval through `retrieval.kb-scope=rag-eval`
- Spring AI VectorStore by default through `retrieval.vector-store.mode=spring`
- scoped L0 hints through the same `retrieval.kb-scope`

The seed importer now provides:

- canonical eval docs under `eval/rag-retrieval/seed-docs`
- real `DocumentManagementService` import/reindex
- stable `source`/`docId` metadata
- `kb_scope=rag-eval` isolation from local non-eval documents
- frontmatter stripping before chunk embedding
- an over-filter decoy seed doc for fallback-path evaluation

The diff now detects:

- aggregate pass/recall regression
- case pass regression
- hit-level regression
- first-rank regression
- selected attempt/fallback/evidence/rerank changes

## Final Spring Live Snapshot

```text
.\scripts\generate_rag_lookup_snapshots.ps1 -Fixtures <temp-fixtures> -RetrievedAt 2026-07-06T00:00:00Z
Evaluated 7 cases: passRate=1.0, recall@5=1.0, failed=0
```

Key fallback trace:

```text
selectedAttempt=UNFILTERED_VECTOR_RETRY
fallbackReason=filtered_vector_no_evidence
rerankTopSource=rag-l0-filter-fallback
```