SuperBizAgent-java/devflow/projects/2026-07-06-rag-eval-pipeline-closure/evidence.md

Evidence: rag-eval-pipeline-closure

Changed Assets

• scripts/eval_rag_retrieval.py
• eval/rag-retrieval/cases/golden-cases.json
• eval/rag-retrieval/fixtures/*.json
• eval/rag-retrieval/reports/baseline.json
• eval/rag-retrieval/reports/baseline.md
• eval/rag-retrieval/README.md
• mvp/architecture/rag-eval-closure.md
• src/test/java/com/superbiz/agent/eval/RagLookupSnapshotGeneratorTest.java
• src/test/java/com/superbiz/agent/eval/RagEvalSeedImporterTest.java
• scripts/generate_rag_lookup_snapshots.ps1
• scripts/prepare_rag_eval_seed.ps1
• eval/rag-retrieval/seed-docs/*.md
• src/main/java/com/superbiz/agent/dto/Frontmatter.java
• src/main/java/com/superbiz/agent/dto/KnowledgeEntry.java
• src/main/java/com/superbiz/agent/service/FrontmatterParser.java
• src/main/java/com/superbiz/agent/service/KnowledgeIndexService.java
• src/main/java/com/superbiz/agent/service/DocumentManagementService.java
• src/main/java/com/superbiz/agent/service/VectorIndexService.java
• src/main/java/com/superbiz/agent/service/VectorSearchService.java
• src/main/java/com/superbiz/agent/service/SpringAiVectorStoreSidecarService.java
• src/main/resources/application.yml

Baseline Result

Evaluated 7 cases: passRate=1.0, recall@5=1.0, failed=0

Regression Signals

The evaluator now fails on:

• missing expected source
• missing breadcrumb or evidence keyword
• non-lookupResult fixture
• selected attempt mismatch
• fallback reason mismatch
• fallback reason outside accepted values
• evidence status mismatch
• missing context source
• rerank top source mismatch

The snapshot generator now provides:

• real LookupKnowledgeTool invocation
• one fixture per golden case
• explicit opt-in through rag.snapshot.enabled=true
• optional post-generation baseline evaluation
• scoped retrieval through retrieval.kb-scope=rag-eval
• Spring AI VectorStore by default through retrieval.vector-store.mode=spring
• scoped L0 hints through the same retrieval.kb-scope

The seed importer now provides:

• canonical eval docs under eval/rag-retrieval/seed-docs
• real DocumentManagementService import/reindex
• stable source/docId metadata
• kb_scope=rag-eval isolation from local non-eval documents
• frontmatter stripping before chunk embedding
• an over-filter decoy seed doc for fallback-path evaluation

The diff now detects:

• aggregate pass/recall regression
• case pass regression
• hit-level regression
• first-rank regression
• selected attempt/fallback/evidence/rerank changes

Final Spring Live Snapshot

.\scripts\generate_rag_lookup_snapshots.ps1 -Fixtures <temp-fixtures> -RetrievedAt 2026-07-06T00:00:00Z
Evaluated 7 cases: passRate=1.0, recall@5=1.0, failed=0

Key fallback trace:

selectedAttempt=UNFILTERED_VECTOR_RETRY
fallbackReason=filtered_vector_no_evidence
rerankTopSource=rag-l0-filter-fallback