feat(rag): close eval pipeline with live snapshots

2026-07-06 21:39:27 +08:00
parent cf3333d607
commit ed7efc58b7
47 changed files with 2613 additions and 177 deletions
@@ -0,0 +1,87 @@
+# Evidence: rag-eval-pipeline-closure
+
+## Changed Assets
+
+- `scripts/eval_rag_retrieval.py`
+- `eval/rag-retrieval/cases/golden-cases.json`
+- `eval/rag-retrieval/fixtures/*.json`
+- `eval/rag-retrieval/reports/baseline.json`
+- `eval/rag-retrieval/reports/baseline.md`
+- `eval/rag-retrieval/README.md`
+- `mvp/architecture/rag-eval-closure.md`
+- `src/test/java/com/superbiz/agent/eval/RagLookupSnapshotGeneratorTest.java`
+- `src/test/java/com/superbiz/agent/eval/RagEvalSeedImporterTest.java`
+- `scripts/generate_rag_lookup_snapshots.ps1`
+- `scripts/prepare_rag_eval_seed.ps1`
+- `eval/rag-retrieval/seed-docs/*.md`
+- `src/main/java/com/superbiz/agent/dto/Frontmatter.java`
+- `src/main/java/com/superbiz/agent/dto/KnowledgeEntry.java`
+- `src/main/java/com/superbiz/agent/service/FrontmatterParser.java`
+- `src/main/java/com/superbiz/agent/service/KnowledgeIndexService.java`
+- `src/main/java/com/superbiz/agent/service/DocumentManagementService.java`
+- `src/main/java/com/superbiz/agent/service/VectorIndexService.java`
+- `src/main/java/com/superbiz/agent/service/VectorSearchService.java`
+- `src/main/java/com/superbiz/agent/service/SpringAiVectorStoreSidecarService.java`
+- `src/main/resources/application.yml`
+
+## Baseline Result
+
+```text
+Evaluated 7 cases: passRate=1.0, recall@5=1.0, failed=0
+```
+
+## Regression Signals
+
+The evaluator now fails on:
+
+- missing expected source
+- missing breadcrumb or evidence keyword
+- non-`lookupResult` fixture
+- selected attempt mismatch
+- fallback reason mismatch
+- fallback reason outside accepted values
+- evidence status mismatch
+- missing context source
+- rerank top source mismatch
+
+The snapshot generator now provides:
+
+- real `LookupKnowledgeTool` invocation
+- one fixture per golden case
+- explicit opt-in through `rag.snapshot.enabled=true`
+- optional post-generation baseline evaluation
+- scoped retrieval through `retrieval.kb-scope=rag-eval`
+- Spring AI VectorStore by default through `retrieval.vector-store.mode=spring`
+- scoped L0 hints through the same `retrieval.kb-scope`
+
+The seed importer now provides:
+
+- canonical eval docs under `eval/rag-retrieval/seed-docs`
+- real `DocumentManagementService` import/reindex
+- stable `source`/`docId` metadata
+- `kb_scope=rag-eval` isolation from local non-eval documents
+- frontmatter stripping before chunk embedding
+- an over-filter decoy seed doc for fallback-path evaluation
+
+The diff now detects:
+
+- aggregate pass/recall regression
+- case pass regression
+- hit-level regression
+- first-rank regression
+- selected attempt/fallback/evidence/rerank changes
+
+## Final Spring Live Snapshot
+
+```text
+.\scripts\generate_rag_lookup_snapshots.ps1 -Fixtures <temp-fixtures> -RetrievedAt 2026-07-06T00:00:00Z
+Evaluated 7 cases: passRate=1.0, recall@5=1.0, failed=0
+```
+
+Key fallback trace:
+
+```text
+selectedAttempt=UNFILTERED_VECTOR_RETRY
+fallbackReason=filtered_vector_no_evidence
+rerankTopSource=rag-l0-filter-fallback
+```