88 lines
2.8 KiB
Markdown
88 lines
2.8 KiB
Markdown
# Evidence: rag-eval-pipeline-closure
|
|
|
|
## Changed Assets
|
|
|
|
- `scripts/eval_rag_retrieval.py`
|
|
- `eval/rag-retrieval/cases/golden-cases.json`
|
|
- `eval/rag-retrieval/fixtures/*.json`
|
|
- `eval/rag-retrieval/reports/baseline.json`
|
|
- `eval/rag-retrieval/reports/baseline.md`
|
|
- `eval/rag-retrieval/README.md`
|
|
- `mvp/architecture/rag-eval-closure.md`
|
|
- `src/test/java/com/superbiz/agent/eval/RagLookupSnapshotGeneratorTest.java`
|
|
- `src/test/java/com/superbiz/agent/eval/RagEvalSeedImporterTest.java`
|
|
- `scripts/generate_rag_lookup_snapshots.ps1`
|
|
- `scripts/prepare_rag_eval_seed.ps1`
|
|
- `eval/rag-retrieval/seed-docs/*.md`
|
|
- `src/main/java/com/superbiz/agent/dto/Frontmatter.java`
|
|
- `src/main/java/com/superbiz/agent/dto/KnowledgeEntry.java`
|
|
- `src/main/java/com/superbiz/agent/service/FrontmatterParser.java`
|
|
- `src/main/java/com/superbiz/agent/service/KnowledgeIndexService.java`
|
|
- `src/main/java/com/superbiz/agent/service/DocumentManagementService.java`
|
|
- `src/main/java/com/superbiz/agent/service/VectorIndexService.java`
|
|
- `src/main/java/com/superbiz/agent/service/VectorSearchService.java`
|
|
- `src/main/java/com/superbiz/agent/service/SpringAiVectorStoreSidecarService.java`
|
|
- `src/main/resources/application.yml`
|
|
|
|
## Baseline Result
|
|
|
|
```text
|
|
Evaluated 7 cases: passRate=1.0, recall@5=1.0, failed=0
|
|
```
|
|
|
|
## Regression Signals
|
|
|
|
The evaluator now fails on:
|
|
|
|
- missing expected source
|
|
- missing breadcrumb or evidence keyword
|
|
- non-`lookupResult` fixture
|
|
- selected attempt mismatch
|
|
- fallback reason mismatch
|
|
- fallback reason outside accepted values
|
|
- evidence status mismatch
|
|
- missing context source
|
|
- rerank top source mismatch
|
|
|
|
The snapshot generator now provides:
|
|
|
|
- real `LookupKnowledgeTool` invocation
|
|
- one fixture per golden case
|
|
- explicit opt-in through `rag.snapshot.enabled=true`
|
|
- optional post-generation baseline evaluation
|
|
- scoped retrieval through `retrieval.kb-scope=rag-eval`
|
|
- Spring AI VectorStore by default through `retrieval.vector-store.mode=spring`
|
|
- scoped L0 hints through the same `retrieval.kb-scope`
|
|
|
|
The seed importer now provides:
|
|
|
|
- canonical eval docs under `eval/rag-retrieval/seed-docs`
|
|
- real `DocumentManagementService` import/reindex
|
|
- stable `source`/`docId` metadata
|
|
- `kb_scope=rag-eval` isolation from local non-eval documents
|
|
- frontmatter stripping before chunk embedding
|
|
- an over-filter decoy seed doc for fallback-path evaluation
|
|
|
|
The diff now detects:
|
|
|
|
- aggregate pass/recall regression
|
|
- case pass regression
|
|
- hit-level regression
|
|
- first-rank regression
|
|
- selected attempt/fallback/evidence/rerank changes
|
|
|
|
## Final Spring Live Snapshot
|
|
|
|
```text
|
|
.\scripts\generate_rag_lookup_snapshots.ps1 -Fixtures <temp-fixtures> -RetrievedAt 2026-07-06T00:00:00Z
|
|
Evaluated 7 cases: passRate=1.0, recall@5=1.0, failed=0
|
|
```
|
|
|
|
Key fallback trace:
|
|
|
|
```text
|
|
selectedAttempt=UNFILTERED_VECTOR_RETRY
|
|
fallbackReason=filtered_vector_no_evidence
|
|
rerankTopSource=rag-l0-filter-fallback
|
|
```
|