zhuyongxin
7ae9707a3b
Persist provider reasoning and assistant text separately on agent_reasoning_audit (DeepSeekAssistantMessage path), extract diagnosis_run.conclusion, enrich RAG tool audit (step_id/query/qualityScore), gate empty mysql tools, drop devtools, and align MVP docs after live E2E verification.
35 lines
1.8 KiB
Markdown
35 lines
1.8 KiB
Markdown
# Tasks: rag-eval-hybrid-baseline
|
|
|
|
## 1. Generator wiring
|
|
|
|
- [x] 1.1 Update `scripts/generate_rag_lookup_snapshots.ps1`: replace `VectorStoreMode` / `vector-store.mode` with `SearchMode` default `hybrid` and `-Dretrieval.search.mode=...`
|
|
- [x] 1.2 Keep `-Dretrieval.kb-scope` (default `rag-eval`); document `-SearchMode dense` override
|
|
- [x] 1.3 Ensure snapshot test picks up `retrieval.search.mode` (system property and/or `@SpringBootTest` properties)
|
|
|
|
## 2. Fixture meta
|
|
|
|
- [x] 2.1 `RagLookupSnapshotGeneratorTest` writes `searchMode` and `kbScope` (when set) on each fixture
|
|
- [x] 2.2 Confirm offline `eval_rag_retrieval.py` still loads fixtures (ignore extra meta)
|
|
|
|
## 3. Docs
|
|
|
|
- [x] 3.1 Rewrite `eval/rag-retrieval/README.md` hybrid-era loop; remove spring vector-store as default generation path
|
|
- [x] 3.2 Note offline vs live responsibilities; point to qualityScore/hybrid main path briefly
|
|
|
|
## 4. Refresh attempt (best-effort)
|
|
|
|
- [x] 4.1 Attempt `prepare_rag_eval_seed` + hybrid snapshot generate + offline eval when environment allows
|
|
- [x] 4.2 On success: update fixtures and `reports/baseline.*` if needed after diff review
|
|
- [x] 4.3 On failure: record exact commands, error summary, and “unverified refresh” in change decisions/acceptance notes — do not block wiring delivery
|
|
|
|
## 5. Verify
|
|
|
|
- [x] 5.1 Static: grep shows no required `vector-store.mode` in snapshot generator path
|
|
- [x] 5.2 Offline: `python scripts/eval_rag_retrieval.py` runs on committed fixtures (pass or documented baseline drift)
|
|
|
|
## 6. Apply-discovered fix (hybrid quality gate)
|
|
|
|
- [x] 6.1 Hybrid attaches optional `denseDistance` without overwriting RRF order/scoreLabel
|
|
- [x] 6.2 `toQualityScore(hybrid)` prefers dense L2 for absolute gates; rank fallback if no dense
|
|
- [x] 6.3 Regenerate fixtures; L0 filter fallback case green again; baseline updated
|