feat(harness,rag): dual LLM audit fields, run conclusion, and hybrid quality
Persist provider reasoning and assistant text separately on agent_reasoning_audit (DeepSeekAssistantMessage path), extract diagnosis_run.conclusion, enrich RAG tool audit (step_id/query/qualityScore), gate empty mysql tools, drop devtools, and align MVP docs after live E2E verification.
This commit is contained in:
zhuyongxin
@@ -14,7 +14,7 @@ lookup_knowledge 同文档多 chunk 在后处理与投影阶段被 source 级去
|
||||
|
||||
## 范围
|
||||
|
||||
Delivery 1 only（见 `docs/milvus-hybrid-search-integration-checklist.md` §1.1）。
|
||||
Delivery 1 only（见 `docs/Milvus-Hybrid接入清单.md` §1.1）。
|
||||
|
||||
## 非目标
|
||||
|
||||
|
||||
@@ -93,7 +93,7 @@ Reference files:
|
||||
- `RagResultProjector.java`
|
||||
- `LookupKnowledgeToolTest.java`
|
||||
- `RagResultProjectorTest.java`
|
||||
- `docs/milvus-hybrid-search-integration-checklist.md`
|
||||
- `docs/Milvus-Hybrid接入清单.md`
|
||||
|
||||
Stack notes:
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
## 规格证据
|
||||
|
||||
- 旧 `openspec/specs/rag-knowledge-retrieval` 要求 source 级 dedup（本 change 以 delta 修正）
|
||||
- `docs/milvus-hybrid-search-integration-checklist.md` §1.1 定义 Delivery 1 地基
|
||||
- `docs/Milvus-Hybrid接入清单.md` §1.1 定义 Delivery 1 地基
|
||||
|
||||
## 验证证据
|
||||
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
# Acceptance: rag-eval-hybrid-baseline
|
||||
|
||||
## Tasks
|
||||
|
||||
All tasks in OpenSpec `tasks.md` checked, including apply-discovered 6.x quality-gate fix.
|
||||
|
||||
## 静态验证
|
||||
|
||||
- Snapshot generator path: no required `retrieval.vector-store.mode`.
|
||||
- README documents hybrid generation and offline/live split.
|
||||
|
||||
## 脚本验证
|
||||
|
||||
```text
|
||||
.\scripts\prepare_rag_eval_seed.ps1
|
||||
.\scripts\generate_rag_lookup_snapshots.ps1 -SearchMode hybrid -SkipEval
|
||||
python scripts\eval_rag_retrieval.py --json-report eval/rag-retrieval/reports/baseline.json --markdown-report eval/rag-retrieval/reports/baseline.md
|
||||
# Result: Evaluated 7 cases: passRate=1.0, recall@5=1.0, failed=0
|
||||
|
||||
mvn -Dtest=RetrievalScoreNormalizerTest,KnowledgeEvidencePostProcessorTest,LookupKnowledgeToolTest,VectorSearchServiceTest,VectorKnowledgeSearchAdapterHybridTest test
|
||||
# exit 0
|
||||
```
|
||||
|
||||
Fixture sample meta: `searchMode=hybrid`, `kbScope=rag-eval`.
|
||||
Fallback case: `UNFILTERED_VECTOR_RETRY` + `filtered_vector_low_quality`.
|
||||
|
||||
## 浏览器/人工
|
||||
|
||||
- 未做 UI 验证。
|
||||
|
||||
## 未验证 / 后续
|
||||
|
||||
- Dense vs hybrid dual-directory comparison report (knife-2).
|
||||
- CI wiring of offline eval as required gate (optional process).
|
||||
- Long-term calibration of hybrid PRECISE distribution under denseDistance quality.
|
||||
|
||||
## Specs
|
||||
|
||||
Main spec synced: `openspec/specs/rag-eval-offline-baseline/spec.md`.
|
||||
@@ -0,0 +1,16 @@
|
||||
# Brief: rag-eval-hybrid-baseline
|
||||
|
||||
## Background
|
||||
|
||||
Offline RAG eval (golden × fixture × key-field baseline) existed but generator/docs still used dead `retrieval.vector-store.mode=spring`. Fixtures lacked search meta and did not reflect hybrid main path.
|
||||
|
||||
## Goals (knife-1 only)
|
||||
|
||||
- Snapshot generation uses `retrieval.search.mode` (default hybrid; dense override).
|
||||
- Fixtures record `searchMode` / `kbScope`.
|
||||
- README documents hybrid-era offline vs live loop.
|
||||
- Best-effort live seed + regenerate fixtures + update baseline.
|
||||
|
||||
## Non-goals
|
||||
|
||||
Dense/hybrid dual fixture trees; golden mustNot/chunk/level hard gates; new eval frameworks.
|
||||
@@ -0,0 +1,18 @@
|
||||
# Decisions: rag-eval-hybrid-baseline（最终版）
|
||||
|
||||
## Process
|
||||
|
||||
sm-flow standard-lean: Discover → Commit → Apply → Archive.
|
||||
|
||||
## Key decisions
|
||||
|
||||
1. Replace eval generator `vector-store.mode` with `retrieval.search.mode` (default hybrid).
|
||||
2. Fixture meta: `searchMode`, `kbScope` when set.
|
||||
3. Knife-2 (dual fixtures / mustNot golden) deferred.
|
||||
4. Live refresh succeeded in apply env; baseline updated to hybrid snapshots.
|
||||
5. **Quality gate refinement (apply-found):** hybrid absolute quality for `isLowQuality` / relevance uses optional dense L2 (`denseDistance`); does not overwrite hybrid scoreLabel or RRF order. Rank mapping remains fallback when dense missing.
|
||||
|
||||
## Trade-offs
|
||||
|
||||
- Extra dense ANN on hybrid path for gate calibration (latency) vs correct filter-fallback behavior.
|
||||
- relevance_level still not a hard golden assertion (ordinal vs absolute mix).
|
||||
@@ -0,0 +1,17 @@
|
||||
# Evidence: rag-eval-hybrid-baseline
|
||||
|
||||
## Pre-change
|
||||
|
||||
- `generate_rag_lookup_snapshots.ps1` passed `-Dretrieval.vector-store.mode=spring`.
|
||||
- Fixtures had `caseId/query/retrievedAt/lookupResult` only.
|
||||
- Offline eval already supported Hit levels, recall@K, baseline diff.
|
||||
|
||||
## User decisions
|
||||
|
||||
- Scope: knife-1 only (no dual fixture dirs).
|
||||
- Acceptance: wiring required; fixture refresh best-effort (env allowed full refresh).
|
||||
|
||||
## Apply-discovered
|
||||
|
||||
- After hybrid refresh, `chat-l0-filter-fallback` failed: pure rank→quality made topSimilarity=1.0 on decoy-only filtered hits → no unfiltered retry.
|
||||
- Fix: optional `denseDistance` on hybrid hits; quality gate uses L2 when present; sort order remains RRF.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Acceptance: rag-quality-score-unify
|
||||
|
||||
## Tasks
|
||||
|
||||
OpenSpec `tasks.md` 全部 `[x]`（1.1–6.2）。
|
||||
|
||||
## 静态验证
|
||||
|
||||
- 生产路径 grep：无 `bm25_only_no_dense` 发射、无 hybrid L2 enrichment（仅 Labels canonicalize 兼容旧串）。
|
||||
- 架构文档 §6 与 `application.yml` 注释已对齐 quality 契约。
|
||||
|
||||
## 脚本验证
|
||||
|
||||
```text
|
||||
mvn -Dtest=RetrievalScoreNormalizerTest,KnowledgeEvidencePostProcessorTest,LookupKnowledgeToolTest,VectorSearchServiceTest,VectorKnowledgeSearchAdapterHybridTest test
|
||||
```
|
||||
|
||||
| 套件 | 结果 |
|
||||
|---|---|
|
||||
| RetrievalScoreNormalizerTest | 4 passed |
|
||||
| KnowledgeEvidencePostProcessorTest | 6 passed |
|
||||
| LookupKnowledgeToolTest | 7 passed |
|
||||
| VectorSearchServiceTest | 2 passed |
|
||||
| VectorKnowledgeSearchAdapterHybridTest | 1 passed |
|
||||
|
||||
（PowerShell 可能将 JVM warning 标为 exit 1；日志中为 BUILD SUCCESS / Failures: 0。）
|
||||
|
||||
## 浏览器 / 人工验证
|
||||
|
||||
- 未跑：live `lookup_knowledge` hybrid vs dense 对照、生产阈值标定。
|
||||
|
||||
## 未验证
|
||||
|
||||
| 项 | 风险 | 建议 |
|
||||
|---|---|---|
|
||||
| 真实 Milvus hybrid 联调 | 序/质量分布与单测 mock 有差 | 启动服务后固定 query 集切 mode 对比 |
|
||||
| 阈值 0.75/0.5 在 hybrid rank 分下的标定 | retry/PRECISE 偏多或偏少 | 看 trace topSimilarity 再调 yml |
|
||||
|
||||
## Specs 同步
|
||||
|
||||
- 主规格新增：`openspec/specs/rag-retrieval-quality-score/spec.md`（archive 时从 delta 同步）。
|
||||
@@ -0,0 +1,21 @@
|
||||
# Brief: rag-quality-score-unify
|
||||
|
||||
## Background
|
||||
|
||||
真 BM25 hybrid（dense + BM25 + RRF）已上线，但后处理仍把 hybrid 结果伪装成 L2 做 `normalizeL2`，并用 L0 domain/entity/keyword contains 加分改序。排序权威与质量闸门分裂，词面信号被 BM25 与后处理双重计分。
|
||||
|
||||
## Goals
|
||||
|
||||
- 一级 `scoreLabel` 仅 `dense` | `hybrid`
|
||||
- 唯一 `toQualityScore`；后处理 label-agnostic
|
||||
- 排序主序 = 检索 `originalRank`；去掉关键词 boost 改序
|
||||
- hybrid quality = 本轮 rank 纯映射（不做 max(rank, denseSim)、不为闸门回填 L2）
|
||||
- 保留 `mode=dense` 作同库召回对照；线上默认 hybrid
|
||||
|
||||
## Scope
|
||||
|
||||
内部 RAG：store 发射、normalizer、evidence post-process、单测、架构文档 §6。
|
||||
|
||||
## Non-goals
|
||||
|
||||
精排 / query rewrite / 邻块、schema rebuild、改 Agent ACI 字段名、删除 dense 对照 mode。
|
||||
@@ -0,0 +1,25 @@
|
||||
# Decisions: rag-quality-score-unify（最终版）
|
||||
|
||||
## Scale / process
|
||||
|
||||
- sm-flow standard：Discover → Commit → Apply → Archive
|
||||
- Committed OpenSpec：`openspec/changes/rag-quality-score-unify/`（归档后见 archive 目录）
|
||||
- 废止：`rag-bm25-hybrid-drop-sdk` 中「dense L2 enrichment for threshold compatibility」
|
||||
|
||||
## Key decisions
|
||||
|
||||
1. **Label**：仅 `dense` | `hybrid`；旧别名 canonicalize。
|
||||
2. **Normalizer**：唯一 `toQualityScore`；dense=L2 公式；hybrid=rank 线性映射（batchSize）。
|
||||
3. **Store**：hybrid 不回填 L2、不发 `bm25_only_*`；返回序即 RRF 序。
|
||||
4. **Post-process**：`originalRank` ASC；L0 重叠只写 hitReasons；relevance/low-quality 只看 qualityScore；PRECISE 不要求 hint support。
|
||||
5. **Mode**：hybrid 主路径；dense 同库对照（架构 §6.0）。
|
||||
|
||||
## Trade-offs
|
||||
|
||||
- hybrid quality 为序数分，跨 query 绝对值不可比；阈值可能需后续标定。
|
||||
- 去掉 boost 改序后，「词面热语义冷」不再被后处理抬升；词面交给 BM25+RRF。
|
||||
|
||||
## Risks accepted
|
||||
|
||||
- `relevance_level` / unfiltered retry 分布变化（产品已接受）。
|
||||
- 未做 live E2E / 人工 hybrid 对照评测（见 acceptance 未验证项）。
|
||||
@@ -0,0 +1,25 @@
|
||||
# Evidence: rag-quality-score-unify
|
||||
|
||||
## Code (pre-change)
|
||||
|
||||
- `MilvusHybridKnowledgeStore.searchHybrid`：RRF 后并行 dense 回填 L2；BM25-only → `bm25_only_no_dense` + maxL2。
|
||||
- `KnowledgeEvidencePostProcessor`：一律 `normalizeL2(score)` + domain/entity/keyword/source_type 加分，按 `finalScore` 降序；PRECISE 需 `hasHintSupport`。
|
||||
|
||||
## User decisions (grill)
|
||||
|
||||
| ID | 结论 |
|
||||
|---|---|
|
||||
| Q1 | 一级 label 仅 dense/hybrid；bm25_only 不作正式 label |
|
||||
| Q2 | 后处理去掉 contains 加分改序，保 originalRank |
|
||||
| Q3 | 唯一 toQualityScore；后处理统一 |
|
||||
| Q4 | dense mode 保留作对照 |
|
||||
| Q7 | 接受 relevance_level / retry 分布变化 |
|
||||
| Q8 | hybrid quality = **纯 rank 映射** |
|
||||
|
||||
## Post-change anchors
|
||||
|
||||
- `RetrievalScoreLabels` / `RetrievalScoreNormalizer`
|
||||
- `MilvusHybridKnowledgeStore`（无 L2 overwrite / 无 bm25_only 发射）
|
||||
- `KnowledgeEvidencePostProcessor`（rank sort + explain-only L0 overlap）
|
||||
- OpenSpec delta：`rag-retrieval-quality-score`
|
||||
- 架构：`mvp/architecture/RAG知识检索架构.md` §6
|
||||
Reference in New Issue
Block a user