feat(rag): chunk evidence identity, dedup, and search port

Preserve same-document multi-chunk evidence with evidenceKey identity, per-document caps, retrieve-k/return-n split, and a dense KnowledgeSearchPort. Archives Delivery 1 OpenSpec change as the foundation for hybrid retrieval.
2026-07-27 18:26:15 +08:00
parent 99d4f6f216
commit ac1f831903
34 changed files with 2880 additions and 298 deletions
@@ -11,6 +11,7 @@

 | 日期 | slug | 说明 | 领域 | 关键词 | 关联 OpenSpec | 状态 |
 |---|---|---|---|---|---|---|
+| 2026-07-27 | rag-chunk-evidence-identity-dedup | chunk 级证据身份、去重、retrieve-k/return-n 与 SearchPort 地基，为 hybrid 铺路。 | RAG/证据身份/去重 | evidenceKey, maxChunksPerDocument, retrieve-k, return-n, KnowledgeSearchPort, document_id chunk-scoped | openspec/changes/rag-chunk-evidence-identity-dedup | accepted-unarchived |
 | 2026-07-21 | single-react-tool-invocation-store | 建立统一 ToolBoundary 与 Redis canonical invocation store，集中生命周期、证据状态、TTL、容量和 Run 所有权。 | Harness/Tool boundary/Canonical store | ISS-014, ToolBoundary, canonical invocation, PROJECTING, READY, ERROR, TTL, RESULT_TOO_LARGE | openspec/changes/archive/2026-07-21-single-react-tool-invocation-store | archived |
 | 2026-07-21 | single-react-harness-run-context | 建立显式 RunContext、Harness Core、预算、取消、类型化重试和 Tool Store 基础。 | Harness/Run lifecycle/Budget | ISS-014, RunContext, deadline, cancellation, budget, retry, ToolCallKey | openspec/changes/archive/2026-07-21-single-react-harness-run-context | archived |
 | 2026-07-21 | single-react-aci-tool-contracts | 冻结 RAG、日志和 MySQL evidence Tool 的 Agent-facing ACI Schema、状态、框架调用引用和描述边界。 | Harness/Agent Tool contract | ISS-014, ACI, tool_call_id, evidence_status, RAG, query_logs, query_mysql, MOCK | openspec/changes/archive/2026-07-21-single-react-aci-tool-contracts | archived |
@@ -0,0 +1,44 @@
+# Acceptance: rag-chunk-evidence-identity-dedup
+
+## 实现结果
+
+Delivery 1 completed:
+
+- chunk identity fields on candidates/evidence blocks
+- `KnowledgeSearchPort` + dense adapter
+- evidenceKey dedup + maxChunksPerDocument + return-n
+- retrieve-k on lookup tool
+- projector keeps same-source distinct chunks; `document_id` is chunk-scoped
+
+## 验证
+
+### 脚本验证
+
+```text
+mvn -q "-Dtest=LookupKnowledgeToolTest,KnowledgeEvidencePostProcessorTest,RagResultProjectorTest" test
+```
+
+结果：通过（exit 0）
+
+### 静态验证
+
+- Search/port wiring reviewed against OpenSpec tasks
+- No new legacy SDK dependency added
+
+### 浏览器/人工验证
+
+未运行（纯检索契约变更，无 UI）
+
+### 未验证
+
+- 全量 harness E2E / 真实 Milvus 联调（Delivery 2 前可补）
+- 生产配置默认 retrieve-k/return-n 调优
+
+## 归档状态
+
+- OpenSpec change ready to archive
+- User pre-authorized archive for sm-flow staged delivery
+
+## 后续
+
+- Delivery 2: milvus hybrid search (separate change)
@@ -0,0 +1,29 @@
+# Brief: rag-chunk-evidence-identity-dedup
+
+## 背景
+
+lookup_knowledge 同文档多 chunk 在后处理与投影阶段被 source 级去重吞掉。Hybrid 多路召回前必须先修证据身份与裁剪契约。
+
+## 目标
+
+- chunk 级 evidenceKey 身份
+- 按 evidenceKey 去重 + 每文档 chunk 上限
+- retrieve-k / return-n 分离
+- Projector 保留同 source 不同 chunk
+- 薄 KnowledgeSearchPort，为 Delivery 2 hybrid 铺路
+
+## 范围
+
+Delivery 1 only（见 `docs/milvus-hybrid-search-integration-checklist.md` §1.1）。
+
+## 非目标
+
+hybrid schema、BM25、删 SDK、session dedup、邻块重建、模型 rerank。
+
+## 分档
+
+standard
+
+## 关联 OpenSpec
+
+`openspec/changes/rag-chunk-evidence-identity-dedup`
@@ -0,0 +1,102 @@
+# Decisions: rag-chunk-evidence-identity-dedup
+
+## Capability sources
+
+- sm-flow orchestration
+- OpenSpec fallback protocol (file-based propose/apply/archive) — external openspec-propose/apply skills used as reference; execution via sm-flow fallback
+- grill: fallback built-in protocol
+- audit: fallback built-in protocol
+
+## Scale
+
+standard
+
+## Clarify
+
+- Problem: same-document multi-chunk evidence collapsed by source-level dedup.
+- Outcome: Delivery 1 foundation before hybrid Delivery 2.
+- Slug: `rag-chunk-evidence-identity-dedup`
+- User authorized apply + archive in advance for sm-flow staged changes.
+
+## Context
+
+- Read: `devflow/glossary/CONTEXT.md`, modular-rag-pipeline brief, `openspec/specs/rag-knowledge-retrieval`, `rag-log-projections`, checklist doc §1.1
+- Constraints into OpenSpec:
+  - L0 hint-only remains
+  - Do not thicken legacy SDK path
+  - Agent tool name/input stable
+  - Hybrid out of scope this change
+
+## Question pool (grill)
+
+| # | Dimension | Mode | Question | Status |
+|---|---|---|---|---|
+| Q1 | 术语 | evidence-driven | evidenceKey / document_id 语义？ | Resolved: evidenceKey=chunk id; projected document_id=evidenceKey |
+| Q2 | 边界 | evidence-driven | Delivery 1 vs 2 边界？ | Resolved: per checklist; no schema/hybrid/SDK delete |
+| Q3 | 验收 | evidence-driven | 如何验收多 chunk？ | Resolved: unit tests multi-chunk keep + projector |
+| Q4 | 接口 | user-interview | document_id 改为 chunk 级是否可接受？ | **Pre-authorized by user** via “apply/archive 直接授权” + prior design agreement on scheme A (document_id=evidenceKey). Recorded as accepted behavior change. |
+| Q5 | 技术 | evidence-driven | SearchPort 是否本 change 必须？ | Resolved: thin port required as foundation |
+
+### Evidence-driven conclusions (reported)
+
+1. Current collapse points: `KnowledgeEvidencePostProcessor.sourceKey` and `RagResultProjector` source fallback.
+2. Metadata already has docId/chunkIndex on write path; not first-class on read path.
+3. Existing main-spec still says source-level dedup — this change intentionally deltas that requirement.
+
+### User-interview
+
+- Q4 accepted under prior design alignment (scheme A) and explicit apply authorization for this sm-flow run. No remaining open product preference questions for Delivery 1.
+
+## Audit
+
+Module chain:
+
+```text
+LookupKnowledgeTool -> SearchPort -> Retriever -> PostProcessor -> Packer -> Assembler -> RagResultProjector
+```
+
+Risks:
+
+1. Agent payload growth — mitigated by return-n + maxChunksPerDocument + projector budgets.
+2. document_id semantic shift — documented L3 behavior change; tests updated.
+3. Old data without chunkIndex — vector id fallback.
+
+No ADR conflict with modular RAG L0/L1 boundary.
+
+## Cross-artifact alignment
+
+| From | To | Status |
+|---|---|---|
+| brief goals | proposal | 已对齐 |
+| proposal scope | design decisions | 已对齐 |
+| design identity/dedup/port | specs | 已对齐 |
+| specs scenarios | tasks | 已对齐 |
+
+## Interface impact
+
+- L2 internal DTO
+- L3 Agent `document_id` chunk-scoped
+
+## Commit gate
+
+- proposal/design/specs/tasks present
+- no open user-interview blockers for Delivery 1
+- apply authorized by user at sm-flow start
+
+## Pre-apply research
+
+Reference files:
+
+- `LookupKnowledgeTool.java`
+- `KnowledgeDocumentRetriever.java`
+- `KnowledgeEvidencePostProcessor.java`
+- `RagResultProjector.java`
+- `LookupKnowledgeToolTest.java`
+- `RagResultProjectorTest.java`
+- `docs/milvus-hybrid-search-integration-checklist.md`
+
+Stack notes:
+
+- No MQ/request envelope changes
+- Spring `@Value` config pattern for rag.* keys
+- Tests use ReflectionTestUtils + Mockito
@@ -0,0 +1,16 @@
+# Evidence: rag-chunk-evidence-identity-dedup
+
+## 代码证据（变更前）
+
+- `KnowledgeEvidencePostProcessor.sourceKey` 使用 source/title 去重
+- `RagResultProjector` 用 source 回退 document_id 并 HashSet 去重
+- 写入路径 metadata 已有 docId/chunkIndex，读路径未一等化
+
+## 规格证据
+
+- 旧 `openspec/specs/rag-knowledge-retrieval` 要求 source 级 dedup（本 change 以 delta 修正）
+- `docs/milvus-hybrid-search-integration-checklist.md` §1.1 定义 Delivery 1 地基
+
+## 验证证据
+
+- 单测覆盖 multi-chunk keep / true-dup merge / maxChunksPerDocument / projector same-source multi-chunk