feat(rag): chunk evidence identity, dedup, and search port

Preserve same-document multi-chunk evidence with evidenceKey identity, per-document caps, retrieve-k/return-n split, and a dense KnowledgeSearchPort. Archives Delivery 1 OpenSpec change as the foundation for hybrid retrieval.
2026-07-27 18:26:15 +08:00
parent 99d4f6f216
commit ac1f831903
34 changed files with 2880 additions and 298 deletions
@@ -0,0 +1,102 @@
+# Decisions: rag-chunk-evidence-identity-dedup
+
+## Capability sources
+
+- sm-flow orchestration
+- OpenSpec fallback protocol (file-based propose/apply/archive) — external openspec-propose/apply skills used as reference; execution via sm-flow fallback
+- grill: fallback built-in protocol
+- audit: fallback built-in protocol
+
+## Scale
+
+standard
+
+## Clarify
+
+- Problem: same-document multi-chunk evidence collapsed by source-level dedup.
+- Outcome: Delivery 1 foundation before hybrid Delivery 2.
+- Slug: `rag-chunk-evidence-identity-dedup`
+- User authorized apply + archive in advance for sm-flow staged changes.
+
+## Context
+
+- Read: `devflow/glossary/CONTEXT.md`, modular-rag-pipeline brief, `openspec/specs/rag-knowledge-retrieval`, `rag-log-projections`, checklist doc §1.1
+- Constraints into OpenSpec:
+  - L0 hint-only remains
+  - Do not thicken legacy SDK path
+  - Agent tool name/input stable
+  - Hybrid out of scope this change
+
+## Question pool (grill)
+
+| # | Dimension | Mode | Question | Status |
+|---|---|---|---|---|
+| Q1 | 术语 | evidence-driven | evidenceKey / document_id 语义？ | Resolved: evidenceKey=chunk id; projected document_id=evidenceKey |
+| Q2 | 边界 | evidence-driven | Delivery 1 vs 2 边界？ | Resolved: per checklist; no schema/hybrid/SDK delete |
+| Q3 | 验收 | evidence-driven | 如何验收多 chunk？ | Resolved: unit tests multi-chunk keep + projector |
+| Q4 | 接口 | user-interview | document_id 改为 chunk 级是否可接受？ | **Pre-authorized by user** via “apply/archive 直接授权” + prior design agreement on scheme A (document_id=evidenceKey). Recorded as accepted behavior change. |
+| Q5 | 技术 | evidence-driven | SearchPort 是否本 change 必须？ | Resolved: thin port required as foundation |
+
+### Evidence-driven conclusions (reported)
+
+1. Current collapse points: `KnowledgeEvidencePostProcessor.sourceKey` and `RagResultProjector` source fallback.
+2. Metadata already has docId/chunkIndex on write path; not first-class on read path.
+3. Existing main-spec still says source-level dedup — this change intentionally deltas that requirement.
+
+### User-interview
+
+- Q4 accepted under prior design alignment (scheme A) and explicit apply authorization for this sm-flow run. No remaining open product preference questions for Delivery 1.
+
+## Audit
+
+Module chain:
+
+```text
+LookupKnowledgeTool -> SearchPort -> Retriever -> PostProcessor -> Packer -> Assembler -> RagResultProjector
+```
+
+Risks:
+
+1. Agent payload growth — mitigated by return-n + maxChunksPerDocument + projector budgets.
+2. document_id semantic shift — documented L3 behavior change; tests updated.
+3. Old data without chunkIndex — vector id fallback.
+
+No ADR conflict with modular RAG L0/L1 boundary.
+
+## Cross-artifact alignment
+
+| From | To | Status |
+|---|---|---|
+| brief goals | proposal | 已对齐 |
+| proposal scope | design decisions | 已对齐 |
+| design identity/dedup/port | specs | 已对齐 |
+| specs scenarios | tasks | 已对齐 |
+
+## Interface impact
+
+- L2 internal DTO
+- L3 Agent `document_id` chunk-scoped
+
+## Commit gate
+
+- proposal/design/specs/tasks present
+- no open user-interview blockers for Delivery 1
+- apply authorized by user at sm-flow start
+
+## Pre-apply research
+
+Reference files:
+
+- `LookupKnowledgeTool.java`
+- `KnowledgeDocumentRetriever.java`
+- `KnowledgeEvidencePostProcessor.java`
+- `RagResultProjector.java`
+- `LookupKnowledgeToolTest.java`
+- `RagResultProjectorTest.java`
+- `docs/milvus-hybrid-search-integration-checklist.md`
+
+Stack notes:
+
+- No MQ/request envelope changes
+- Spring `@Value` config pattern for rag.* keys
+- Tests use ReflectionTestUtils + Mockito