zhuyongxin
ac1f831903
Preserve same-document multi-chunk evidence with evidenceKey identity, per-document caps, retrieve-k/return-n split, and a dense KnowledgeSearchPort. Archives Delivery 1 OpenSpec change as the foundation for hybrid retrieval.
103 lines
3.6 KiB
Markdown
103 lines
3.6 KiB
Markdown
# Decisions: rag-chunk-evidence-identity-dedup
|
||
|
||
## Capability sources
|
||
|
||
- sm-flow orchestration
|
||
- OpenSpec fallback protocol (file-based propose/apply/archive) — external openspec-propose/apply skills used as reference; execution via sm-flow fallback
|
||
- grill: fallback built-in protocol
|
||
- audit: fallback built-in protocol
|
||
|
||
## Scale
|
||
|
||
standard
|
||
|
||
## Clarify
|
||
|
||
- Problem: same-document multi-chunk evidence collapsed by source-level dedup.
|
||
- Outcome: Delivery 1 foundation before hybrid Delivery 2.
|
||
- Slug: `rag-chunk-evidence-identity-dedup`
|
||
- User authorized apply + archive in advance for sm-flow staged changes.
|
||
|
||
## Context
|
||
|
||
- Read: `devflow/glossary/CONTEXT.md`, modular-rag-pipeline brief, `openspec/specs/rag-knowledge-retrieval`, `rag-log-projections`, checklist doc §1.1
|
||
- Constraints into OpenSpec:
|
||
- L0 hint-only remains
|
||
- Do not thicken legacy SDK path
|
||
- Agent tool name/input stable
|
||
- Hybrid out of scope this change
|
||
|
||
## Question pool (grill)
|
||
|
||
| # | Dimension | Mode | Question | Status |
|
||
|---|---|---|---|---|
|
||
| Q1 | 术语 | evidence-driven | evidenceKey / document_id 语义？ | Resolved: evidenceKey=chunk id; projected document_id=evidenceKey |
|
||
| Q2 | 边界 | evidence-driven | Delivery 1 vs 2 边界？ | Resolved: per checklist; no schema/hybrid/SDK delete |
|
||
| Q3 | 验收 | evidence-driven | 如何验收多 chunk？ | Resolved: unit tests multi-chunk keep + projector |
|
||
| Q4 | 接口 | user-interview | document_id 改为 chunk 级是否可接受？ | **Pre-authorized by user** via “apply/archive 直接授权” + prior design agreement on scheme A (document_id=evidenceKey). Recorded as accepted behavior change. |
|
||
| Q5 | 技术 | evidence-driven | SearchPort 是否本 change 必须？ | Resolved: thin port required as foundation |
|
||
|
||
### Evidence-driven conclusions (reported)
|
||
|
||
1. Current collapse points: `KnowledgeEvidencePostProcessor.sourceKey` and `RagResultProjector` source fallback.
|
||
2. Metadata already has docId/chunkIndex on write path; not first-class on read path.
|
||
3. Existing main-spec still says source-level dedup — this change intentionally deltas that requirement.
|
||
|
||
### User-interview
|
||
|
||
- Q4 accepted under prior design alignment (scheme A) and explicit apply authorization for this sm-flow run. No remaining open product preference questions for Delivery 1.
|
||
|
||
## Audit
|
||
|
||
Module chain:
|
||
|
||
```text
|
||
LookupKnowledgeTool -> SearchPort -> Retriever -> PostProcessor -> Packer -> Assembler -> RagResultProjector
|
||
```
|
||
|
||
Risks:
|
||
|
||
1. Agent payload growth — mitigated by return-n + maxChunksPerDocument + projector budgets.
|
||
2. document_id semantic shift — documented L3 behavior change; tests updated.
|
||
3. Old data without chunkIndex — vector id fallback.
|
||
|
||
No ADR conflict with modular RAG L0/L1 boundary.
|
||
|
||
## Cross-artifact alignment
|
||
|
||
| From | To | Status |
|
||
|---|---|---|
|
||
| brief goals | proposal | 已对齐 |
|
||
| proposal scope | design decisions | 已对齐 |
|
||
| design identity/dedup/port | specs | 已对齐 |
|
||
| specs scenarios | tasks | 已对齐 |
|
||
|
||
## Interface impact
|
||
|
||
- L2 internal DTO
|
||
- L3 Agent `document_id` chunk-scoped
|
||
|
||
## Commit gate
|
||
|
||
- proposal/design/specs/tasks present
|
||
- no open user-interview blockers for Delivery 1
|
||
- apply authorized by user at sm-flow start
|
||
|
||
## Pre-apply research
|
||
|
||
Reference files:
|
||
|
||
- `LookupKnowledgeTool.java`
|
||
- `KnowledgeDocumentRetriever.java`
|
||
- `KnowledgeEvidencePostProcessor.java`
|
||
- `RagResultProjector.java`
|
||
- `LookupKnowledgeToolTest.java`
|
||
- `RagResultProjectorTest.java`
|
||
- `docs/milvus-hybrid-search-integration-checklist.md`
|
||
|
||
Stack notes:
|
||
|
||
- No MQ/request envelope changes
|
||
- Spring `@Value` config pattern for rag.* keys
|
||
- Tests use ReflectionTestUtils + Mockito
|