feat(rag): modularize knowledge retrieval pipeline
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# RAG 新架构
|
||||
|
||||
**更新日期**:2026-07-05
|
||||
**更新日期**:2026-07-06
|
||||
**状态**:当前主架构 + 后续演进边界
|
||||
**关联计划**:`mvp/issues/rag-refactor-plan.md`
|
||||
|
||||
@@ -44,9 +44,13 @@ flowchart TD
|
||||
SdkFallback --> Results
|
||||
SdkOnly --> Results
|
||||
|
||||
Results --> Normalize["relevance normalization"]
|
||||
Normalize --> Dedup["session dedup: RetrievedDocTracker"]
|
||||
Dedup --> Output["LookupResult"]
|
||||
Results --> Retry{"filtered result usable?"}
|
||||
Retry -->|no| RetryL1["raw query unfiltered L1 retry"]
|
||||
Retry -->|yes| Post["post-retrieval processing"]
|
||||
RetryL1 --> Post
|
||||
Post --> Pack["context packing"]
|
||||
Pack --> Dedup["session dedup: RetrievedDocTracker"]
|
||||
Dedup --> Output["LookupResult: evidenceBlocks / contextPack / traces"]
|
||||
Output --> Record["tool_invocation record"]
|
||||
Output --> Agent
|
||||
```
|
||||
@@ -66,10 +70,13 @@ Agent Executor
|
||||
-> Spring AI VectorStore only
|
||||
-> mode=sdk
|
||||
-> Milvus SDK only
|
||||
-> result normalization
|
||||
-> post-retrieval processing
|
||||
-> relevanceLevel
|
||||
-> completenessHint
|
||||
-> score/rawScore/scoreLabel
|
||||
-> evidenceBlocks
|
||||
-> rerankTrace
|
||||
-> context packing
|
||||
-> contextPack
|
||||
-> session dedup
|
||||
-> RetrievedDocTracker
|
||||
-> tool_invocation record
|
||||
@@ -180,10 +187,11 @@ L0 负责:
|
||||
- metadata/category filter candidate
|
||||
- trace 中的 hit reason
|
||||
|
||||
L0 不再默认负责:
|
||||
L0 不再负责:
|
||||
|
||||
```text
|
||||
L0 unique hit -> 直接作为最终检索结果
|
||||
L1 no result -> 返回 L0 文档作为事实证据
|
||||
```
|
||||
|
||||
当前职责是:
|
||||
@@ -192,8 +200,10 @@ L0 unique hit -> 直接作为最终检索结果
|
||||
query / AIOps payload
|
||||
-> L0 matched keywords / domains / entities
|
||||
-> category filter candidate
|
||||
-> L1 semantic retrieval
|
||||
-> relevance normalization
|
||||
-> filtered L1 semantic retrieval
|
||||
-> low-quality? raw query unfiltered L1 retry
|
||||
-> post-retrieval processing
|
||||
-> context packing
|
||||
```
|
||||
|
||||
这样既保留精确关键词和领域 hint 的价值,也避免 L0 误召回直接污染最终证据。
|
||||
@@ -319,33 +329,24 @@ AIOps payload
|
||||
|
||||
## 9. Evidence 与去重
|
||||
|
||||
当前 evidence 输出仍以 `LookupResult` 和工具返回文本为主,已经具备:
|
||||
当前 evidence 输出已从旧 `primary/supplement` 迁移为 evidence-first contract,核心字段包括:
|
||||
|
||||
- L0/L1 命中数量。
|
||||
- 检索层记录。
|
||||
- relevance level。
|
||||
- completeness hint。
|
||||
- session 级文档去重。
|
||||
- domain 行动记忆。
|
||||
- `tool_invocation` 明细记录。
|
||||
- `evidenceBlocks`
|
||||
- `contextPack`
|
||||
- `retrievalTrace`
|
||||
- `rerankTrace`
|
||||
- `relevanceLevel`
|
||||
- `completenessHint`
|
||||
- `retrievedDomainsThisSession`
|
||||
- `tool_invocation.retrieval_details`
|
||||
|
||||
后续更完整的 evidence block 目标:
|
||||
evidence block 结构:
|
||||
|
||||
```text
|
||||
source
|
||||
docId
|
||||
chunkIndex
|
||||
title
|
||||
breadcrumb
|
||||
score
|
||||
rawScore
|
||||
scoreLabel
|
||||
hitReason
|
||||
content
|
||||
expandedFrom
|
||||
source / title / breadcrumb / retrievalLayer / content / score / hitReasons
|
||||
```
|
||||
|
||||
这部分应作为下一阶段增强,而不是当前已完全完成能力。
|
||||
context pack 会按重排后的证据顺序生成 Agent 可消费的紧凑上下文,并保留 included/omitted sources 供 trace 检查。
|
||||
|
||||
## 10. 评测与验收
|
||||
|
||||
@@ -380,18 +381,18 @@ RAG 架构变更必须先过评测,再认为可合入主链路。
|
||||
- Markdown chunk 保留 `title` 和 `breadcrumb`。
|
||||
- embedding 输入包含 `title`、`breadcrumb` 和 `content`。
|
||||
- AIOps payload 生成推荐知识库 query。
|
||||
- `tool_invocation` 记录 relevance level 和 dedup reason。
|
||||
- `tool_invocation` 记录 relevance level、dedup reason、evidence summaries、retrieval trace、rerank trace 和 context pack summary。
|
||||
- `lookup_knowledge` 输出使用 evidence-first contract,不再暴露旧 `primary/supplement` 字段。
|
||||
- RAG offline baseline 和 live acceptance 脚本已补齐。
|
||||
|
||||
## 12. 后续演进
|
||||
|
||||
近期优先:
|
||||
|
||||
1. 完整 evidence block 结构化输出。
|
||||
2. 命中 chunk 的相邻 chunk / 同章节上下文扩展。
|
||||
3. metadata taxonomy 清理,例如 `database` 与 `infrastructure` 的分类边界。
|
||||
4. Query Transformer / MultiQuery 的可回退接入。
|
||||
5. VectorStore 写入路径评估。
|
||||
1. 命中 chunk 的相邻 chunk / 同章节上下文扩展。
|
||||
2. metadata taxonomy 清理,例如 `database` 与 `infrastructure` 的分类边界。
|
||||
3. Query Transformer / MultiQuery 的可回退接入。
|
||||
4. VectorStore 写入路径评估。
|
||||
|
||||
暂不优先:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user