docs(mvp): document dense+BM25 hybrid knowledge retrieval
Add current RAG architecture covering MilvusClientV2 hybrid search, chunk evidence identity, rebuild ops, and update the MVP architecture index and system overview links.
This commit is contained in:
zhuyongxin
@@ -0,0 +1,328 @@
|
||||
# RAG 知识检索架构
|
||||
|
||||
**更新日期**：2026-07-28
|
||||
**状态**：当前可运行架构
|
||||
**关联实现**：`lookup_knowledge`、`MilvusHybridKnowledgeStore`、`KnowledgeSearchPort`
|
||||
**关联运维**：`scripts/rebuild_hybrid_knowledge.py`、`POST /api/knowledge/rebuild-hybrid`
|
||||
|
||||
## 1. 定位
|
||||
|
||||
知识检索是 Diagnosis Agent 的显式证据工具，不是隐式 Advisor。
|
||||
|
||||
```text
|
||||
Diagnosis Agent
|
||||
-> lookup_knowledge(query)
|
||||
-> Harness ToolBoundary / ACI projection
|
||||
-> EvidenceGuard 只认当前 Run 的 READY canonical 证据
|
||||
```
|
||||
|
||||
目标：
|
||||
|
||||
- 保留 Agent 可见的工具调用与证据边界
|
||||
- 用单一向量后端完成 dense + BM25 hybrid 检索
|
||||
- 用 chunk 级证据身份保证同文档多片段可同时进入上下文
|
||||
- 检索行为可配置、可重建、可审计
|
||||
|
||||
## 2. 稳定边界
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph AgentBoundary["Agent boundary"]
|
||||
Agent["Diagnosis Agent"]
|
||||
Tool["lookup_knowledge"]
|
||||
end
|
||||
|
||||
subgraph HarnessBoundary["Harness boundary"]
|
||||
Adapter["RagToolAdapter"]
|
||||
Projector["RagResultProjector"]
|
||||
Canonical["Redis canonical invocation"]
|
||||
end
|
||||
|
||||
subgraph RetrievalBoundary["Retrieval boundary"]
|
||||
Backend["LookupKnowledgeTool"]
|
||||
Port["KnowledgeSearchPort"]
|
||||
Store["MilvusHybridKnowledgeStore"]
|
||||
end
|
||||
|
||||
Agent --> Tool
|
||||
Tool --> Adapter
|
||||
Adapter --> Backend
|
||||
Backend --> Port
|
||||
Port --> Store
|
||||
Adapter --> Projector
|
||||
Adapter --> Canonical
|
||||
```
|
||||
|
||||
| 边界 | 职责 | 不负责 |
|
||||
|---|---|---|
|
||||
| Agent | 决定何时检索、如何用证据写报告 | 不直接访问 Milvus / MySQL 元数据表 |
|
||||
| Harness | Tool 校验、投影裁剪、canonical 存证 | 不改写检索排序算法 |
|
||||
| Retrieval | L0 hint、dense/BM25 召回、后处理、打包 | 不绕过 ACI 直接给 Agent 原始库响应 |
|
||||
|
||||
## 3. 当前主链路
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["lookup_knowledge(query)"] --> B["KnowledgeQueryTransformer"]
|
||||
B --> C["L0 hint: domain / keywords / categoryFilter"]
|
||||
C --> D["KnowledgeDocumentRetriever"]
|
||||
D --> E["KnowledgeSearchPort"]
|
||||
E --> F["VectorSearchService"]
|
||||
F --> G{"retrieval.search.mode"}
|
||||
G -->|dense| H["MilvusHybridKnowledgeStore.searchDense"]
|
||||
G -->|hybrid| I["MilvusHybridKnowledgeStore.searchHybrid"]
|
||||
H --> J["candidates + chunk identity"]
|
||||
I --> J
|
||||
J --> K["KnowledgeEvidencePostProcessor"]
|
||||
K --> L["evidenceKey dedup / maxChunksPerDocument / return-n"]
|
||||
L --> M{"filtered low quality?"}
|
||||
M -->|yes and had categoryFilter| N["unfiltered retry"]
|
||||
N --> K
|
||||
M -->|no| O["KnowledgeContextPacker"]
|
||||
O --> P["LookupResultAssembler"]
|
||||
P --> Q["RagResultProjector"]
|
||||
Q --> R["Agent-facing RagToolResult"]
|
||||
```
|
||||
|
||||
对应代码：
|
||||
|
||||
| 阶段 | 类 | 职责 |
|
||||
|---|---|---|
|
||||
| Tool 编排 | `LookupKnowledgeTool` | 串联 transform / retrieve / post / pack |
|
||||
| Query 理解 | `KnowledgeQueryTransformer` + `KnowledgeIndexService` | L0 只产 hint 与可选 category filter |
|
||||
| 检索端口 | `KnowledgeSearchPort` / `VectorKnowledgeSearchAdapter` | 屏蔽底层存储细节 |
|
||||
| 检索门面 | `VectorSearchService` | `dense` 或 `hybrid` 路由 |
|
||||
| 向量后端 | `MilvusHybridKnowledgeStore` | 唯一知识库读写后端（MilvusClientV2） |
|
||||
| 后处理 | `KnowledgeEvidencePostProcessor` | 归一化、规则 boost、chunk 去重、相关度等级 |
|
||||
| 打包 | `KnowledgeContextPacker` | 有界 context pack |
|
||||
| 投影 | `RagResultProjector` | 只暴露 Agent 可见 evidence 字段 |
|
||||
|
||||
## 4. 唯一向量后端：MilvusClientV2
|
||||
|
||||
### 4.1 已废弃路径
|
||||
|
||||
以下路径**不再**用于 `lookup_knowledge`：
|
||||
|
||||
- legacy `MilvusServiceClient` search / insert
|
||||
- `retrieval.vector-store.mode=sdk|spring|auto`
|
||||
- Spring AI `VectorStore` 作为知识检索主路径
|
||||
|
||||
### 4.2 当前后端
|
||||
|
||||
```text
|
||||
写入:
|
||||
VectorIndexService
|
||||
-> MilvusHybridKnowledgeStore.upsertChunk
|
||||
|
||||
读取:
|
||||
VectorSearchService
|
||||
-> MilvusHybridKnowledgeStore.searchDense
|
||||
-> MilvusHybridKnowledgeStore.searchHybrid
|
||||
```
|
||||
|
||||
默认 collection：
|
||||
|
||||
```yaml
|
||||
milvus:
|
||||
collection: biz
|
||||
```
|
||||
|
||||
重建时会 drop + recreate 该 collection，并按 dense + BM25 schema 重建。
|
||||
|
||||
## 5. Collection Schema
|
||||
|
||||
`biz`（可配置）逻辑字段：
|
||||
|
||||
| 字段 | 类型 | 用途 |
|
||||
|---|---|---|
|
||||
| `id` | VarChar PK | chunk 级主键 |
|
||||
| `content` | VarChar | 返回给 Agent 的原文片段 |
|
||||
| `search_text` | VarChar + analyzer | BM25 输入文本 |
|
||||
| `sparse_vector` | SparseFloatVector | BM25 Function 输出 |
|
||||
| `vector` | FloatVector | dense embedding |
|
||||
| `metadata` | JSON | docId / chunkIndex / category / kb_scope / title / breadcrumb 等 |
|
||||
|
||||
Function：
|
||||
|
||||
```text
|
||||
BM25(search_text -> sparse_vector)
|
||||
```
|
||||
|
||||
索引：
|
||||
|
||||
```text
|
||||
vector -> IVF_FLAT + L2
|
||||
sparse_vector -> SPARSE_INVERTED_INDEX + BM25
|
||||
```
|
||||
|
||||
写入时：
|
||||
|
||||
- `content` 保存原始 chunk 正文
|
||||
- `search_text` / dense embedding 使用 `Title + Path + Content` 拼装文本
|
||||
- metadata 必须带 `docId`、`chunkIndex`，供 chunk 级证据身份使用
|
||||
|
||||
## 6. 检索模式
|
||||
|
||||
配置：
|
||||
|
||||
```yaml
|
||||
retrieval:
|
||||
search:
|
||||
mode: hybrid # dense | hybrid
|
||||
hybrid:
|
||||
rrf-k: 60
|
||||
kb-scope: ""
|
||||
rag:
|
||||
retrieve-k: 20
|
||||
return-n: 5
|
||||
max-chunks-per-document: 2
|
||||
```
|
||||
|
||||
### 6.1 dense
|
||||
|
||||
```text
|
||||
query
|
||||
-> embedding
|
||||
-> dense ANN on vector
|
||||
-> topK
|
||||
```
|
||||
|
||||
### 6.2 hybrid（当前默认）
|
||||
|
||||
```text
|
||||
query
|
||||
-> path A: dense ANN(query embedding)
|
||||
-> path B: BM25 sparse ANN(raw query text)
|
||||
-> Milvus hybridSearch + RRFRanker(k)
|
||||
-> topK fused hits
|
||||
```
|
||||
|
||||
阈值兼容：
|
||||
|
||||
- 后处理仍按 dense 兼容 L2 距离做 `normalizeL2`
|
||||
- hybrid 命中若能从并行 dense 结果拿到同 id 的 L2，则回填该分数
|
||||
- 仅 BM25 命中、无 dense 分时，按弱相关处理，避免虚高 `PRECISE`
|
||||
|
||||
### 6.3 category filter 与降级
|
||||
|
||||
```text
|
||||
if L0 给出唯一 domain:
|
||||
先 filtered 检索
|
||||
if 无证据或 topSimilarity < referenceThreshold:
|
||||
再 unfiltered retry
|
||||
else:
|
||||
直接 unfiltered
|
||||
```
|
||||
|
||||
这里的 filter 是 metadata category / kb_scope 约束，不是第二套向量库。
|
||||
|
||||
## 7. 证据身份与去重
|
||||
|
||||
Delivery 1 已落地：
|
||||
|
||||
```text
|
||||
evidenceKey =
|
||||
docId#chunk-{chunkIndex}
|
||||
fallback: vector:{id}
|
||||
fallback: rank:{n}
|
||||
```
|
||||
|
||||
规则：
|
||||
|
||||
- 去重按 `evidenceKey`，不是按 source 文档路径
|
||||
- 同文档不同 chunk 可同时保留
|
||||
- `rag.max-chunks-per-document` 限制单文档最多进入结果的 chunk 数
|
||||
- `rag.return-n` 限制后处理后最多返回条数
|
||||
- Agent 投影中的 `document_id` 使用 chunk 级 evidenceKey
|
||||
|
||||
这保证 hybrid 召回的多片段不会在后处理/投影阶段被文档级折叠吞掉。
|
||||
|
||||
## 8. L0 / L1 职责
|
||||
|
||||
| 层 | 做什么 | 不做什么 |
|
||||
|---|---|---|
|
||||
| L0 | domain/keyword hint、可选 category filter、trace 解释、轻规则 boost | 不直接当事实 evidence |
|
||||
| L1 dense/BM25 | 事实证据召回 | 不依赖 frontmatter 关键词命中才返回正文 |
|
||||
|
||||
L0 命中文档正文不会在 L1 失败时兜底成 evidence。
|
||||
|
||||
## 9. Agent 可见契约
|
||||
|
||||
Agent 只看到有界 `RagToolResult`：
|
||||
|
||||
- `evidence_status`
|
||||
- `tool_call_id`
|
||||
- `query`
|
||||
- `evidence[]`：`document_id` / `source` / `title` / `breadcrumb` / `excerpt`
|
||||
- `relevance_level`
|
||||
- `truncated` / `returned_count`
|
||||
|
||||
不暴露：
|
||||
|
||||
- raw score / fused score
|
||||
- retrievalTrace / rerankTrace
|
||||
- contextPack 全文
|
||||
- Milvus 内部字段与凭据
|
||||
|
||||
完整内部结果仍在 `LookupResult` 中，供审计与调试使用。
|
||||
|
||||
## 10. 写入与重建
|
||||
|
||||
### 10.1 日常写入
|
||||
|
||||
文档上传 / 知识库初始化：
|
||||
|
||||
```text
|
||||
markdown
|
||||
-> frontmatter + body
|
||||
-> DocumentChunkService
|
||||
-> dense embedding + search_text
|
||||
-> MilvusHybridKnowledgeStore.upsertChunk
|
||||
-> MySQL api_document + L0 memory index
|
||||
```
|
||||
|
||||
### 10.2 全量重建
|
||||
|
||||
危险操作，需显式确认：
|
||||
|
||||
```bash
|
||||
python scripts/rebuild_hybrid_knowledge.py --confirm REBUILD
|
||||
```
|
||||
|
||||
等价 API：
|
||||
|
||||
```text
|
||||
POST /api/knowledge/rebuild-hybrid?confirm=REBUILD
|
||||
```
|
||||
|
||||
服务端顺序：
|
||||
|
||||
1. drop + recreate `milvus.collection`（默认 `biz`）
|
||||
2. 清空 MySQL `api_document`
|
||||
3. 清空内存 L0
|
||||
4. 扫描 `knowledge_base/**/*.md`（跳过 `README.md`）force 导入
|
||||
|
||||
不会修改磁盘上的 `knowledge_base/` 源文件。
|
||||
|
||||
## 11. 与旧文档的差异
|
||||
|
||||
| 旧描述（已归档） | 当前实现 |
|
||||
|---|---|
|
||||
| Spring AI VectorStore 主路径 + SDK fallback | 单一 MilvusClientV2 后端 |
|
||||
| `retrieval.vector-store.mode=auto/sdk/spring` | 已移除；改为 `retrieval.search.mode=dense/hybrid` |
|
||||
| source 级 evidence 去重 | chunk 级 `evidenceKey` 去重 |
|
||||
| 应用层 sparse-lite lexical 伪 hybrid | 库内 dense ANN + BM25 + RRFRanker |
|
||||
| 新建 `biz_hybrid` 过渡 collection | 默认使用并重建 `biz` |
|
||||
|
||||
历史材料见：
|
||||
|
||||
- `mvp/architecture/archive/2026-07-22-legacy/rag-architecture.md`
|
||||
- `mvp/architecture/archive/2026-07-22-legacy/modular-rag-pipeline.md`
|
||||
|
||||
## 12. 当前已知边界
|
||||
|
||||
- hybrid 依赖云端/实例支持 BM25 Function 与 sparse index
|
||||
- 全量重建受 embedding API 与 Milvus 写入延迟影响，可能较慢
|
||||
- L0 关键词匹配仍较粗，只作 hint，不作主召回
|
||||
- 尚未做邻块上下文自动扩展、cross-encoder rerank、真 query rewrite
|
||||
- `totalVectors` 统计接口仍可能返回 0，不代表 collection 为空；以 rebuild/init 结果与检索命中为准
|
||||
Reference in New Issue
Block a user