zhuyongxin
5c369f3b6c
Add confirm-gated rebuild-hybrid endpoint that drops biz_hybrid, clears api_document and L0, then force-imports knowledge_base markdown into the dense+BM25 store. Include PowerShell runner and ops README.
98 lines
2.4 KiB
Markdown
98 lines
2.4 KiB
Markdown
# 重建 hybrid 知识库（dense + BM25）
|
||
|
||
面向当前 `knowledge_base/` 目录文档，清空并重建 `biz_hybrid` collection。
|
||
|
||
## 前提
|
||
|
||
1. 应用已启动（默认 `http://localhost:9900`）
|
||
2. `MILVUS_TOKEN` 等连接配置可用
|
||
3. `application.yml` 已配置：
|
||
|
||
```yaml
|
||
milvus:
|
||
collection: biz_hybrid
|
||
retrieval:
|
||
search:
|
||
mode: hybrid
|
||
knowledge:
|
||
base-path: knowledge_base/
|
||
```
|
||
|
||
## 一键脚本
|
||
|
||
在项目根目录执行：
|
||
|
||
```powershell
|
||
.\scripts\rebuild-hybrid-knowledge.ps1 -Confirm REBUILD
|
||
```
|
||
|
||
指定服务地址：
|
||
|
||
```powershell
|
||
.\scripts\rebuild-hybrid-knowledge.ps1 -BaseUrl http://127.0.0.1:9900 -Confirm REBUILD
|
||
```
|
||
|
||
## 脚本会做什么
|
||
|
||
| 步骤 | 动作 |
|
||
|---|---|
|
||
| 1 | 检查 `/milvus/health` |
|
||
| 2 | 打印重建前 `/api/knowledge/stats` |
|
||
| 3 | `POST /api/knowledge/rebuild-hybrid?confirm=REBUILD` |
|
||
| 4 | 打印重建后 stats |
|
||
|
||
服务端 `rebuild-hybrid` 内部顺序：
|
||
|
||
1. **Drop + recreate** Milvus collection（`milvus.collection`，默认 `biz_hybrid`）
|
||
2. **清空** MySQL `api_document`
|
||
3. **清空** 内存 L0 索引
|
||
4. **扫描** `knowledge_base/**/*.md` 并 `force` 全量导入
|
||
- 写 MySQL 元数据
|
||
- 切片
|
||
- 写 dense 向量 + BM25 `search_text`
|
||
- 更新 L0
|
||
|
||
## 不会做什么
|
||
|
||
- **不会**删除旧 collection `biz`（需你在 Zilliz 控制台自行决定是否删）
|
||
- **不会**动 `knowledge_base/` 源文件
|
||
- **不会**在未传 `confirm=REBUILD` 时执行
|
||
|
||
## 手动 curl 等价命令
|
||
|
||
```bash
|
||
# 重建（危险）
|
||
curl -X POST "http://localhost:9900/api/knowledge/rebuild-hybrid?confirm=REBUILD"
|
||
|
||
# 仅增量/强制导入（不 drop collection）
|
||
curl -X POST "http://localhost:9900/api/knowledge/init?force=true"
|
||
|
||
# 统计
|
||
curl "http://localhost:9900/api/knowledge/stats"
|
||
```
|
||
|
||
## 成功判据
|
||
|
||
响应中大致应有：
|
||
|
||
```json
|
||
{
|
||
"success": true,
|
||
"collection": "biz_hybrid",
|
||
"inserted": <大于0>,
|
||
"failed": 0,
|
||
"milvus": { "recreated": true, "loaded": true }
|
||
}
|
||
```
|
||
|
||
然后用一条知识库里真实存在的术语/故障词走 `lookup_knowledge` 或 chat 验证 hybrid 命中。
|
||
|
||
## 失败排查
|
||
|
||
| 现象 | 可能原因 |
|
||
|---|---|
|
||
| connect / token 错误 | `MILVUS_TOKEN`、host、database |
|
||
| BM25 / analyzer 相关报错 | 云端 Milvus/Zilliz 版本不支持 BM25 Function |
|
||
| inserted=0 | `knowledge_base` 路径不对，或 md 缺 frontmatter/title |
|
||
| failed>0 | 看响应 `details` 与应用日志 |
|