chore(rag): rebuild script in Python and use biz collection

Switch default knowledge collection back to biz (drop+recreate on rebuild),
replace PowerShell rebuild runner with Python, skip README.md imports, and
merge duplicate rag keys in application.yml.
This commit is contained in:
zhuyongxin
2026-07-28 09:49:43 +08:00
parent 38f781b157
commit 1e532fb851
8 changed files with 225 additions and 36 deletions
+24 -15
View File
@@ -1,6 +1,6 @@
# 重建 hybrid 知识库(dense + BM25)
面向当前 `knowledge_base/` 目录文档,清空并重建 `biz_hybrid` collection。
面向当前 `knowledge_base/` 目录文档,**清空并重建**配置中的 Milvus collection(默认 **`biz`**)。
## 前提
@@ -10,7 +10,7 @@
```yaml
milvus:
collection: biz_hybrid
collection: biz
retrieval:
search:
mode: hybrid
@@ -18,20 +18,28 @@ knowledge:
base-path: knowledge_base/
```
## 一键脚本
## 一键脚本(Python)
在项目根目录执行:
```powershell
.\scripts\rebuild-hybrid-knowledge.ps1 -Confirm REBUILD
```bash
python scripts/rebuild_hybrid_knowledge.py --confirm REBUILD
```
指定服务地址:
```powershell
.\scripts\rebuild-hybrid-knowledge.ps1 -BaseUrl http://127.0.0.1:9900 -Confirm REBUILD
```bash
python scripts/rebuild_hybrid_knowledge.py --base-url http://127.0.0.1:9900 --confirm REBUILD
```
跳过前后 stats:
```bash
python scripts/rebuild_hybrid_knowledge.py --confirm REBUILD --skip-stats
```
依赖:Python 3.9+ 标准库即可(无需 pip 包)。
## 脚本会做什么
| 步骤 | 动作 |
@@ -43,10 +51,12 @@ knowledge:
服务端 `rebuild-hybrid` 内部顺序:
1. **Drop + recreate** Milvus collection(`milvus.collection`,默认 `biz_hybrid`)
1. **Drop + recreate** Milvus collection(`milvus.collection`,默认 `biz`)
- 原有向量数据会被删除
- 按 dense + BM25 schema 重建
2. **清空** MySQL `api_document`
3. **清空** 内存 L0 索引
4. **扫描** `knowledge_base/**/*.md` 并 `force` 全量导入
4. **扫描** `knowledge_base/**/*.md`(跳过 `README.md`)并 force 全量导入
- 写 MySQL 元数据
- 切片
- 写 dense 向量 + BM25 `search_text`
@@ -54,17 +64,16 @@ knowledge:
## 不会做什么
- **不会**删除旧 collection `biz`(需你在 Zilliz 控制台自行决定是否删)
- **不会**动 `knowledge_base/` 源文件
- **不会**在未传 `confirm=REBUILD` 时执行
- **不会**在未传 `--confirm REBUILD` 时执行
## 手动 curl 等价命令
```bash
# 重建(危险)
# 重建(危险:会清空 biz collection + api_document)
curl -X POST "http://localhost:9900/api/knowledge/rebuild-hybrid?confirm=REBUILD"
# 仅增量/强制导入(不 drop collection)
# 仅强制导入(不 drop collection)
curl -X POST "http://localhost:9900/api/knowledge/init?force=true"
# 统计
@@ -78,8 +87,8 @@ curl "http://localhost:9900/api/knowledge/stats"
```json
{
"success": true,
"collection": "biz_hybrid",
"inserted": <大于0>,
"collection": "biz",
"inserted": 15,
"failed": 0,
"milvus": { "recreated": true, "loaded": true }
}