chore(rag): rebuild script in Python and use biz collection
Switch default knowledge collection back to biz (drop+recreate on rebuild), replace PowerShell rebuild runner with Python, skip README.md imports, and merge duplicate rag keys in application.yml.
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# 重建 hybrid 知识库(dense + BM25)
|
||||
|
||||
面向当前 `knowledge_base/` 目录文档,清空并重建 `biz_hybrid` collection。
|
||||
面向当前 `knowledge_base/` 目录文档,**清空并重建**配置中的 Milvus collection(默认 **`biz`**)。
|
||||
|
||||
## 前提
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
|
||||
```yaml
|
||||
milvus:
|
||||
collection: biz_hybrid
|
||||
collection: biz
|
||||
retrieval:
|
||||
search:
|
||||
mode: hybrid
|
||||
@@ -18,20 +18,28 @@ knowledge:
|
||||
base-path: knowledge_base/
|
||||
```
|
||||
|
||||
## 一键脚本
|
||||
## 一键脚本(Python)
|
||||
|
||||
在项目根目录执行:
|
||||
|
||||
```powershell
|
||||
.\scripts\rebuild-hybrid-knowledge.ps1 -Confirm REBUILD
|
||||
```bash
|
||||
python scripts/rebuild_hybrid_knowledge.py --confirm REBUILD
|
||||
```
|
||||
|
||||
指定服务地址:
|
||||
|
||||
```powershell
|
||||
.\scripts\rebuild-hybrid-knowledge.ps1 -BaseUrl http://127.0.0.1:9900 -Confirm REBUILD
|
||||
```bash
|
||||
python scripts/rebuild_hybrid_knowledge.py --base-url http://127.0.0.1:9900 --confirm REBUILD
|
||||
```
|
||||
|
||||
跳过前后 stats:
|
||||
|
||||
```bash
|
||||
python scripts/rebuild_hybrid_knowledge.py --confirm REBUILD --skip-stats
|
||||
```
|
||||
|
||||
依赖:Python 3.9+ 标准库即可(无需 pip 包)。
|
||||
|
||||
## 脚本会做什么
|
||||
|
||||
| 步骤 | 动作 |
|
||||
@@ -43,10 +51,12 @@ knowledge:
|
||||
|
||||
服务端 `rebuild-hybrid` 内部顺序:
|
||||
|
||||
1. **Drop + recreate** Milvus collection(`milvus.collection`,默认 `biz_hybrid`)
|
||||
1. **Drop + recreate** Milvus collection(`milvus.collection`,默认 `biz`)
|
||||
- 原有向量数据会被删除
|
||||
- 按 dense + BM25 schema 重建
|
||||
2. **清空** MySQL `api_document`
|
||||
3. **清空** 内存 L0 索引
|
||||
4. **扫描** `knowledge_base/**/*.md` 并 `force` 全量导入
|
||||
4. **扫描** `knowledge_base/**/*.md`(跳过 `README.md`)并 force 全量导入
|
||||
- 写 MySQL 元数据
|
||||
- 切片
|
||||
- 写 dense 向量 + BM25 `search_text`
|
||||
@@ -54,17 +64,16 @@ knowledge:
|
||||
|
||||
## 不会做什么
|
||||
|
||||
- **不会**删除旧 collection `biz`(需你在 Zilliz 控制台自行决定是否删)
|
||||
- **不会**动 `knowledge_base/` 源文件
|
||||
- **不会**在未传 `confirm=REBUILD` 时执行
|
||||
- **不会**在未传 `--confirm REBUILD` 时执行
|
||||
|
||||
## 手动 curl 等价命令
|
||||
|
||||
```bash
|
||||
# 重建(危险)
|
||||
# 重建(危险:会清空 biz collection + api_document)
|
||||
curl -X POST "http://localhost:9900/api/knowledge/rebuild-hybrid?confirm=REBUILD"
|
||||
|
||||
# 仅增量/强制导入(不 drop collection)
|
||||
# 仅强制导入(不 drop collection)
|
||||
curl -X POST "http://localhost:9900/api/knowledge/init?force=true"
|
||||
|
||||
# 统计
|
||||
@@ -78,8 +87,8 @@ curl "http://localhost:9900/api/knowledge/stats"
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"collection": "biz_hybrid",
|
||||
"inserted": <大于0>,
|
||||
"collection": "biz",
|
||||
"inserted": 15,
|
||||
"failed": 0,
|
||||
"milvus": { "recreated": true, "loaded": true }
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user