feat(rag): add hybrid knowledge rebuild API and script

Add confirm-gated rebuild-hybrid endpoint that drops biz_hybrid, clears api_document and L0, then force-imports knowledge_base markdown into the dense+BM25 store. Include PowerShell runner and ops README.
2026-07-27 18:55:00 +08:00
parent f035538531
commit 5c369f3b6c
6 changed files with 407 additions and 3 deletions
@@ -0,0 +1,97 @@
+# 重建 hybrid 知识库（dense + BM25）
+
+面向当前 `knowledge_base/` 目录文档，清空并重建 `biz_hybrid` collection。
+
+## 前提
+
+1. 应用已启动（默认 `http://localhost:9900`）
+2. `MILVUS_TOKEN` 等连接配置可用
+3. `application.yml` 已配置：
+
+```yaml
+milvus:
+  collection: biz_hybrid
+retrieval:
+  search:
+    mode: hybrid
+knowledge:
+  base-path: knowledge_base/
+```
+
+## 一键脚本
+
+在项目根目录执行：
+
+```powershell
+.\scripts\rebuild-hybrid-knowledge.ps1 -Confirm REBUILD
+```
+
+指定服务地址：
+
+```powershell
+.\scripts\rebuild-hybrid-knowledge.ps1 -BaseUrl http://127.0.0.1:9900 -Confirm REBUILD
+```
+
+## 脚本会做什么
+
+| 步骤 | 动作 |
+|---|---|
+| 1 | 检查 `/milvus/health` |
+| 2 | 打印重建前 `/api/knowledge/stats` |
+| 3 | `POST /api/knowledge/rebuild-hybrid?confirm=REBUILD` |
+| 4 | 打印重建后 stats |
+
+服务端 `rebuild-hybrid` 内部顺序：
+
+1. **Drop + recreate** Milvus collection（`milvus.collection`，默认 `biz_hybrid`）
+2. **清空** MySQL `api_document`
+3. **清空** 内存 L0 索引
+4. **扫描** `knowledge_base/**/*.md` 并 `force` 全量导入  
+   - 写 MySQL 元数据  
+   - 切片  
+   - 写 dense 向量 + BM25 `search_text`  
+   - 更新 L0
+
+## 不会做什么
+
+- **不会**删除旧 collection `biz`（需你在 Zilliz 控制台自行决定是否删）
+- **不会**动 `knowledge_base/` 源文件
+- **不会**在未传 `confirm=REBUILD` 时执行
+
+## 手动 curl 等价命令
+
+```bash
+# 重建（危险）
+curl -X POST "http://localhost:9900/api/knowledge/rebuild-hybrid?confirm=REBUILD"
+
+# 仅增量/强制导入（不 drop collection）
+curl -X POST "http://localhost:9900/api/knowledge/init?force=true"
+
+# 统计
+curl "http://localhost:9900/api/knowledge/stats"
+```
+
+## 成功判据
+
+响应中大致应有：
+
+```json
+{
+  "success": true,
+  "collection": "biz_hybrid",
+  "inserted": <大于0>,
+  "failed": 0,
+  "milvus": { "recreated": true, "loaded": true }
+}
+```
+
+然后用一条知识库里真实存在的术语/故障词走 `lookup_knowledge` 或 chat 验证 hybrid 命中。
+
+## 失败排查
+
+| 现象 | 可能原因 |
+|---|---|
+| connect / token 错误 | `MILVUS_TOKEN`、host、database |
+| BM25 / analyzer 相关报错 | 云端 Milvus/Zilliz 版本不支持 BM25 Function |
+| inserted=0 | `knowledge_base` 路径不对，或 md 缺 frontmatter/title |
+| failed>0 | 看响应 `details` 与应用日志 |
@@ -0,0 +1,157 @@
+<#
+.SYNOPSIS
+  清空 hybrid 知识库（Milvus biz_hybrid + MySQL api_document + L0），并从 knowledge_base 全量重建。
+
+.DESCRIPTION
+  对应 dense+BM25 混合检索上线后的数据迁移步骤：
+
+    1) 检查服务健康
+    2) 可选：查看当前 /api/knowledge/stats
+    3) POST /api/knowledge/rebuild-hybrid?confirm=REBUILD
+       - drop + recreate milvus.collection（默认 biz_hybrid）
+       - 清空 MySQL api_document
+       - 清空内存 L0 索引
+       - 扫描 knowledge_base/**/*.md 强制导入并写入 dense+BM25
+
+  不会删除旧的 legacy collection `biz`（知识主路径已不再使用它）。
+
+.PARAMETER BaseUrl
+  服务根地址，默认 http://localhost:9900
+
+.PARAMETER Confirm
+  必须为 REBUILD 才会真正执行（防止误触）
+
+.PARAMETER SkipStats
+  跳过重建前后的 stats 查询
+
+.EXAMPLE
+  # 先启动 Spring Boot，再执行：
+  .\scripts\rebuild-hybrid-knowledge.ps1 -Confirm REBUILD
+
+.EXAMPLE
+  .\scripts\rebuild-hybrid-knowledge.ps1 -BaseUrl http://127.0.0.1:9900 -Confirm REBUILD
+#>
+
+[CmdletBinding()]
+param(
+    [string]$BaseUrl = "http://localhost:9900",
+    [ValidateSet("REBUILD")]
+    [Parameter(Mandatory = $true)]
+    [string]$Confirm,
+    [switch]$SkipStats
+)
+
+$ErrorActionPreference = "Stop"
+
+function Write-Step([string]$Message) {
+    Write-Host ""
+    Write-Host "==> $Message" -ForegroundColor Cyan
+}
+
+function Invoke-Json {
+    param(
+        [Parameter(Mandatory = $true)][string]$Method,
+        [Parameter(Mandatory = $true)][string]$Url,
+        [int]$TimeoutSec = 3600
+    )
+    try {
+        $resp = Invoke-WebRequest -Method $Method -Uri $Url -TimeoutSec $TimeoutSec
+        $body = $resp.Content
+        if ([string]::IsNullOrWhiteSpace($body)) {
+            return @{ statusCode = [int]$resp.StatusCode; body = $null }
+        }
+        return @{
+            statusCode = [int]$resp.StatusCode
+            body       = $body | ConvertFrom-Json
+        }
+    }
+    catch {
+        $status = $null
+        $raw = $null
+        if ($_.Exception.Response) {
+            $status = [int]$_.Exception.Response.StatusCode
+            try {
+                $stream = $_.Exception.Response.GetResponseStream()
+                $reader = New-Object System.IO.StreamReader($stream)
+                $raw = $reader.ReadToEnd()
+            }
+            catch { }
+        }
+        throw "HTTP $Method $Url failed (status=$status): $($_.Exception.Message)`n$raw"
+    }
+}
+
+$BaseUrl = $BaseUrl.TrimEnd("/")
+
+Write-Host "Hybrid knowledge rebuild" -ForegroundColor Green
+Write-Host "  BaseUrl : $BaseUrl"
+Write-Host "  Confirm : $Confirm"
+Write-Host "  Source  : knowledge_base/ (server-side knowledge.base-path)"
+Write-Host ""
+Write-Host "This will DESTROY data in:" -ForegroundColor Yellow
+Write-Host "  - Milvus collection milvus.collection (default: biz_hybrid)"
+Write-Host "  - MySQL table api_document"
+Write-Host "  - In-memory L0 knowledge index"
+Write-Host "Then re-import all markdown under knowledge_base."
+Write-Host ""
+
+# 1) health
+Write-Step "Check service health"
+try {
+    $health = Invoke-Json -Method GET -Url "$BaseUrl/milvus/health" -TimeoutSec 30
+    Write-Host ("  milvus health status={0}" -f $health.statusCode)
+    if ($health.body) {
+        $health.body | ConvertTo-Json -Depth 6 | Write-Host
+    }
+}
+catch {
+    Write-Host "  WARN: /milvus/health failed: $_" -ForegroundColor Yellow
+    Write-Host "  Continue if app is up but milvus health endpoint has issues."
+}
+
+# 2) stats before
+if (-not $SkipStats) {
+    Write-Step "Knowledge stats (before)"
+    try {
+        $before = Invoke-Json -Method GET -Url "$BaseUrl/api/knowledge/stats" -TimeoutSec 30
+        $before.body | ConvertTo-Json -Depth 6 | Write-Host
+    }
+    catch {
+        Write-Host "  WARN: stats before failed: $_" -ForegroundColor Yellow
+    }
+}
+
+# 3) rebuild
+Write-Step "POST /api/knowledge/rebuild-hybrid?confirm=REBUILD"
+$rebuildUrl = "$BaseUrl/api/knowledge/rebuild-hybrid?confirm=$Confirm"
+$rebuild = Invoke-Json -Method POST -Url $rebuildUrl -TimeoutSec 7200
+Write-Host ("  HTTP {0}" -f $rebuild.statusCode)
+$rebuild.body | ConvertTo-Json -Depth 12 | Write-Host
+
+if (-not $rebuild.body.success) {
+    Write-Host ""
+    Write-Host "Rebuild reported failure. Inspect details above." -ForegroundColor Red
+    exit 2
+}
+
+# 4) stats after
+if (-not $SkipStats) {
+    Write-Step "Knowledge stats (after)"
+    try {
+        $after = Invoke-Json -Method GET -Url "$BaseUrl/api/knowledge/stats" -TimeoutSec 30
+        $after.body | ConvertTo-Json -Depth 6 | Write-Host
+    }
+    catch {
+        Write-Host "  WARN: stats after failed: $_" -ForegroundColor Yellow
+    }
+}
+
+Write-Host ""
+Write-Host "Done." -ForegroundColor Green
+Write-Host "Next:"
+Write-Host "  1) Ensure application.yml has:"
+Write-Host "       milvus.collection: biz_hybrid"
+Write-Host "       retrieval.search.mode: hybrid"
+Write-Host "  2) Smoke test lookup_knowledge / chat with a known doc query"
+Write-Host "  3) Optional: drop legacy collection 'biz' manually in Zilliz console if no longer needed"
+exit 0