feat(rag): dense+BM25 hybrid on MilvusClientV2, drop SDK path

Replace legacy MilvusServiceClient knowledge search/write with a single MilvusClientV2 hybrid store (BM25 function + dense ANN + RRFRanker). Use collection biz_hybrid and require knowledge reindex.
2026-07-27 18:49:20 +08:00
parent 376ad0c241
commit f035538531
22 changed files with 813 additions and 936 deletions
@@ -0,0 +1,57 @@
+# Design: rag-bm25-hybrid-drop-sdk
+
+## Decision: single backend
+
+```text
+KnowledgeSearchPort
+  -> MilvusHybridKnowledgeStore (MilvusClientV2 only)
+Write
+  -> VectorIndexService -> MilvusHybridKnowledgeStore
+```
+
+No `retrieval.vector-store.mode=sdk|spring|auto` for knowledge lookup.
+
+## Schema (`milvus.collection`, default `biz_hybrid`)
+
+| field | type | notes |
+|---|---|---|
+| id | VarChar PK | chunk id |
+| content | VarChar | original chunk body returned to agent |
+| search_text | VarChar + analyzer | BM25 input (title/path/content) |
+| sparse_vector | SparseFloatVector | BM25 function output |
+| vector | FloatVector | dense embedding |
+| metadata | JSON | docId, chunkIndex, category, kb_scope, ... |
+
+Function: `BM25(search_text -> sparse_vector)`
+
+Indexes:
+
+- `vector`: IVF_FLAT / L2 (or COSINE if configured later)
+- `sparse_vector`: SPARSE_INVERTED_INDEX / BM25
+
+## Hybrid query
+
+```text
+AnnSearchReq(vector, FloatVec(queryEmbedding), topK, filter?)
+AnnSearchReq(sparse_vector, EmbeddedText(query), topK, filter?)
+HybridSearchReq + RRFRanker(k)
+```
+
+Dense-compatible score for thresholds: if entity lacks dense distance, map fused score conservatively or re-use dense-only probe. Preferred: keep dense distance when available from a parallel dense search metadata; for hybrid hits use inverse-rank placeholder only in metadata and set score from optional dense sub-hit map.
+
+Implementation approach for threshold stability:
+
+1. Run hybridSearch for ordering/identity
+2. Build map evidenceKey -> dense L2 from a concurrent dense search (same filter/topK)
+3. Attach dense score onto fused hits when present; else maxL2 (low similarity) so weak BM25-only hits don't fake PRECISE
+
+## Migration
+
+- New collection name avoids mutating legacy `biz`
+- Operators re-run knowledge init / document reindex
+- Document in acceptance
+
+## Drop SDK search
+
+- Delete SDK search methods usage from knowledge path
+- `MilvusServiceClient` bean may remain temporarily only if other non-search utilities need it; prefer migrate write/delete fully to V2 and stop creating V1 client if unused