feat(rag): hybrid multi-path search with RRF fusion

Add configurable hybrid mode on KnowledgeSearchPort that fuses dense unfiltered, dense filtered, and lexical ranks via RRF while preserving dense-compatible threshold scores. Archives Delivery 2 OpenSpec change.
2026-07-27 18:33:31 +08:00
parent ac1f831903
commit 376ad0c241
19 changed files with 661 additions and 8 deletions
@@ -0,0 +1,58 @@
+# Design: rag-hybrid-search-rrf
+
+## Context
+
+- Delivery 1 archived: chunk evidenceKey, SearchPort, return caps.
+- Legacy SDK path will be abandoned later; hybrid must live on SearchPort.
+- Full sparse schema rebuild is operationally heavy; ship fusion first.
+
+## Decisions
+
+### D1. Hybrid = multi-path + RRF on SearchPort
+
+When `retrieval.search.mode=hybrid`:
+
+```text
+paths:
+  1) dense(query, filter=null)                 # always
+  2) dense(query, filter=category)             # if category present
+  3) lexical rank over union of dense hits     # sparse-lite
+fuse by evidenceKey using RRF(k)
+return topK fused hits
+```
+
+### D2. RRF formula
+
+```text
+score(d) = Σ w_i / (k + rank_i(d))
+```
+
+Defaults: k=60, all w_i=1.0. Optional weights via config.
+
+### D3. Score semantics
+
+- `KnowledgeSearchHit.score` remains **compatible L2 distance from best dense hit** for post-process thresholds.
+- Fused RRF score is carried in metadata (`fusedScore`, `fusionRanks`) for trace, not as L2.
+
+### D4. Lexical path (sparse-lite)
+
+Until BM25 schema:
+
+- Tokenize query (simple whitespace / non-alnum split, lower-case)
+- Score each candidate by term coverage over title+breadcrumb+content
+- Rank candidates for RRF path only
+- Not a replacement for true BM25 inverted index
+
+### D5. Serial filter retry
+
+Lookup tool keeps low-quality unfiltered retry as safety net even in hybrid, because hybrid already includes unfiltered dense; retry remains cheap no-op when already fused well.
+
+## Non-goals now
+
+- New Milvus collection fields
+- Reindex jobs
+- SDK hybridSearch API calls
+
+## Risks
+
+- Lexical path only ranks already-recalled dense candidates → does not expand pure-term misses outside dense topK. Acceptable intermediate; true BM25 later expands recall.