feat(rag): hybrid multi-path search with RRF fusion

Add configurable hybrid mode on KnowledgeSearchPort that fuses dense unfiltered, dense filtered, and lexical ranks via RRF while preserving dense-compatible threshold scores. Archives Delivery 2 OpenSpec change.
2026-07-27 18:33:31 +08:00
parent ac1f831903
commit 376ad0c241
19 changed files with 661 additions and 8 deletions
@@ -0,0 +1 @@
+ready: 2026-07-27
@@ -0,0 +1,3 @@
+committed: 2026-07-27
+change: rag-hybrid-search-rrf
+authorized-apply: user-preauthorized-sm-flow
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-07-27
@@ -0,0 +1,58 @@
+# Design: rag-hybrid-search-rrf
+
+## Context
+
+- Delivery 1 archived: chunk evidenceKey, SearchPort, return caps.
+- Legacy SDK path will be abandoned later; hybrid must live on SearchPort.
+- Full sparse schema rebuild is operationally heavy; ship fusion first.
+
+## Decisions
+
+### D1. Hybrid = multi-path + RRF on SearchPort
+
+When `retrieval.search.mode=hybrid`:
+
+```text
+paths:
+  1) dense(query, filter=null)                 # always
+  2) dense(query, filter=category)             # if category present
+  3) lexical rank over union of dense hits     # sparse-lite
+fuse by evidenceKey using RRF(k)
+return topK fused hits
+```
+
+### D2. RRF formula
+
+```text
+score(d) = Σ w_i / (k + rank_i(d))
+```
+
+Defaults: k=60, all w_i=1.0. Optional weights via config.
+
+### D3. Score semantics
+
+- `KnowledgeSearchHit.score` remains **compatible L2 distance from best dense hit** for post-process thresholds.
+- Fused RRF score is carried in metadata (`fusedScore`, `fusionRanks`) for trace, not as L2.
+
+### D4. Lexical path (sparse-lite)
+
+Until BM25 schema:
+
+- Tokenize query (simple whitespace / non-alnum split, lower-case)
+- Score each candidate by term coverage over title+breadcrumb+content
+- Rank candidates for RRF path only
+- Not a replacement for true BM25 inverted index
+
+### D5. Serial filter retry
+
+Lookup tool keeps low-quality unfiltered retry as safety net even in hybrid, because hybrid already includes unfiltered dense; retry remains cheap no-op when already fused well.
+
+## Non-goals now
+
+- New Milvus collection fields
+- Reindex jobs
+- SDK hybridSearch API calls
+
+## Risks
+
+- Lexical path only ranks already-recalled dense candidates → does not expand pure-term misses outside dense topK. Acceptable intermediate; true BM25 later expands recall.
@@ -0,0 +1,48 @@
+# Change: RAG hybrid search with RRF
+
+## Why
+
+Delivery 1 fixed chunk identity/dedup. Delivery 2 enables multi-path retrieval fused by RRF so category filters no longer whole-replace good results and lexical signals can complement dense ranks.
+
+True Milvus sparse/BM25 schema migration remains staged: this change ships a **working hybrid mode on KnowledgeSearchPort** using multi-path dense (+ lexical rank path) and RRF, without thickening the legacy SDK API surface.
+
+## What Changes
+
+- `retrieval.search.mode=dense|hybrid` (default dense)
+- Hybrid search on `KnowledgeSearchPort`:
+  - dense unfiltered path
+  - dense filtered path when category present
+  - lexical rank path over the candidate union (sparse-lite until BM25 schema lands)
+  - RRF / optional weighted RRF fusion by evidenceKey
+- Preserve dense L2 scores for quality thresholds; RRF only orders
+- Lookup tool uses search mode; filter low-quality retry remains as safety net
+- Config for rrf-k and path weights
+- Tests for RRF fusion ordering and hybrid adapter behavior
+
+## Non-goals
+
+- Full Milvus BM25/sparse collection rebuild (documented follow-up)
+- Deleting legacy SDK mode entirely
+- Cross-encoder model rerank
+- Changing Agent tool schema name/input
+
+## Capabilities
+
+### New Capabilities
+
+- `rag-hybrid-search`: hybrid mode, multi-path recall, RRF fusion via search port
+
+### Modified Capabilities
+
+- `rag-chunk-evidence-identity`: hybrid hits must keep chunk identity
+- `rag-knowledge-retrieval`: filtered/unfiltered cooperation via fusion rather than only serial replace
+
+## Impact
+
+- Code: retrieval package, lookup wiring/config, tests
+- Runtime default remains dense unless hybrid enabled
+- Interface: L2 internal (search port behavior)
+
+## Depends on
+
+- Archived Delivery 1: `rag-chunk-evidence-identity-dedup`
@@ -0,0 +1,44 @@
+# rag-hybrid-search Specification
+
+## ADDED Requirements
+
+### Requirement: Knowledge search SHALL support configurable dense and hybrid modes
+
+The knowledge search port SHALL support `retrieval.search.mode` values `dense` and `hybrid`. Default SHALL be `dense` for backward-compatible behavior.
+
+#### Scenario: Dense mode single path
+
+- **WHEN** mode is dense
+- **THEN** the port SHALL perform a single dense vector search with the provided category filter
+
+#### Scenario: Hybrid mode multi-path fusion
+
+- **WHEN** mode is hybrid
+- **THEN** the port SHALL gather multiple retrieval paths and fuse them by RRF before returning hits
+
+### Requirement: Hybrid fusion SHALL use RRF over evidence identities
+
+Hybrid fusion SHALL score candidates by reciprocal rank fusion on evidenceKey identities and SHALL NOT require comparable raw scores across paths.
+
+#### Scenario: Multi-path candidate rises with RRF
+
+- **WHEN** a chunk ranks highly on more than one hybrid path
+- **THEN** its fused rank SHALL improve relative to a chunk that ranks highly on only one path
+
+### Requirement: Hybrid SHALL preserve dense quality scores for thresholds
+
+Fused ordering SHALL NOT replace the dense compatibility score used by post-process quality thresholds.
+
+#### Scenario: Threshold score remains dense-compatible
+
+- **WHEN** hybrid returns a hit that originated from dense search
+- **THEN** hit.score SHALL remain a dense-compatible distance/similarity mapping usable by existing normalizeL2 thresholds
+
+### Requirement: Hybrid SHALL keep chunk identity fields
+
+Hybrid hits SHALL populate docId, chunkIndex, and evidenceKey consistently with Delivery 1 identity rules.
+
+#### Scenario: Fused hit keeps evidenceKey
+
+- **WHEN** hybrid merges the same chunk from two paths
+- **THEN** the returned hit SHALL use one evidenceKey and retain chunk identity metadata
@@ -0,0 +1,9 @@
+# Tasks: rag-hybrid-search-rrf
+
+- [x] 1. Add RRF fusion utility
+- [x] 2. Extend KnowledgeSearchRequest/Hit metadata for fusion ranks
+- [x] 3. Implement hybrid multi-path search in VectorKnowledgeSearchAdapter
+- [x] 4. Wire retrieval.search.mode and rrf config
+- [x] 5. Pass mode from LookupKnowledgeTool / retriever
+- [x] 6. Unit tests for RRF and hybrid adapter ordering
+- [x] 7. Verify dense mode regression tests still pass
@@ -0,0 +1,46 @@
+# rag-hybrid-search Specification
+
+## Purpose
+TBD - created by archiving change rag-hybrid-search-rrf. Update Purpose after archive.
+## Requirements
+### Requirement: Knowledge search SHALL support configurable dense and hybrid modes
+
+The knowledge search port SHALL support `retrieval.search.mode` values `dense` and `hybrid`. Default SHALL be `dense` for backward-compatible behavior.
+
+#### Scenario: Dense mode single path
+
+- **WHEN** mode is dense
+- **THEN** the port SHALL perform a single dense vector search with the provided category filter
+
+#### Scenario: Hybrid mode multi-path fusion
+
+- **WHEN** mode is hybrid
+- **THEN** the port SHALL gather multiple retrieval paths and fuse them by RRF before returning hits
+
+### Requirement: Hybrid fusion SHALL use RRF over evidence identities
+
+Hybrid fusion SHALL score candidates by reciprocal rank fusion on evidenceKey identities and SHALL NOT require comparable raw scores across paths.
+
+#### Scenario: Multi-path candidate rises with RRF
+
+- **WHEN** a chunk ranks highly on more than one hybrid path
+- **THEN** its fused rank SHALL improve relative to a chunk that ranks highly on only one path
+
+### Requirement: Hybrid SHALL preserve dense quality scores for thresholds
+
+Fused ordering SHALL NOT replace the dense compatibility score used by post-process quality thresholds.
+
+#### Scenario: Threshold score remains dense-compatible
+
+- **WHEN** hybrid returns a hit that originated from dense search
+- **THEN** hit.score SHALL remain a dense-compatible distance/similarity mapping usable by existing normalizeL2 thresholds
+
+### Requirement: Hybrid SHALL keep chunk identity fields
+
+Hybrid hits SHALL populate docId, chunkIndex, and evidenceKey consistently with Delivery 1 identity rules.
+
+#### Scenario: Fused hit keeps evidenceKey
+
+- **WHEN** hybrid merges the same chunk from two paths
+- **THEN** the returned hit SHALL use one evidenceKey and retain chunk identity metadata
+