zhuyongxin
376ad0c241
Add configurable hybrid mode on KnowledgeSearchPort that fuses dense unfiltered, dense filtered, and lexical ranks via RRF while preserving dense-compatible threshold scores. Archives Delivery 2 OpenSpec change.
49 lines
1.8 KiB
Markdown
49 lines
1.8 KiB
Markdown
# Change: RAG hybrid search with RRF
|
|
|
|
## Why
|
|
|
|
Delivery 1 fixed chunk identity/dedup. Delivery 2 enables multi-path retrieval fused by RRF so category filters no longer whole-replace good results and lexical signals can complement dense ranks.
|
|
|
|
True Milvus sparse/BM25 schema migration remains staged: this change ships a **working hybrid mode on KnowledgeSearchPort** using multi-path dense (+ lexical rank path) and RRF, without thickening the legacy SDK API surface.
|
|
|
|
## What Changes
|
|
|
|
- `retrieval.search.mode=dense|hybrid` (default dense)
|
|
- Hybrid search on `KnowledgeSearchPort`:
|
|
- dense unfiltered path
|
|
- dense filtered path when category present
|
|
- lexical rank path over the candidate union (sparse-lite until BM25 schema lands)
|
|
- RRF / optional weighted RRF fusion by evidenceKey
|
|
- Preserve dense L2 scores for quality thresholds; RRF only orders
|
|
- Lookup tool uses search mode; filter low-quality retry remains as safety net
|
|
- Config for rrf-k and path weights
|
|
- Tests for RRF fusion ordering and hybrid adapter behavior
|
|
|
|
## Non-goals
|
|
|
|
- Full Milvus BM25/sparse collection rebuild (documented follow-up)
|
|
- Deleting legacy SDK mode entirely
|
|
- Cross-encoder model rerank
|
|
- Changing Agent tool schema name/input
|
|
|
|
## Capabilities
|
|
|
|
### New Capabilities
|
|
|
|
- `rag-hybrid-search`: hybrid mode, multi-path recall, RRF fusion via search port
|
|
|
|
### Modified Capabilities
|
|
|
|
- `rag-chunk-evidence-identity`: hybrid hits must keep chunk identity
|
|
- `rag-knowledge-retrieval`: filtered/unfiltered cooperation via fusion rather than only serial replace
|
|
|
|
## Impact
|
|
|
|
- Code: retrieval package, lookup wiring/config, tests
|
|
- Runtime default remains dense unless hybrid enabled
|
|
- Interface: L2 internal (search port behavior)
|
|
|
|
## Depends on
|
|
|
|
- Archived Delivery 1: `rag-chunk-evidence-identity-dedup`
|