feat(rag): hybrid multi-path search with RRF fusion
Add configurable hybrid mode on KnowledgeSearchPort that fuses dense unfiltered, dense filtered, and lexical ranks via RRF while preserving dense-compatible threshold scores. Archives Delivery 2 OpenSpec change.
This commit is contained in:
zhuyongxin
+2
-1
@@ -11,7 +11,8 @@
|
|||||||
|
|
||||||
| 日期 | slug | 说明 | 领域 | 关键词 | 关联 OpenSpec | 状态 |
|
| 日期 | slug | 说明 | 领域 | 关键词 | 关联 OpenSpec | 状态 |
|
||||||
|---|---|---|---|---|---|---|
|
|---|---|---|---|---|---|---|
|
||||||
| 2026-07-27 | rag-chunk-evidence-identity-dedup | chunk 级证据身份、去重、retrieve-k/return-n 与 SearchPort 地基，为 hybrid 铺路。 | RAG/证据身份/去重 | evidenceKey, maxChunksPerDocument, retrieve-k, return-n, KnowledgeSearchPort, document_id chunk-scoped | openspec/changes/rag-chunk-evidence-identity-dedup | accepted-unarchived |
|
| 2026-07-27 | rag-chunk-evidence-identity-dedup | chunk 级证据身份、去重、retrieve-k/return-n 与 SearchPort 地基，为 hybrid 铺路。 | RAG/证据身份/去重 | evidenceKey, maxChunksPerDocument, retrieve-k, return-n, KnowledgeSearchPort, document_id chunk-scoped | openspec/changes/archive/2026-07-27-rag-chunk-evidence-identity-dedup | archived |
|
||||||
|
| 2026-07-27 | rag-hybrid-search-rrf | Delivery 2：可配置 hybrid 检索与 RRF 多路融合（不绑旧 SDK）。 | RAG/hybrid/RRF | hybrid mode, RRF, KnowledgeSearchPort, filtered+unfiltered fusion, sparse-lite lexical | openspec/changes/archive/2026-07-27-rag-hybrid-search-rrf | archived |
|
||||||
| 2026-07-21 | single-react-tool-invocation-store | 建立统一 ToolBoundary 与 Redis canonical invocation store，集中生命周期、证据状态、TTL、容量和 Run 所有权。 | Harness/Tool boundary/Canonical store | ISS-014, ToolBoundary, canonical invocation, PROJECTING, READY, ERROR, TTL, RESULT_TOO_LARGE | openspec/changes/archive/2026-07-21-single-react-tool-invocation-store | archived |
|
| 2026-07-21 | single-react-tool-invocation-store | 建立统一 ToolBoundary 与 Redis canonical invocation store，集中生命周期、证据状态、TTL、容量和 Run 所有权。 | Harness/Tool boundary/Canonical store | ISS-014, ToolBoundary, canonical invocation, PROJECTING, READY, ERROR, TTL, RESULT_TOO_LARGE | openspec/changes/archive/2026-07-21-single-react-tool-invocation-store | archived |
|
||||||
| 2026-07-21 | single-react-harness-run-context | 建立显式 RunContext、Harness Core、预算、取消、类型化重试和 Tool Store 基础。 | Harness/Run lifecycle/Budget | ISS-014, RunContext, deadline, cancellation, budget, retry, ToolCallKey | openspec/changes/archive/2026-07-21-single-react-harness-run-context | archived |
|
| 2026-07-21 | single-react-harness-run-context | 建立显式 RunContext、Harness Core、预算、取消、类型化重试和 Tool Store 基础。 | Harness/Run lifecycle/Budget | ISS-014, RunContext, deadline, cancellation, budget, retry, ToolCallKey | openspec/changes/archive/2026-07-21-single-react-harness-run-context | archived |
|
||||||
| 2026-07-21 | single-react-aci-tool-contracts | 冻结 RAG、日志和 MySQL evidence Tool 的 Agent-facing ACI Schema、状态、框架调用引用和描述边界。 | Harness/Agent Tool contract | ISS-014, ACI, tool_call_id, evidence_status, RAG, query_logs, query_mysql, MOCK | openspec/changes/archive/2026-07-21-single-react-aci-tool-contracts | archived |
|
| 2026-07-21 | single-react-aci-tool-contracts | 冻结 RAG、日志和 MySQL evidence Tool 的 Agent-facing ACI Schema、状态、框架调用引用和描述边界。 | Harness/Agent Tool contract | ISS-014, ACI, tool_call_id, evidence_status, RAG, query_logs, query_mysql, MOCK | openspec/changes/archive/2026-07-21-single-react-aci-tool-contracts | archived |
|
||||||
|
|||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Acceptance: rag-hybrid-search-rrf
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
Hybrid mode implemented on KnowledgeSearchPort:
|
||||||
|
|
||||||
|
- dense unfiltered + dense filtered + lexical rank over union
|
||||||
|
- RRF fusion by evidenceKey
|
||||||
|
- dense-compatible score preserved for thresholds
|
||||||
|
- default mode remains dense
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
```text
|
||||||
|
mvn -q "-Dtest=RrfFusionTest,VectorKnowledgeSearchAdapterHybridTest,LookupKnowledgeToolTest" test
|
||||||
|
```
|
||||||
|
|
||||||
|
Pass.
|
||||||
|
|
||||||
|
## Residual
|
||||||
|
|
||||||
|
- True Milvus BM25/sparse schema + reindex still follow-up
|
||||||
|
- Lexical path only ranks dense-recalled candidates (does not expand pure-term misses outside dense topK)
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
# Brief: rag-hybrid-search-rrf
|
||||||
|
|
||||||
|
Delivery 2 after chunk identity. Enable hybrid multi-path + RRF on KnowledgeSearchPort without legacy SDK hybrid API. True BM25 schema rebuild is staged follow-up; this change ships sparse-lite lexical ranking over dense candidate union + filtered/unfiltered dense fusion.
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# Decisions: rag-hybrid-search-rrf
|
||||||
|
|
||||||
|
## Capability
|
||||||
|
|
||||||
|
sm-flow + OpenSpec fallback; apply pre-authorized.
|
||||||
|
|
||||||
|
## Depends
|
||||||
|
|
||||||
|
Delivery 1 archived.
|
||||||
|
|
||||||
|
## Grill (compressed, pre-authorized)
|
||||||
|
|
||||||
|
- Q: Full BM25 schema now? A: No — sparse-lite + RRF first; schema rebuild follow-up.
|
||||||
|
- Q: Default mode? A: dense default; hybrid opt-in.
|
||||||
|
- Q: Threshold score? A: keep dense-compatible L2 mapping.
|
||||||
|
|
||||||
|
## Design
|
||||||
|
|
||||||
|
Hybrid paths: dense unfiltered + dense filtered + lexical rank over union; RRF fuse by evidenceKey.
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Evidence
|
||||||
|
|
||||||
|
- Delivery 1 identity/port foundation required
|
||||||
|
- RRF utility and hybrid adapter unit tests green
|
||||||
|
- Lexical sparse-lite intentionally intermediate until BM25 schema
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
ready: 2026-07-27
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
committed: 2026-07-27
|
||||||
|
change: rag-hybrid-search-rrf
|
||||||
|
authorized-apply: user-preauthorized-sm-flow
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
schema: spec-driven
|
||||||
|
created: 2026-07-27
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
# Design: rag-hybrid-search-rrf
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
- Delivery 1 archived: chunk evidenceKey, SearchPort, return caps.
|
||||||
|
- Legacy SDK path will be abandoned later; hybrid must live on SearchPort.
|
||||||
|
- Full sparse schema rebuild is operationally heavy; ship fusion first.
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
### D1. Hybrid = multi-path + RRF on SearchPort
|
||||||
|
|
||||||
|
When `retrieval.search.mode=hybrid`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
paths:
|
||||||
|
1) dense(query, filter=null) # always
|
||||||
|
2) dense(query, filter=category) # if category present
|
||||||
|
3) lexical rank over union of dense hits # sparse-lite
|
||||||
|
fuse by evidenceKey using RRF(k)
|
||||||
|
return topK fused hits
|
||||||
|
```
|
||||||
|
|
||||||
|
### D2. RRF formula
|
||||||
|
|
||||||
|
```text
|
||||||
|
score(d) = Σ w_i / (k + rank_i(d))
|
||||||
|
```
|
||||||
|
|
||||||
|
Defaults: k=60, all w_i=1.0. Optional weights via config.
|
||||||
|
|
||||||
|
### D3. Score semantics
|
||||||
|
|
||||||
|
- `KnowledgeSearchHit.score` remains **compatible L2 distance from best dense hit** for post-process thresholds.
|
||||||
|
- Fused RRF score is carried in metadata (`fusedScore`, `fusionRanks`) for trace, not as L2.
|
||||||
|
|
||||||
|
### D4. Lexical path (sparse-lite)
|
||||||
|
|
||||||
|
Until BM25 schema:
|
||||||
|
|
||||||
|
- Tokenize query (simple whitespace / non-alnum split, lower-case)
|
||||||
|
- Score each candidate by term coverage over title+breadcrumb+content
|
||||||
|
- Rank candidates for RRF path only
|
||||||
|
- Not a replacement for true BM25 inverted index
|
||||||
|
|
||||||
|
### D5. Serial filter retry
|
||||||
|
|
||||||
|
Lookup tool keeps low-quality unfiltered retry as safety net even in hybrid, because hybrid already includes unfiltered dense; retry remains cheap no-op when already fused well.
|
||||||
|
|
||||||
|
## Non-goals now
|
||||||
|
|
||||||
|
- New Milvus collection fields
|
||||||
|
- Reindex jobs
|
||||||
|
- SDK hybridSearch API calls
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
- Lexical path only ranks already-recalled dense candidates → does not expand pure-term misses outside dense topK. Acceptable intermediate; true BM25 later expands recall.
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# Change: RAG hybrid search with RRF
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
Delivery 1 fixed chunk identity/dedup. Delivery 2 enables multi-path retrieval fused by RRF so category filters no longer whole-replace good results and lexical signals can complement dense ranks.
|
||||||
|
|
||||||
|
True Milvus sparse/BM25 schema migration remains staged: this change ships a **working hybrid mode on KnowledgeSearchPort** using multi-path dense (+ lexical rank path) and RRF, without thickening the legacy SDK API surface.
|
||||||
|
|
||||||
|
## What Changes
|
||||||
|
|
||||||
|
- `retrieval.search.mode=dense|hybrid` (default dense)
|
||||||
|
- Hybrid search on `KnowledgeSearchPort`:
|
||||||
|
- dense unfiltered path
|
||||||
|
- dense filtered path when category present
|
||||||
|
- lexical rank path over the candidate union (sparse-lite until BM25 schema lands)
|
||||||
|
- RRF / optional weighted RRF fusion by evidenceKey
|
||||||
|
- Preserve dense L2 scores for quality thresholds; RRF only orders
|
||||||
|
- Lookup tool uses search mode; filter low-quality retry remains as safety net
|
||||||
|
- Config for rrf-k and path weights
|
||||||
|
- Tests for RRF fusion ordering and hybrid adapter behavior
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- Full Milvus BM25/sparse collection rebuild (documented follow-up)
|
||||||
|
- Deleting legacy SDK mode entirely
|
||||||
|
- Cross-encoder model rerank
|
||||||
|
- Changing Agent tool schema name/input
|
||||||
|
|
||||||
|
## Capabilities
|
||||||
|
|
||||||
|
### New Capabilities
|
||||||
|
|
||||||
|
- `rag-hybrid-search`: hybrid mode, multi-path recall, RRF fusion via search port
|
||||||
|
|
||||||
|
### Modified Capabilities
|
||||||
|
|
||||||
|
- `rag-chunk-evidence-identity`: hybrid hits must keep chunk identity
|
||||||
|
- `rag-knowledge-retrieval`: filtered/unfiltered cooperation via fusion rather than only serial replace
|
||||||
|
|
||||||
|
## Impact
|
||||||
|
|
||||||
|
- Code: retrieval package, lookup wiring/config, tests
|
||||||
|
- Runtime default remains dense unless hybrid enabled
|
||||||
|
- Interface: L2 internal (search port behavior)
|
||||||
|
|
||||||
|
## Depends on
|
||||||
|
|
||||||
|
- Archived Delivery 1: `rag-chunk-evidence-identity-dedup`
|
||||||
+44
@@ -0,0 +1,44 @@
|
|||||||
|
# rag-hybrid-search Specification
|
||||||
|
|
||||||
|
## ADDED Requirements
|
||||||
|
|
||||||
|
### Requirement: Knowledge search SHALL support configurable dense and hybrid modes
|
||||||
|
|
||||||
|
The knowledge search port SHALL support `retrieval.search.mode` values `dense` and `hybrid`. Default SHALL be `dense` for backward-compatible behavior.
|
||||||
|
|
||||||
|
#### Scenario: Dense mode single path
|
||||||
|
|
||||||
|
- **WHEN** mode is dense
|
||||||
|
- **THEN** the port SHALL perform a single dense vector search with the provided category filter
|
||||||
|
|
||||||
|
#### Scenario: Hybrid mode multi-path fusion
|
||||||
|
|
||||||
|
- **WHEN** mode is hybrid
|
||||||
|
- **THEN** the port SHALL gather multiple retrieval paths and fuse them by RRF before returning hits
|
||||||
|
|
||||||
|
### Requirement: Hybrid fusion SHALL use RRF over evidence identities
|
||||||
|
|
||||||
|
Hybrid fusion SHALL score candidates by reciprocal rank fusion on evidenceKey identities and SHALL NOT require comparable raw scores across paths.
|
||||||
|
|
||||||
|
#### Scenario: Multi-path candidate rises with RRF
|
||||||
|
|
||||||
|
- **WHEN** a chunk ranks highly on more than one hybrid path
|
||||||
|
- **THEN** its fused rank SHALL improve relative to a chunk that ranks highly on only one path
|
||||||
|
|
||||||
|
### Requirement: Hybrid SHALL preserve dense quality scores for thresholds
|
||||||
|
|
||||||
|
Fused ordering SHALL NOT replace the dense compatibility score used by post-process quality thresholds.
|
||||||
|
|
||||||
|
#### Scenario: Threshold score remains dense-compatible
|
||||||
|
|
||||||
|
- **WHEN** hybrid returns a hit that originated from dense search
|
||||||
|
- **THEN** hit.score SHALL remain a dense-compatible distance/similarity mapping usable by existing normalizeL2 thresholds
|
||||||
|
|
||||||
|
### Requirement: Hybrid SHALL keep chunk identity fields
|
||||||
|
|
||||||
|
Hybrid hits SHALL populate docId, chunkIndex, and evidenceKey consistently with Delivery 1 identity rules.
|
||||||
|
|
||||||
|
#### Scenario: Fused hit keeps evidenceKey
|
||||||
|
|
||||||
|
- **WHEN** hybrid merges the same chunk from two paths
|
||||||
|
- **THEN** the returned hit SHALL use one evidenceKey and retain chunk identity metadata
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
# Tasks: rag-hybrid-search-rrf
|
||||||
|
|
||||||
|
- [x] 1. Add RRF fusion utility
|
||||||
|
- [x] 2. Extend KnowledgeSearchRequest/Hit metadata for fusion ranks
|
||||||
|
- [x] 3. Implement hybrid multi-path search in VectorKnowledgeSearchAdapter
|
||||||
|
- [x] 4. Wire retrieval.search.mode and rrf config
|
||||||
|
- [x] 5. Pass mode from LookupKnowledgeTool / retriever
|
||||||
|
- [x] 6. Unit tests for RRF and hybrid adapter ordering
|
||||||
|
- [x] 7. Verify dense mode regression tests still pass
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# rag-hybrid-search Specification
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
TBD - created by archiving change rag-hybrid-search-rrf. Update Purpose after archive.
|
||||||
|
## Requirements
|
||||||
|
### Requirement: Knowledge search SHALL support configurable dense and hybrid modes
|
||||||
|
|
||||||
|
The knowledge search port SHALL support `retrieval.search.mode` values `dense` and `hybrid`. Default SHALL be `dense` for backward-compatible behavior.
|
||||||
|
|
||||||
|
#### Scenario: Dense mode single path
|
||||||
|
|
||||||
|
- **WHEN** mode is dense
|
||||||
|
- **THEN** the port SHALL perform a single dense vector search with the provided category filter
|
||||||
|
|
||||||
|
#### Scenario: Hybrid mode multi-path fusion
|
||||||
|
|
||||||
|
- **WHEN** mode is hybrid
|
||||||
|
- **THEN** the port SHALL gather multiple retrieval paths and fuse them by RRF before returning hits
|
||||||
|
|
||||||
|
### Requirement: Hybrid fusion SHALL use RRF over evidence identities
|
||||||
|
|
||||||
|
Hybrid fusion SHALL score candidates by reciprocal rank fusion on evidenceKey identities and SHALL NOT require comparable raw scores across paths.
|
||||||
|
|
||||||
|
#### Scenario: Multi-path candidate rises with RRF
|
||||||
|
|
||||||
|
- **WHEN** a chunk ranks highly on more than one hybrid path
|
||||||
|
- **THEN** its fused rank SHALL improve relative to a chunk that ranks highly on only one path
|
||||||
|
|
||||||
|
### Requirement: Hybrid SHALL preserve dense quality scores for thresholds
|
||||||
|
|
||||||
|
Fused ordering SHALL NOT replace the dense compatibility score used by post-process quality thresholds.
|
||||||
|
|
||||||
|
#### Scenario: Threshold score remains dense-compatible
|
||||||
|
|
||||||
|
- **WHEN** hybrid returns a hit that originated from dense search
|
||||||
|
- **THEN** hit.score SHALL remain a dense-compatible distance/similarity mapping usable by existing normalizeL2 thresholds
|
||||||
|
|
||||||
|
### Requirement: Hybrid SHALL keep chunk identity fields
|
||||||
|
|
||||||
|
Hybrid hits SHALL populate docId, chunkIndex, and evidenceKey consistently with Delivery 1 identity rules.
|
||||||
|
|
||||||
|
#### Scenario: Fused hit keeps evidenceKey
|
||||||
|
|
||||||
|
- **WHEN** hybrid merges the same chunk from two paths
|
||||||
|
- **THEN** the returned hit SHALL use one evidenceKey and retain chunk identity metadata
|
||||||
|
|
||||||
@@ -3,12 +3,15 @@ package com.superbiz.agent.service;
|
|||||||
import com.superbiz.agent.dto.RetrievalTrace;
|
import com.superbiz.agent.dto.RetrievalTrace;
|
||||||
import com.superbiz.agent.dto.RetrievedEvidenceCandidate;
|
import com.superbiz.agent.dto.RetrievedEvidenceCandidate;
|
||||||
import com.superbiz.agent.service.retrieval.KnowledgeSearchHit;
|
import com.superbiz.agent.service.retrieval.KnowledgeSearchHit;
|
||||||
|
import com.superbiz.agent.service.retrieval.KnowledgeSearchMode;
|
||||||
import com.superbiz.agent.service.retrieval.KnowledgeSearchPort;
|
import com.superbiz.agent.service.retrieval.KnowledgeSearchPort;
|
||||||
import com.superbiz.agent.service.retrieval.KnowledgeSearchRequest;
|
import com.superbiz.agent.service.retrieval.KnowledgeSearchRequest;
|
||||||
|
import org.springframework.beans.factory.annotation.Value;
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import java.util.Locale;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* L1 向量检索适配器。
|
* L1 向量检索适配器。
|
||||||
@@ -23,6 +26,9 @@ public class KnowledgeDocumentRetriever {
|
|||||||
|
|
||||||
private final KnowledgeSearchPort knowledgeSearchPort;
|
private final KnowledgeSearchPort knowledgeSearchPort;
|
||||||
|
|
||||||
|
@Value("${retrieval.search.mode:dense}")
|
||||||
|
private String searchMode = "dense";
|
||||||
|
|
||||||
public KnowledgeDocumentRetriever(KnowledgeSearchPort knowledgeSearchPort) {
|
public KnowledgeDocumentRetriever(KnowledgeSearchPort knowledgeSearchPort) {
|
||||||
this.knowledgeSearchPort = knowledgeSearchPort;
|
this.knowledgeSearchPort = knowledgeSearchPort;
|
||||||
}
|
}
|
||||||
@@ -39,8 +45,11 @@ public class KnowledgeDocumentRetriever {
|
|||||||
public RetrievalAttemptResult retrieve(String attemptName, String query, String categoryFilter, int topK) {
|
public RetrievalAttemptResult retrieve(String attemptName, String query, String categoryFilter, int topK) {
|
||||||
long start = System.currentTimeMillis();
|
long start = System.currentTimeMillis();
|
||||||
try {
|
try {
|
||||||
|
KnowledgeSearchMode mode = "hybrid".equalsIgnoreCase(trim(searchMode))
|
||||||
|
? KnowledgeSearchMode.HYBRID
|
||||||
|
: KnowledgeSearchMode.DENSE;
|
||||||
List<KnowledgeSearchHit> hits = knowledgeSearchPort.search(
|
List<KnowledgeSearchHit> hits = knowledgeSearchPort.search(
|
||||||
KnowledgeSearchRequest.dense(query, topK, categoryFilter));
|
new KnowledgeSearchRequest(query, topK, categoryFilter, mode));
|
||||||
List<RetrievedEvidenceCandidate> candidates = toCandidates(attemptName, hits);
|
List<RetrievedEvidenceCandidate> candidates = toCandidates(attemptName, hits);
|
||||||
return new RetrievalAttemptResult(
|
return new RetrievalAttemptResult(
|
||||||
attempt(attemptName, query, categoryFilter, candidates.size(), null,
|
attempt(attemptName, query, categoryFilter, candidates.size(), null,
|
||||||
@@ -110,6 +119,10 @@ public class KnowledgeDocumentRetriever {
|
|||||||
return hits.get(0).score();
|
return hits.get(0).score();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static String trim(String value) {
|
||||||
|
return value == null ? "" : value.trim().toLowerCase(Locale.ROOT);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 单次检索 attempt 的结果包。
|
* 单次检索 attempt 的结果包。
|
||||||
*/
|
*/
|
||||||
|
|||||||
@@ -0,0 +1,73 @@
|
|||||||
|
package com.superbiz.agent.service.retrieval;
|
||||||
|
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.LinkedHashSet;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Locale;
|
||||||
|
import java.util.Set;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Sparse-lite lexical ranking over already recalled candidates.
|
||||||
|
* Not a substitute for inverted-index BM25; expands ordering signal only.
|
||||||
|
*/
|
||||||
|
public final class LexicalRanker {
|
||||||
|
|
||||||
|
private LexicalRanker() {
|
||||||
|
}
|
||||||
|
|
||||||
|
public static List<KnowledgeSearchHit> rank(String query, List<KnowledgeSearchHit> candidates) {
|
||||||
|
if (candidates == null || candidates.isEmpty()) {
|
||||||
|
return List.of();
|
||||||
|
}
|
||||||
|
Set<String> terms = tokenize(query);
|
||||||
|
if (terms.isEmpty()) {
|
||||||
|
return List.copyOf(candidates);
|
||||||
|
}
|
||||||
|
List<ScoredHit> scored = new ArrayList<>(candidates.size());
|
||||||
|
for (KnowledgeSearchHit hit : candidates) {
|
||||||
|
String haystack = (nullToEmpty(hit.title()) + " "
|
||||||
|
+ nullToEmpty(hit.breadcrumb()) + " "
|
||||||
|
+ nullToEmpty(hit.content())).toLowerCase(Locale.ROOT);
|
||||||
|
int hits = 0;
|
||||||
|
for (String term : terms) {
|
||||||
|
if (haystack.contains(term)) {
|
||||||
|
hits++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
double coverage = hits / (double) terms.size();
|
||||||
|
scored.add(new ScoredHit(hit, coverage, hits));
|
||||||
|
}
|
||||||
|
scored.sort(Comparator
|
||||||
|
.comparingDouble((ScoredHit s) -> s.coverage).reversed()
|
||||||
|
.thenComparingInt((ScoredHit s) -> s.hits).reversed()
|
||||||
|
.thenComparingInt(s -> s.hit.originalRank()));
|
||||||
|
return scored.stream().map(s -> s.hit).toList();
|
||||||
|
}
|
||||||
|
|
||||||
|
static Set<String> tokenize(String query) {
|
||||||
|
if (query == null || query.isBlank()) {
|
||||||
|
return Set.of();
|
||||||
|
}
|
||||||
|
String normalized = query.toLowerCase(Locale.ROOT);
|
||||||
|
String[] parts = normalized.split("[^\\p{IsAlphabetic}\\p{IsDigit}]+");
|
||||||
|
Set<String> terms = new LinkedHashSet<>();
|
||||||
|
for (String part : parts) {
|
||||||
|
if (part == null) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
String term = part.trim();
|
||||||
|
if (term.length() >= 2) {
|
||||||
|
terms.add(term);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return terms;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static String nullToEmpty(String value) {
|
||||||
|
return value == null ? "" : value;
|
||||||
|
}
|
||||||
|
|
||||||
|
private record ScoredHit(KnowledgeSearchHit hit, double coverage, int hits) {
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
package com.superbiz.agent.service.retrieval;
|
||||||
|
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.LinkedHashMap;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Objects;
|
||||||
|
import java.util.function.Function;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Reciprocal Rank Fusion helpers.
|
||||||
|
*
|
||||||
|
* <pre>
|
||||||
|
* RRF_w(d) = Σ w_i / (k + rank_i(d))
|
||||||
|
* </pre>
|
||||||
|
*/
|
||||||
|
public final class RrfFusion {
|
||||||
|
|
||||||
|
private RrfFusion() {
|
||||||
|
}
|
||||||
|
|
||||||
|
public static <T> List<Scored<T>> fuse(List<RankedPath<T>> paths,
|
||||||
|
int rrfK,
|
||||||
|
Function<T, String> identityFn) {
|
||||||
|
if (paths == null || paths.isEmpty()) {
|
||||||
|
return List.of();
|
||||||
|
}
|
||||||
|
int k = Math.max(1, rrfK);
|
||||||
|
Map<String, Acc<T>> acc = new LinkedHashMap<>();
|
||||||
|
for (RankedPath<T> path : paths) {
|
||||||
|
if (path == null || path.items() == null || path.items().isEmpty()) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
double weight = path.weight() <= 0 ? 1.0 : path.weight();
|
||||||
|
List<T> items = path.items();
|
||||||
|
for (int i = 0; i < items.size(); i++) {
|
||||||
|
T item = items.get(i);
|
||||||
|
if (item == null) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
String id = identityFn.apply(item);
|
||||||
|
if (id == null || id.isBlank()) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
int rank = i + 1;
|
||||||
|
double contrib = weight / (k + rank);
|
||||||
|
Acc<T> bucket = acc.computeIfAbsent(id, ignored -> new Acc<>(item));
|
||||||
|
bucket.score += contrib;
|
||||||
|
bucket.ranks.put(path.name(), rank);
|
||||||
|
// Prefer first-seen item payload; callers should put preferred path first if needed.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
List<Scored<T>> scored = new ArrayList<>(acc.size());
|
||||||
|
for (Map.Entry<String, Acc<T>> entry : acc.entrySet()) {
|
||||||
|
Acc<T> value = entry.getValue();
|
||||||
|
scored.add(new Scored<>(entry.getKey(), value.item, value.score, Map.copyOf(value.ranks)));
|
||||||
|
}
|
||||||
|
scored.sort(Comparator
|
||||||
|
.comparingDouble((Scored<T> s) -> s.rrfScore()).reversed()
|
||||||
|
.thenComparing(Scored::identity));
|
||||||
|
return scored;
|
||||||
|
}
|
||||||
|
|
||||||
|
public record RankedPath<T>(String name, List<T> items, double weight) {
|
||||||
|
public RankedPath {
|
||||||
|
Objects.requireNonNull(name, "name");
|
||||||
|
items = items == null ? List.of() : List.copyOf(items);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public record Scored<T>(String identity, T item, double rrfScore, Map<String, Integer> ranks) {
|
||||||
|
}
|
||||||
|
|
||||||
|
private static final class Acc<T> {
|
||||||
|
private final T item;
|
||||||
|
private double score;
|
||||||
|
private final Map<String, Integer> ranks = new HashMap<>();
|
||||||
|
|
||||||
|
private Acc(T item) {
|
||||||
|
this.item = item;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
+123
-6
@@ -2,15 +2,22 @@ package com.superbiz.agent.service.retrieval;
|
|||||||
|
|
||||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
import com.superbiz.agent.service.VectorSearchService;
|
import com.superbiz.agent.service.VectorSearchService;
|
||||||
|
import org.springframework.beans.factory.annotation.Value;
|
||||||
import org.springframework.stereotype.Component;
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.LinkedHashMap;
|
import java.util.LinkedHashMap;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import java.util.Locale;
|
||||||
import java.util.Map;
|
import java.util.Map;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Dense-only {@link KnowledgeSearchPort} backed by the existing vector search facade.
|
* {@link KnowledgeSearchPort} backed by the existing vector search facade.
|
||||||
|
*
|
||||||
|
* <ul>
|
||||||
|
* <li>{@code DENSE}: single dense path (legacy behavior)</li>
|
||||||
|
* <li>{@code HYBRID}: dense unfiltered + optional dense filtered + lexical rank, fused by RRF</li>
|
||||||
|
* </ul>
|
||||||
*/
|
*/
|
||||||
@Component
|
@Component
|
||||||
public class VectorKnowledgeSearchAdapter implements KnowledgeSearchPort {
|
public class VectorKnowledgeSearchAdapter implements KnowledgeSearchPort {
|
||||||
@@ -18,6 +25,21 @@ public class VectorKnowledgeSearchAdapter implements KnowledgeSearchPort {
|
|||||||
private final VectorSearchService vectorSearchService;
|
private final VectorSearchService vectorSearchService;
|
||||||
private final ObjectMapper objectMapper;
|
private final ObjectMapper objectMapper;
|
||||||
|
|
||||||
|
@Value("${retrieval.search.mode:dense}")
|
||||||
|
private String configuredMode = "dense";
|
||||||
|
|
||||||
|
@Value("${retrieval.hybrid.rrf-k:60}")
|
||||||
|
private int rrfK = 60;
|
||||||
|
|
||||||
|
@Value("${retrieval.hybrid.weight.dense-unfiltered:1.0}")
|
||||||
|
private double weightDenseUnfiltered = 1.0;
|
||||||
|
|
||||||
|
@Value("${retrieval.hybrid.weight.dense-filtered:1.0}")
|
||||||
|
private double weightDenseFiltered = 1.0;
|
||||||
|
|
||||||
|
@Value("${retrieval.hybrid.weight.lexical:1.0}")
|
||||||
|
private double weightLexical = 1.0;
|
||||||
|
|
||||||
public VectorKnowledgeSearchAdapter(VectorSearchService vectorSearchService, ObjectMapper objectMapper) {
|
public VectorKnowledgeSearchAdapter(VectorSearchService vectorSearchService, ObjectMapper objectMapper) {
|
||||||
this.vectorSearchService = vectorSearchService;
|
this.vectorSearchService = vectorSearchService;
|
||||||
this.objectMapper = objectMapper;
|
this.objectMapper = objectMapper;
|
||||||
@@ -25,13 +47,101 @@ public class VectorKnowledgeSearchAdapter implements KnowledgeSearchPort {
|
|||||||
|
|
||||||
@Override
|
@Override
|
||||||
public List<KnowledgeSearchHit> search(KnowledgeSearchRequest request) {
|
public List<KnowledgeSearchHit> search(KnowledgeSearchRequest request) {
|
||||||
if (request.mode() == KnowledgeSearchMode.HYBRID) {
|
KnowledgeSearchMode mode = resolveMode(request.mode());
|
||||||
// Delivery 2 will implement true hybrid. Until then, fall back to dense.
|
if (mode == KnowledgeSearchMode.HYBRID) {
|
||||||
|
return searchHybrid(request);
|
||||||
}
|
}
|
||||||
|
return searchDense(request.query(), request.topK(), request.categoryFilter());
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<KnowledgeSearchHit> searchDense(String query, int topK, String categoryFilter) {
|
||||||
List<VectorSearchService.SearchResult> results = vectorSearchService.searchSimilarDocuments(
|
List<VectorSearchService.SearchResult> results = vectorSearchService.searchSimilarDocuments(
|
||||||
request.query(),
|
query, topK, categoryFilter);
|
||||||
request.topK(),
|
return toHits(results);
|
||||||
request.categoryFilter());
|
}
|
||||||
|
|
||||||
|
private List<KnowledgeSearchHit> searchHybrid(KnowledgeSearchRequest request) {
|
||||||
|
String query = request.query();
|
||||||
|
int topK = request.topK();
|
||||||
|
String category = trimToNull(request.categoryFilter());
|
||||||
|
|
||||||
|
List<KnowledgeSearchHit> unfiltered = searchDense(query, topK, null);
|
||||||
|
List<KnowledgeSearchHit> filtered = category == null
|
||||||
|
? List.of()
|
||||||
|
: searchDense(query, topK, category);
|
||||||
|
|
||||||
|
Map<String, KnowledgeSearchHit> unionByKey = new LinkedHashMap<>();
|
||||||
|
for (KnowledgeSearchHit hit : unfiltered) {
|
||||||
|
unionByKey.putIfAbsent(hit.evidenceKey(), hit);
|
||||||
|
}
|
||||||
|
for (KnowledgeSearchHit hit : filtered) {
|
||||||
|
unionByKey.putIfAbsent(hit.evidenceKey(), hit);
|
||||||
|
}
|
||||||
|
List<KnowledgeSearchHit> union = new ArrayList<>(unionByKey.values());
|
||||||
|
List<KnowledgeSearchHit> lexical = LexicalRanker.rank(query, union);
|
||||||
|
|
||||||
|
List<RrfFusion.RankedPath<KnowledgeSearchHit>> paths = new ArrayList<>();
|
||||||
|
paths.add(new RrfFusion.RankedPath<>("dense_unfiltered", unfiltered, weightDenseUnfiltered));
|
||||||
|
if (!filtered.isEmpty()) {
|
||||||
|
paths.add(new RrfFusion.RankedPath<>("dense_filtered", filtered, weightDenseFiltered));
|
||||||
|
}
|
||||||
|
if (!lexical.isEmpty()) {
|
||||||
|
paths.add(new RrfFusion.RankedPath<>("lexical", lexical, weightLexical));
|
||||||
|
}
|
||||||
|
|
||||||
|
List<RrfFusion.Scored<KnowledgeSearchHit>> fused = RrfFusion.fuse(
|
||||||
|
paths, rrfK, KnowledgeSearchHit::evidenceKey);
|
||||||
|
|
||||||
|
List<KnowledgeSearchHit> ordered = new ArrayList<>();
|
||||||
|
int rank = 1;
|
||||||
|
for (RrfFusion.Scored<KnowledgeSearchHit> scored : fused) {
|
||||||
|
if (ordered.size() >= topK) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
KnowledgeSearchHit base = scored.item();
|
||||||
|
Map<String, String> metadata = new LinkedHashMap<>(
|
||||||
|
base.metadata() == null ? Map.of() : base.metadata());
|
||||||
|
metadata.put("fusedScore", Double.toString(scored.rrfScore()));
|
||||||
|
metadata.put("fusionRanks", scored.ranks().toString());
|
||||||
|
metadata.put("fusionRank", Integer.toString(rank));
|
||||||
|
ordered.add(new KnowledgeSearchHit(
|
||||||
|
base.id(),
|
||||||
|
base.content(),
|
||||||
|
base.score(),
|
||||||
|
base.rawScore(),
|
||||||
|
base.scoreLabel(),
|
||||||
|
base.metadataJson(),
|
||||||
|
metadata,
|
||||||
|
base.docId(),
|
||||||
|
base.chunkIndex(),
|
||||||
|
base.evidenceKey(),
|
||||||
|
base.source(),
|
||||||
|
base.title(),
|
||||||
|
base.breadcrumb(),
|
||||||
|
rank
|
||||||
|
));
|
||||||
|
rank++;
|
||||||
|
}
|
||||||
|
return ordered;
|
||||||
|
}
|
||||||
|
|
||||||
|
private KnowledgeSearchMode resolveMode(KnowledgeSearchMode requestMode) {
|
||||||
|
if (requestMode == KnowledgeSearchMode.HYBRID) {
|
||||||
|
return KnowledgeSearchMode.HYBRID;
|
||||||
|
}
|
||||||
|
if (requestMode == KnowledgeSearchMode.DENSE) {
|
||||||
|
// Allow global config to force hybrid even if caller passes DENSE default.
|
||||||
|
String configured = configuredMode == null ? "dense" : configuredMode.trim().toLowerCase(Locale.ROOT);
|
||||||
|
if ("hybrid".equals(configured)) {
|
||||||
|
return KnowledgeSearchMode.HYBRID;
|
||||||
|
}
|
||||||
|
return KnowledgeSearchMode.DENSE;
|
||||||
|
}
|
||||||
|
String configured = configuredMode == null ? "dense" : configuredMode.trim().toLowerCase(Locale.ROOT);
|
||||||
|
return "hybrid".equals(configured) ? KnowledgeSearchMode.HYBRID : KnowledgeSearchMode.DENSE;
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<KnowledgeSearchHit> toHits(List<VectorSearchService.SearchResult> results) {
|
||||||
if (results == null || results.isEmpty()) {
|
if (results == null || results.isEmpty()) {
|
||||||
return List.of();
|
return List.of();
|
||||||
}
|
}
|
||||||
@@ -91,4 +201,11 @@ public class VectorKnowledgeSearchAdapter implements KnowledgeSearchPort {
|
|||||||
return Map.of();
|
return Map.of();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static String trimToNull(String value) {
|
||||||
|
if (value == null || value.isBlank()) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return value.trim();
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,50 @@
|
|||||||
|
package com.superbiz.agent.service.retrieval;
|
||||||
|
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||||
|
|
||||||
|
class RrfFusionTest {
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void multiPathAgreementOutranksSinglePathHead() {
|
||||||
|
List<String> dense = List.of("a", "b", "c");
|
||||||
|
List<String> lexical = List.of("c", "b", "d");
|
||||||
|
|
||||||
|
List<RrfFusion.Scored<String>> fused = RrfFusion.fuse(
|
||||||
|
List.of(
|
||||||
|
new RrfFusion.RankedPath<>("dense", dense, 1.0),
|
||||||
|
new RrfFusion.RankedPath<>("lexical", lexical, 1.0)
|
||||||
|
),
|
||||||
|
60,
|
||||||
|
s -> s
|
||||||
|
);
|
||||||
|
|
||||||
|
// c: dense#3 + lexical#1 ; b: dense#2 + lexical#2 ; a: dense#1 only
|
||||||
|
// With k=60, c edges b slightly, and both beat single-path a.
|
||||||
|
assertEquals("c", fused.get(0).identity());
|
||||||
|
assertEquals("b", fused.get(1).identity());
|
||||||
|
assertEquals("a", fused.get(2).identity());
|
||||||
|
assertTrue(fused.get(0).rrfScore() > fused.get(2).rrfScore());
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void pathWeightCanElevateSecondaryPath() {
|
||||||
|
List<String> dense = List.of("a", "b");
|
||||||
|
List<String> lexical = List.of("b", "a");
|
||||||
|
|
||||||
|
List<RrfFusion.Scored<String>> fused = RrfFusion.fuse(
|
||||||
|
List.of(
|
||||||
|
new RrfFusion.RankedPath<>("dense", dense, 1.0),
|
||||||
|
new RrfFusion.RankedPath<>("lexical", lexical, 2.0)
|
||||||
|
),
|
||||||
|
60,
|
||||||
|
s -> s
|
||||||
|
);
|
||||||
|
|
||||||
|
assertEquals("b", fused.get(0).identity());
|
||||||
|
}
|
||||||
|
}
|
||||||
+53
@@ -0,0 +1,53 @@
|
|||||||
|
package com.superbiz.agent.service.retrieval;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import com.superbiz.agent.service.VectorSearchService;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.springframework.test.util.ReflectionTestUtils;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||||
|
import static org.mockito.Mockito.mock;
|
||||||
|
import static org.mockito.Mockito.when;
|
||||||
|
|
||||||
|
class VectorKnowledgeSearchAdapterHybridTest {
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void hybridFusesFilteredAndUnfilteredDensePaths() {
|
||||||
|
VectorSearchService vectorSearchService = mock(VectorSearchService.class);
|
||||||
|
when(vectorSearchService.searchSimilarDocuments("pool timeout", 3, null)).thenReturn(List.of(
|
||||||
|
result("u1", "{\"_source\":\"a.md\",\"docId\":\"a\",\"chunkIndex\":0,\"title\":\"generic\"}", "generic pool", 0.4f),
|
||||||
|
result("u2", "{\"_source\":\"b.md\",\"docId\":\"b\",\"chunkIndex\":0,\"title\":\"other\"}", "other", 0.5f)
|
||||||
|
));
|
||||||
|
when(vectorSearchService.searchSimilarDocuments("pool timeout", 3, "mysql")).thenReturn(List.of(
|
||||||
|
result("f1", "{\"_source\":\"c.md\",\"docId\":\"c\",\"chunkIndex\":0,\"title\":\"mysql pool timeout\"}", "mysql pool timeout runbook", 0.35f)
|
||||||
|
));
|
||||||
|
|
||||||
|
VectorKnowledgeSearchAdapter adapter = new VectorKnowledgeSearchAdapter(vectorSearchService, new ObjectMapper());
|
||||||
|
ReflectionTestUtils.setField(adapter, "configuredMode", "hybrid");
|
||||||
|
ReflectionTestUtils.setField(adapter, "rrfK", 60);
|
||||||
|
ReflectionTestUtils.setField(adapter, "weightDenseUnfiltered", 1.0);
|
||||||
|
ReflectionTestUtils.setField(adapter, "weightDenseFiltered", 1.0);
|
||||||
|
ReflectionTestUtils.setField(adapter, "weightLexical", 1.0);
|
||||||
|
|
||||||
|
List<KnowledgeSearchHit> hits = adapter.search(
|
||||||
|
new KnowledgeSearchRequest("pool timeout", 3, "mysql", KnowledgeSearchMode.HYBRID));
|
||||||
|
|
||||||
|
assertEquals(3, hits.size());
|
||||||
|
assertTrue(hits.stream().anyMatch(hit -> "c#chunk-0".equals(hit.evidenceKey())));
|
||||||
|
assertTrue(hits.get(0).metadata().containsKey("fusedScore"));
|
||||||
|
}
|
||||||
|
|
||||||
|
private static VectorSearchService.SearchResult result(String id, String metadata, String content, float score) {
|
||||||
|
VectorSearchService.SearchResult result = new VectorSearchService.SearchResult();
|
||||||
|
result.setId(id);
|
||||||
|
result.setMetadata(metadata);
|
||||||
|
result.setContent(content);
|
||||||
|
result.setScore(score);
|
||||||
|
result.setRawScore((double) score);
|
||||||
|
result.setScoreLabel("l2_distance");
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user