feat(rag): dense+BM25 hybrid on MilvusClientV2, drop SDK path
Replace legacy MilvusServiceClient knowledge search/write with a single MilvusClientV2 hybrid store (BM25 function + dense ANN + RRFRanker). Use collection biz_hybrid and require knowledge reindex.
This commit is contained in:
zhuyongxin
@@ -12,6 +12,7 @@
|
||||
| 日期 | slug | 说明 | 领域 | 关键词 | 关联 OpenSpec | 状态 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 2026-07-27 | rag-chunk-evidence-identity-dedup | chunk 级证据身份、去重、retrieve-k/return-n 与 SearchPort 地基，为 hybrid 铺路。 | RAG/证据身份/去重 | evidenceKey, maxChunksPerDocument, retrieve-k, return-n, KnowledgeSearchPort, document_id chunk-scoped | openspec/changes/archive/2026-07-27-rag-chunk-evidence-identity-dedup | archived |
|
||||
| 2026-07-27 | rag-bm25-hybrid-drop-sdk | 真 dense+BM25 hybrid（MilvusClientV2），废弃知识路径旧 SDK 检索/写入。 | RAG/BM25/hybrid | MilvusClientV2, BM25, hybridSearch, RRFRanker, biz_hybrid, drop SDK path | openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk | archived |
|
||||
| 2026-07-27 | rag-hybrid-search-rrf | Delivery 2：可配置 hybrid 检索与 RRF 多路融合（不绑旧 SDK）。 | RAG/hybrid/RRF | hybrid mode, RRF, KnowledgeSearchPort, filtered+unfiltered fusion, sparse-lite lexical | openspec/changes/archive/2026-07-27-rag-hybrid-search-rrf | archived |
|
||||
| 2026-07-21 | single-react-tool-invocation-store | 建立统一 ToolBoundary 与 Redis canonical invocation store，集中生命周期、证据状态、TTL、容量和 Run 所有权。 | Harness/Tool boundary/Canonical store | ISS-014, ToolBoundary, canonical invocation, PROJECTING, READY, ERROR, TTL, RESULT_TOO_LARGE | openspec/changes/archive/2026-07-21-single-react-tool-invocation-store | archived |
|
||||
| 2026-07-21 | single-react-harness-run-context | 建立显式 RunContext、Harness Core、预算、取消、类型化重试和 Tool Store 基础。 | Harness/Run lifecycle/Budget | ISS-014, RunContext, deadline, cancellation, budget, retry, ToolCallKey | openspec/changes/archive/2026-07-21-single-react-harness-run-context | archived |
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
# Acceptance
|
||||
|
||||
## Done
|
||||
|
||||
- `MilvusHybridKnowledgeStore`: schema BM25 function + dense, hybridSearch+RRFRanker, dense search
|
||||
- `VectorSearchService` only routes dense|hybrid to V2 store
|
||||
- `VectorIndexService` writes via V2 store
|
||||
- Removed knowledge-path `MilvusServiceClient` bean wiring
|
||||
- Config: `milvus.collection=biz_hybrid`, `retrieval.search.mode=hybrid`
|
||||
|
||||
## Tests
|
||||
|
||||
```text
|
||||
mvn -Dtest=LookupKnowledgeToolTest,KnowledgeEvidencePostProcessorTest,RagResultProjectorTest,RrfFusionTest,VectorKnowledgeSearchAdapterHybridTest,VectorSearchServiceTest,VectorIndexServiceTest test
|
||||
```
|
||||
|
||||
EXIT:0
|
||||
|
||||
## Ops note
|
||||
|
||||
Reindex all knowledge docs into `biz_hybrid` before production hybrid search is meaningful.
|
||||
Legacy `biz` collection is unused by knowledge path.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Brief: rag-bm25-hybrid-drop-sdk
|
||||
|
||||
True dense+BM25 hybrid on a single MilvusClientV2 backend. Legacy SDK search/write for knowledge path removed. New collection `biz_hybrid` requires reindex.
|
||||
@@ -0,0 +1,7 @@
|
||||
# Decisions
|
||||
|
||||
- Single backend: MilvusClientV2 only for knowledge RAG
|
||||
- Drop sdk/spring/auto retrieval routing
|
||||
- New collection biz_hybrid to avoid mutating legacy biz schema in place
|
||||
- Hybrid = dense ANN + BM25 sparse ANN + RRFRanker
|
||||
- Dense L2 enrichment for threshold compatibility on hybrid hits
|
||||
@@ -32,6 +32,26 @@
|
||||
- 分块去重与 hybrid 同规划、分里程碑交付（先共用地基，再开 hybrid）
|
||||
```
|
||||
|
||||
## 实现状态（2026-07-27）
|
||||
|
||||
| 里程碑 | 状态 | 说明 |
|
||||
|---|---|---|
|
||||
| 交付 1 chunk 身份/去重/SearchPort | **已完成并归档** | `2026-07-27-rag-chunk-evidence-identity-dedup` |
|
||||
| 交付 2a 应用层 multi-path+RRF | **已完成并归档** | `2026-07-27-rag-hybrid-search-rrf`（已被 2b 取代为生产路径） |
|
||||
| 交付 2b 真 BM25 hybrid + 废弃 SDK | **已完成并归档** | `2026-07-27-rag-bm25-hybrid-drop-sdk` |
|
||||
|
||||
**当前生产知识路径：**
|
||||
|
||||
```text
|
||||
VectorIndexService / VectorSearchService
|
||||
-> MilvusHybridKnowledgeStore (MilvusClientV2 only)
|
||||
collection: milvus.collection (default biz_hybrid)
|
||||
mode: retrieval.search.mode = dense | hybrid
|
||||
hybrid: dense ANN + BM25 sparse ANN + RRFRanker
|
||||
```
|
||||
|
||||
**运维必做：** 全量重灌知识库到 `biz_hybrid`；旧 `biz` collection 不再被知识路径使用。
|
||||
|
||||
---
|
||||
|
||||
## 1.1 交付拆分：分块去重 + Hybrid 同规划
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
committed: 2026-07-27
|
||||
authorized-apply: user-preauthorized
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-27
|
||||
@@ -0,0 +1,57 @@
|
||||
# Design: rag-bm25-hybrid-drop-sdk
|
||||
|
||||
## Decision: single backend
|
||||
|
||||
```text
|
||||
KnowledgeSearchPort
|
||||
-> MilvusHybridKnowledgeStore (MilvusClientV2 only)
|
||||
Write
|
||||
-> VectorIndexService -> MilvusHybridKnowledgeStore
|
||||
```
|
||||
|
||||
No `retrieval.vector-store.mode=sdk|spring|auto` for knowledge lookup.
|
||||
|
||||
## Schema (`milvus.collection`, default `biz_hybrid`)
|
||||
|
||||
| field | type | notes |
|
||||
|---|---|---|
|
||||
| id | VarChar PK | chunk id |
|
||||
| content | VarChar | original chunk body returned to agent |
|
||||
| search_text | VarChar + analyzer | BM25 input (title/path/content) |
|
||||
| sparse_vector | SparseFloatVector | BM25 function output |
|
||||
| vector | FloatVector | dense embedding |
|
||||
| metadata | JSON | docId, chunkIndex, category, kb_scope, ... |
|
||||
|
||||
Function: `BM25(search_text -> sparse_vector)`
|
||||
|
||||
Indexes:
|
||||
|
||||
- `vector`: IVF_FLAT / L2 (or COSINE if configured later)
|
||||
- `sparse_vector`: SPARSE_INVERTED_INDEX / BM25
|
||||
|
||||
## Hybrid query
|
||||
|
||||
```text
|
||||
AnnSearchReq(vector, FloatVec(queryEmbedding), topK, filter?)
|
||||
AnnSearchReq(sparse_vector, EmbeddedText(query), topK, filter?)
|
||||
HybridSearchReq + RRFRanker(k)
|
||||
```
|
||||
|
||||
Dense-compatible score for thresholds: if entity lacks dense distance, map fused score conservatively or re-use dense-only probe. Preferred: keep dense distance when available from a parallel dense search metadata; for hybrid hits use inverse-rank placeholder only in metadata and set score from optional dense sub-hit map.
|
||||
|
||||
Implementation approach for threshold stability:
|
||||
|
||||
1. Run hybridSearch for ordering/identity
|
||||
2. Build map evidenceKey -> dense L2 from a concurrent dense search (same filter/topK)
|
||||
3. Attach dense score onto fused hits when present; else maxL2 (low similarity) so weak BM25-only hits don't fake PRECISE
|
||||
|
||||
## Migration
|
||||
|
||||
- New collection name avoids mutating legacy `biz`
|
||||
- Operators re-run knowledge init / document reindex
|
||||
- Document in acceptance
|
||||
|
||||
## Drop SDK search
|
||||
|
||||
- Delete SDK search methods usage from knowledge path
|
||||
- `MilvusServiceClient` bean may remain temporarily only if other non-search utilities need it; prefer migrate write/delete fully to V2 and stop creating V1 client if unused
|
||||
@@ -0,0 +1,34 @@
|
||||
# Change: Real BM25 hybrid search and drop legacy SDK path
|
||||
|
||||
## Why
|
||||
|
||||
Delivery 2 shipped application-layer multi-path + sparse-lite lexical ranking. Product requirement is **true dense + BM25 hybrid retrieval**. Legacy `MilvusServiceClient` search path will be abandoned in this change; keep a single knowledge vector backend based on Milvus Java SDK v2 (`MilvusClientV2`) with BM25 function + `hybridSearch` + RRF.
|
||||
|
||||
## What Changes
|
||||
|
||||
- New hybrid collection schema: dense float vector + analyzed text + sparse BM25 output
|
||||
- Write path inserts text for BM25 auto-sparsification and dense embedding
|
||||
- Search path: single backend
|
||||
- dense mode: dense ANN only
|
||||
- hybrid mode: dense ANN + BM25 sparse ANN fused by RRFRanker
|
||||
- Remove/disable legacy SDK `search` / `searchSimilarDocumentsWithSdk` knowledge path and mode routing (`sdk|spring|auto`)
|
||||
- KnowledgeSearchPort adapter calls the single backend
|
||||
- Config: collection name, search mode, rrf-k, path weights optional
|
||||
- Tests for store mapping/fusion wiring; no dependency on live Zilliz in unit tests
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Automatic full production reindex job UI
|
||||
- Cross-encoder model rerank
|
||||
- Keeping dual Spring-AI + SDK knowledge search modes
|
||||
|
||||
## Impact
|
||||
|
||||
- **Breaking for existing `biz` collection**: requires new hybrid collection + reindex
|
||||
- Agent tool contract unchanged
|
||||
- Interface: L2 internal retrieval backend
|
||||
|
||||
## Depends on
|
||||
|
||||
- Delivery 1 chunk identity
|
||||
- Delivery 2 SearchPort (will replace sparse-lite hybrid implementation)
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
# rag-bm25-hybrid Specification
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Knowledge vector backend SHALL be a single Milvus V2 store
|
||||
|
||||
Knowledge indexing and retrieval SHALL use one MilvusClientV2-backed store. Legacy MilvusServiceClient SDK search modes (`sdk`, `auto` fallback to SDK) SHALL NOT be used for `lookup_knowledge`.
|
||||
|
||||
#### Scenario: No SDK search mode
|
||||
|
||||
- **WHEN** knowledge retrieval executes
|
||||
- **THEN** it SHALL NOT call legacy SDK `search` APIs for candidate generation
|
||||
|
||||
### Requirement: Hybrid mode SHALL fuse dense ANN and BM25 sparse ANN
|
||||
|
||||
When search mode is hybrid, the store SHALL query dense vectors and BM25 sparse vectors and fuse results with RRF (or equivalent ranker) before returning hits.
|
||||
|
||||
#### Scenario: Hybrid uses BM25 text query
|
||||
|
||||
- **WHEN** hybrid search runs with a text query
|
||||
- **THEN** one search leg SHALL use the BM25/sparse field with the raw query text
|
||||
- **AND** another leg SHALL use the dense embedding of the query
|
||||
|
||||
### Requirement: Write path SHALL populate BM25 input text and dense vectors
|
||||
|
||||
Document chunk indexing SHALL write original content, BM25 input text, dense embedding, and metadata required for chunk identity.
|
||||
|
||||
#### Scenario: Index writes search_text and vector
|
||||
|
||||
- **WHEN** a chunk is indexed
|
||||
- **THEN** the store row SHALL include searchable text for BM25 and a dense vector field
|
||||
@@ -0,0 +1,10 @@
|
||||
# Tasks
|
||||
|
||||
- [x] 1. Add MilvusClientV2 factory + hybrid collection bootstrap
|
||||
- [x] 2. Implement MilvusHybridKnowledgeStore (insert/delete/dense/hybrid)
|
||||
- [x] 3. Point VectorIndexService writes at V2 store
|
||||
- [x] 4. Replace VectorSearchService knowledge path with V2 store (drop SDK/spring mode routing)
|
||||
- [x] 5. Update VectorKnowledgeSearchAdapter to delegate single backend
|
||||
- [x] 6. Config + constants for collection/mode
|
||||
- [x] 7. Unit tests (store routing / adapter / no SDK search)
|
||||
- [x] 8. Archive + commit
|
||||
@@ -0,0 +1,33 @@
|
||||
# rag-bm25-hybrid Specification
|
||||
|
||||
## Purpose
|
||||
TBD - created by archiving change rag-bm25-hybrid-drop-sdk. Update Purpose after archive.
|
||||
## Requirements
|
||||
### Requirement: Knowledge vector backend SHALL be a single Milvus V2 store
|
||||
|
||||
Knowledge indexing and retrieval SHALL use one MilvusClientV2-backed store. Legacy MilvusServiceClient SDK search modes (`sdk`, `auto` fallback to SDK) SHALL NOT be used for `lookup_knowledge`.
|
||||
|
||||
#### Scenario: No SDK search mode
|
||||
|
||||
- **WHEN** knowledge retrieval executes
|
||||
- **THEN** it SHALL NOT call legacy SDK `search` APIs for candidate generation
|
||||
|
||||
### Requirement: Hybrid mode SHALL fuse dense ANN and BM25 sparse ANN
|
||||
|
||||
When search mode is hybrid, the store SHALL query dense vectors and BM25 sparse vectors and fuse results with RRF (or equivalent ranker) before returning hits.
|
||||
|
||||
#### Scenario: Hybrid uses BM25 text query
|
||||
|
||||
- **WHEN** hybrid search runs with a text query
|
||||
- **THEN** one search leg SHALL use the BM25/sparse field with the raw query text
|
||||
- **AND** another leg SHALL use the dense embedding of the query
|
||||
|
||||
### Requirement: Write path SHALL populate BM25 input text and dense vectors
|
||||
|
||||
Document chunk indexing SHALL write original content, BM25 input text, dense embedding, and metadata required for chunk identity.
|
||||
|
||||
#### Scenario: Index writes search_text and vector
|
||||
|
||||
- **WHEN** a chunk is indexed
|
||||
- **THEN** the store row SHALL include searchable text for BM25 and a dense vector field
|
||||
|
||||
@@ -1,51 +1,22 @@
|
||||
package com.superbiz.agent.config;
|
||||
|
||||
import io.milvus.client.MilvusServiceClient;
|
||||
import com.superbiz.agent.client.MilvusClientFactory;
|
||||
import com.superbiz.agent.service.milvus.MilvusHybridKnowledgeStore;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
|
||||
import jakarta.annotation.PreDestroy;
|
||||
|
||||
/**
|
||||
* Milvus 配置类
|
||||
* 负责创建和管理 MilvusServiceClient Bean
|
||||
* Milvus configuration notes.
|
||||
*
|
||||
* <p>Knowledge RAG uses {@link MilvusHybridKnowledgeStore} (MilvusClientV2) exclusively.
|
||||
* Legacy {@code MilvusServiceClient} bean is no longer created for the knowledge path.</p>
|
||||
*/
|
||||
@Configuration
|
||||
public class MilvusConfig {
|
||||
|
||||
private static final Logger logger = LoggerFactory.getLogger(MilvusConfig.class);
|
||||
|
||||
@Autowired
|
||||
private MilvusClientFactory milvusClientFactory;
|
||||
|
||||
private MilvusServiceClient milvusClient;
|
||||
|
||||
/**
|
||||
* 创建 MilvusServiceClient Bean
|
||||
*
|
||||
* @return MilvusServiceClient 实例
|
||||
*/
|
||||
@Bean
|
||||
public MilvusServiceClient milvusServiceClient() {
|
||||
logger.info("正在初始化 Milvus 客户端...");
|
||||
milvusClient = milvusClientFactory.createClient();
|
||||
logger.info("Milvus 客户端初始化完成");
|
||||
return milvusClient;
|
||||
}
|
||||
|
||||
/**
|
||||
* 应用关闭时清理资源
|
||||
*/
|
||||
@PreDestroy
|
||||
public void cleanup() {
|
||||
if (milvusClient != null) {
|
||||
logger.info("正在关闭 Milvus 客户端连接...");
|
||||
milvusClient.close();
|
||||
logger.info("Milvus 客户端连接已关闭");
|
||||
}
|
||||
public MilvusConfig() {
|
||||
logger.info("Milvus knowledge path: MilvusClientV2 hybrid store only (legacy SDK search disabled)");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -8,10 +8,15 @@ public class MilvusConstants {
|
||||
public static final String MILVUS_DB_NAME = "default";
|
||||
|
||||
/**
|
||||
* Milvus 集合名称
|
||||
* Legacy collection name (pre hybrid). Knowledge path uses milvus.collection (default biz_hybrid).
|
||||
*/
|
||||
public static final String MILVUS_COLLECTION_NAME = "biz";
|
||||
|
||||
/**
|
||||
* Default hybrid dense+BM25 collection.
|
||||
*/
|
||||
public static final String MILVUS_HYBRID_COLLECTION_NAME = "biz_hybrid";
|
||||
|
||||
/**
|
||||
* 向量维度（豆包 embedding 模型的维度）
|
||||
*/
|
||||
|
||||
@@ -1,47 +1,36 @@
|
||||
package com.superbiz.agent.controller;
|
||||
|
||||
import io.milvus.client.MilvusServiceClient;
|
||||
import io.milvus.grpc.ShowCollectionsResponse;
|
||||
import io.milvus.param.R;
|
||||
import io.milvus.param.collection.ShowCollectionsParam;
|
||||
import com.superbiz.agent.service.milvus.MilvusHybridKnowledgeStore;
|
||||
import io.milvus.v2.service.collection.response.ListCollectionsResp;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.*;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* Milvus 测试控制器
|
||||
* 用于测试数据库连接和数据读取
|
||||
* Milvus health check using the single V2 knowledge backend.
|
||||
*/
|
||||
@RestController
|
||||
@RequestMapping("/milvus")
|
||||
public class MilvusCheckController {
|
||||
|
||||
@Autowired
|
||||
private MilvusServiceClient milvusClient;
|
||||
private MilvusHybridKnowledgeStore knowledgeStore;
|
||||
|
||||
/**
|
||||
* 简单的健康检查
|
||||
*/
|
||||
@GetMapping("/health")
|
||||
public ResponseEntity<Map<String, Object>> simpleHealth() {
|
||||
Map<String, Object> result = new HashMap<>();
|
||||
|
||||
try {
|
||||
R<ShowCollectionsResponse> response = milvusClient.showCollections(
|
||||
ShowCollectionsParam.newBuilder().build()
|
||||
);
|
||||
|
||||
if (response.getStatus() == 0) {
|
||||
ListCollectionsResp response = knowledgeStore.client().listCollections();
|
||||
result.put("message", "ok");
|
||||
result.put("collections", response.getData().getCollectionNamesList());
|
||||
result.put("backend", "milvus-client-v2");
|
||||
result.put("knowledgeCollection", knowledgeStore.collectionName());
|
||||
result.put("collections", response == null ? null : response.getCollectionNames());
|
||||
return ResponseEntity.ok(result);
|
||||
} else {
|
||||
result.put("message", response.getMessage());
|
||||
return ResponseEntity.status(503).body(result);
|
||||
}
|
||||
} catch (Exception e) {
|
||||
result.put("error", e.getMessage());
|
||||
return ResponseEntity.status(503).body(result);
|
||||
|
||||
@@ -1,17 +1,10 @@
|
||||
package com.superbiz.agent.service;
|
||||
|
||||
import io.milvus.client.MilvusServiceClient;
|
||||
import io.milvus.grpc.MutationResult;
|
||||
import io.milvus.param.R;
|
||||
import io.milvus.param.RpcStatus;
|
||||
import io.milvus.param.collection.LoadCollectionParam;
|
||||
import io.milvus.param.dml.DeleteParam;
|
||||
import io.milvus.param.dml.InsertParam;
|
||||
import lombok.Getter;
|
||||
import lombok.Setter;
|
||||
import com.superbiz.agent.constant.MilvusConstants;
|
||||
import com.superbiz.agent.dto.DocumentChunk;
|
||||
import com.superbiz.agent.dto.Frontmatter;
|
||||
import com.superbiz.agent.service.milvus.MilvusHybridKnowledgeStore;
|
||||
import lombok.Getter;
|
||||
import lombok.Setter;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
@@ -23,20 +16,15 @@ import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.Paths;
|
||||
import java.time.LocalDateTime;
|
||||
import java.util.*;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* 向量索引写入服务（RAG 入库侧）。
|
||||
*
|
||||
* <p>负责：读文件/文档块 -> 切片 -> embedding -> 写入 Milvus。
|
||||
* 检索读取走 {@link VectorSearchService}，双方通过 collection + metadata 约定衔接。</p>
|
||||
*
|
||||
* <h3>metadata 关键字段</h3>
|
||||
* docId / _source / chunkIndex / totalChunks / title / breadcrumb / category / kb_scope
|
||||
*
|
||||
* <h3>embedding 文本</h3>
|
||||
* 见 {@link #buildEmbeddingText(DocumentChunk)}：会把 title、breadcrumb 拼进向量文本，
|
||||
* 而入库 content 字段仍保存原始 chunk 正文（检索返回的是 content，不是 embedding 拼接串）。
|
||||
* <p>写入单一后端 {@link MilvusHybridKnowledgeStore}（dense + BM25 search_text）。
|
||||
* 不再使用 legacy {@code MilvusServiceClient} insert/delete。</p>
|
||||
*/
|
||||
@Service
|
||||
public class VectorIndexService {
|
||||
@@ -44,7 +32,7 @@ public class VectorIndexService {
|
||||
private static final Logger logger = LoggerFactory.getLogger(VectorIndexService.class);
|
||||
|
||||
@Autowired
|
||||
private MilvusServiceClient milvusClient;
|
||||
private MilvusHybridKnowledgeStore knowledgeStore;
|
||||
|
||||
@Autowired
|
||||
private VectorEmbeddingService embeddingService;
|
||||
@@ -55,18 +43,11 @@ public class VectorIndexService {
|
||||
@Value("${file.upload.path}")
|
||||
private String uploadPath;
|
||||
|
||||
/**
|
||||
* 索引指定目录下的所有文件
|
||||
*
|
||||
* @param directoryPath 目录路径（可选，默认使用配置的上传目录）
|
||||
* @return 索引结果 这里可以优化：定时重建目录下所有文件的索引
|
||||
*/
|
||||
public IndexingResult indexDirectory(String directoryPath) {
|
||||
IndexingResult result = new IndexingResult();
|
||||
result.setStartTime(LocalDateTime.now());
|
||||
|
||||
try {
|
||||
// 使用指定目录或默认上传目录
|
||||
String targetPath = (directoryPath != null && !directoryPath.trim().isEmpty())
|
||||
? directoryPath : uploadPath;
|
||||
|
||||
@@ -79,7 +60,6 @@ public class VectorIndexService {
|
||||
|
||||
result.setDirectoryPath(directory.getAbsolutePath());
|
||||
|
||||
// 获取所有支持的文件
|
||||
File[] files = directory.listFiles((dir, name) ->
|
||||
name.endsWith(".txt") || name.endsWith(".md")
|
||||
);
|
||||
@@ -95,25 +75,20 @@ public class VectorIndexService {
|
||||
result.setTotalFiles(files.length);
|
||||
logger.info("开始索引目录: {}, 找到 {} 个文件", targetPath, files.length);
|
||||
|
||||
// 遍历并索引每个文件
|
||||
for (File file : files) {
|
||||
try {
|
||||
indexSingleFile(file.getAbsolutePath());
|
||||
result.incrementSuccessCount();
|
||||
logger.info("✓ 文件索引成功: {}", file.getName());
|
||||
logger.info("文件索引成功: {}", file.getName());
|
||||
} catch (Exception e) {
|
||||
result.incrementFailCount();
|
||||
result.addFailedFile(file.getAbsolutePath(), e.getMessage());
|
||||
logger.error("✗ 文件索引失败: {}", file.getName(), e);
|
||||
logger.error("文件索引失败: {}", file.getName(), e);
|
||||
}
|
||||
}
|
||||
|
||||
result.setSuccess(result.getFailCount() == 0);
|
||||
result.setEndTime(LocalDateTime.now());
|
||||
|
||||
logger.info("目录索引完成: 总数={}, 成功={}, 失败={}",
|
||||
result.getTotalFiles(), result.getSuccessCount(), result.getFailCount());
|
||||
|
||||
return result;
|
||||
|
||||
} catch (Exception e) {
|
||||
@@ -125,12 +100,6 @@ public class VectorIndexService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 索引单个文件
|
||||
*
|
||||
* @param filePath 文件路径
|
||||
* @throws Exception 索引失败时抛出异常
|
||||
*/
|
||||
public void indexSingleFile(String filePath) throws Exception {
|
||||
Path path = Paths.get(filePath).normalize();
|
||||
File file = path.toFile();
|
||||
@@ -140,133 +109,76 @@ public class VectorIndexService {
|
||||
}
|
||||
|
||||
logger.info("开始索引文件: {}", path);
|
||||
|
||||
// 1. 读取文件内容
|
||||
String content = Files.readString(path);
|
||||
logger.info("读取文件: {}, 内容长度: {} 字符", path, content.length());
|
||||
|
||||
// 2. 删除该文件的旧数据（如果存在）
|
||||
deleteExistingData(path.toString());
|
||||
|
||||
// 3. 文档分片
|
||||
List<DocumentChunk> chunks = chunkService.chunkDocument(content, path.toString());
|
||||
logger.info("文档分片完成: {} -> {} 个分片", filePath, chunks.size());
|
||||
|
||||
// 4. 为每个分片生成向量并插入 Milvus
|
||||
for (int i = 0; i < chunks.size(); i++) {
|
||||
DocumentChunk chunk = chunks.get(i);
|
||||
|
||||
try {
|
||||
// 生成向量
|
||||
List<Float> vector = embeddingService.generateEmbedding(buildEmbeddingText(chunk));
|
||||
|
||||
// 构建元数据（包含文件信息）
|
||||
Map<String, Object> metadata = buildMetadata(path.toString(), chunk, chunks.size());
|
||||
|
||||
// 插入到 Milvus
|
||||
insertToMilvus(chunk.getContent(), vector, metadata, chunk.getChunkIndex());
|
||||
|
||||
logger.info("✓ 分片 {}/{} 索引成功", i + 1, chunks.size());
|
||||
|
||||
knowledgeStore.upsertChunk(
|
||||
chunk.getContent(),
|
||||
buildSearchText(chunk),
|
||||
vector,
|
||||
metadata,
|
||||
chunk.getChunkIndex());
|
||||
logger.info("分片 {}/{} 索引成功", i + 1, chunks.size());
|
||||
} catch (Exception e) {
|
||||
logger.error("✗ 分片 {}/{} 索引失败", i + 1, chunks.size(), e);
|
||||
logger.error("分片 {}/{} 索引失败", i + 1, chunks.size(), e);
|
||||
throw new RuntimeException("分片索引失败: " + e.getMessage(), e);
|
||||
}
|
||||
}
|
||||
|
||||
logger.info("文件索引完成: {}, 共 {} 个分片", filePath, chunks.size());
|
||||
}
|
||||
|
||||
/**
|
||||
* 索引文档分块（用于上传文档的向量化）
|
||||
*
|
||||
* @param docId 文档ID
|
||||
* @param chunks 文档分块列表
|
||||
* @param category 文档类别（api、domain、troubleshoot 等）
|
||||
* @throws Exception 索引失败时抛出异常
|
||||
*/
|
||||
public void indexDocumentChunks(String docId, List<DocumentChunk> chunks, String category) throws Exception {
|
||||
indexDocumentChunks(docId, chunks, category, null);
|
||||
}
|
||||
|
||||
public void indexDocumentChunks(String docId, List<DocumentChunk> chunks, String category, Frontmatter frontmatter) throws Exception {
|
||||
public void indexDocumentChunks(String docId,
|
||||
List<DocumentChunk> chunks,
|
||||
String category,
|
||||
Frontmatter frontmatter) throws Exception {
|
||||
if (chunks == null || chunks.isEmpty()) {
|
||||
throw new IllegalArgumentException("文档分块列表为空");
|
||||
}
|
||||
|
||||
logger.info("开始索引文档分块，docId: {}, 分块数: {}, 类别: {}", docId, chunks.size(), category);
|
||||
|
||||
// 1. 删除该文档的旧数据（如果存在）
|
||||
deleteDocumentChunks(docId);
|
||||
|
||||
// 2. 为每个分块生成向量并插入 Milvus
|
||||
for (int i = 0; i < chunks.size(); i++) {
|
||||
DocumentChunk chunk = chunks.get(i);
|
||||
|
||||
try {
|
||||
// 生成向量
|
||||
List<Float> vector = embeddingService.generateEmbedding(buildEmbeddingText(chunk));
|
||||
|
||||
// 构建元数据（使用 docId 和 category）
|
||||
Map<String, Object> metadata = buildDocumentMetadata(docId, chunk, chunks.size(), category, frontmatter);
|
||||
|
||||
// 插入到 Milvus
|
||||
insertToMilvus(chunk.getContent(), vector, metadata, chunk.getChunkIndex());
|
||||
|
||||
logger.info("✓ 文档分块 {}/{} 索引成功，docId: {}", i + 1, chunks.size(), docId);
|
||||
|
||||
knowledgeStore.upsertChunk(
|
||||
chunk.getContent(),
|
||||
buildSearchText(chunk),
|
||||
vector,
|
||||
metadata,
|
||||
chunk.getChunkIndex());
|
||||
logger.info("文档分块 {}/{} 索引成功，docId: {}", i + 1, chunks.size(), docId);
|
||||
} catch (Exception e) {
|
||||
logger.error("✗ 文档分块 {}/{} 索引失败，docId: {}", i + 1, chunks.size(), docId, e);
|
||||
logger.error("文档分块 {}/{} 索引失败，docId: {}", i + 1, chunks.size(), docId, e);
|
||||
throw new RuntimeException("文档分块索引失败: " + e.getMessage(), e);
|
||||
}
|
||||
}
|
||||
|
||||
logger.info("文档索引完成，docId: {}, 共 {} 个分块，类别: {}", docId, chunks.size(), category);
|
||||
}
|
||||
|
||||
/**
|
||||
* 删除文档的所有分块（根据 docId）
|
||||
*/
|
||||
public void deleteDocumentChunks(String docId) {
|
||||
try {
|
||||
// 构建删除表达式：metadata["docId"] == "xxx"
|
||||
String expr = String.format("metadata[\"docId\"] == \"%s\"", docId);
|
||||
|
||||
logger.info("准备删除文档旧数据，docId: {}, 表达式: {}", docId, expr);
|
||||
|
||||
// 确保 collection 已加载
|
||||
R<RpcStatus> loadResponse = milvusClient.loadCollection(
|
||||
LoadCollectionParam.newBuilder()
|
||||
.withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME)
|
||||
.build()
|
||||
);
|
||||
|
||||
if (loadResponse.getStatus() != 0 && loadResponse.getStatus() != 65535) {
|
||||
logger.warn("加载 collection 失败: {}", loadResponse.getMessage());
|
||||
return;
|
||||
}
|
||||
|
||||
DeleteParam deleteParam = DeleteParam.newBuilder()
|
||||
.withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME)
|
||||
.withExpr(expr)
|
||||
.build();
|
||||
|
||||
R<MutationResult> deleteResponse = milvusClient.delete(deleteParam);
|
||||
|
||||
if (deleteResponse.getStatus() == 0) {
|
||||
knowledgeStore.deleteByDocId(docId);
|
||||
logger.info("删除文档旧数据成功，docId: {}", docId);
|
||||
} else {
|
||||
logger.warn("删除文档旧数据失败，docId: {}, 原因: {}", docId, deleteResponse.getMessage());
|
||||
}
|
||||
|
||||
} catch (Exception e) {
|
||||
logger.warn("删除文档旧数据异常，docId: {}", docId, e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 构建文档元数据（用于上传文档）
|
||||
*/
|
||||
static Map<String, Object> buildDocumentMetadata(String docId, DocumentChunk chunk, int totalChunks, String category) {
|
||||
return buildDocumentMetadata(docId, chunk, totalChunks, category, null);
|
||||
}
|
||||
@@ -277,47 +189,37 @@ public class VectorIndexService {
|
||||
String category,
|
||||
Frontmatter frontmatter) {
|
||||
Map<String, Object> metadata = new HashMap<>();
|
||||
|
||||
// 文档标识
|
||||
String source = firstNonBlank(frontmatter != null ? frontmatter.getSource() : null, "upload:" + docId);
|
||||
metadata.put("docId", docId);
|
||||
metadata.put("_source", source); // 区分文件索引和上传文档
|
||||
metadata.put("_source", source);
|
||||
metadata.put("source", source);
|
||||
|
||||
// 分片信息
|
||||
metadata.put("chunkIndex", chunk.getChunkIndex());
|
||||
metadata.put("totalChunks", totalChunks);
|
||||
|
||||
// 标题信息
|
||||
String title = firstNonBlank(chunk.getTitle(), frontmatter != null ? frontmatter.getTitle() : null);
|
||||
if (title != null) {
|
||||
metadata.put("title", title);
|
||||
}
|
||||
|
||||
// 面包屑导航（完整标题层级路径）
|
||||
String breadcrumb = firstNonBlank(frontmatter != null ? frontmatter.getBreadcrumb() : null, chunk.getBreadcrumb());
|
||||
if (breadcrumb != null) {
|
||||
metadata.put("breadcrumb", breadcrumb);
|
||||
}
|
||||
|
||||
// 文档类别
|
||||
metadata.put("category", category != null && !category.isBlank() ? category : "upload");
|
||||
|
||||
String kbScope = trimToNull(frontmatter != null ? frontmatter.getKbScope() : null);
|
||||
if (kbScope != null) {
|
||||
metadata.put("kb_scope", kbScope);
|
||||
}
|
||||
|
||||
return metadata;
|
||||
}
|
||||
|
||||
/**
|
||||
* 构造送入 embedding 模型的文本。
|
||||
*
|
||||
* <p>把标题链路注入向量语义，缓解“正文片段缺上下文”导致的召回漂移。
|
||||
* 注意：这里只影响向量，不影响 Milvus content 字段存储的原文。</p>
|
||||
*/
|
||||
static String buildEmbeddingText(DocumentChunk chunk) {
|
||||
return buildSearchText(chunk);
|
||||
}
|
||||
|
||||
/**
|
||||
* Text used for BM25 {@code search_text} and dense embedding.
|
||||
*/
|
||||
static String buildSearchText(DocumentChunk chunk) {
|
||||
String content = trimToEmpty(chunk.getContent());
|
||||
String title = trimToEmpty(chunk.getTitle());
|
||||
String breadcrumb = trimToEmpty(chunk.getBreadcrumb());
|
||||
@@ -358,64 +260,22 @@ public class VectorIndexService {
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* 删除文件的旧数据（根据 metadata._source）
|
||||
*/
|
||||
private void deleteExistingData(String filePath) {
|
||||
try {
|
||||
// 使用统一的路径分隔符（正斜杠）用于Milvus存储，避免表达式解析错误
|
||||
// 将系统路径转换为统一格式
|
||||
Path path = Paths.get(filePath).normalize();
|
||||
String normalizedPath = path.toString().replace(File.separator, "/");
|
||||
|
||||
// 构建删除表达式：metadata["_source"] == "xxx"
|
||||
String expr = String.format("metadata[\"_source\"] == \"%s\"", normalizedPath);
|
||||
|
||||
logger.info("准备删除旧数据，路径: {}, 表达式: {}", normalizedPath, expr);
|
||||
|
||||
// 确保 collection 已加载（删除操作需要集合已加载）
|
||||
R<RpcStatus> loadResponse = milvusClient.loadCollection(
|
||||
LoadCollectionParam.newBuilder()
|
||||
.withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME)
|
||||
.build()
|
||||
);
|
||||
|
||||
// 状态码 65535 表示集合已经加载，这不是错误
|
||||
if (loadResponse.getStatus() != 0 && loadResponse.getStatus() != 65535) {
|
||||
logger.warn("加载 collection 失败: {}", loadResponse.getMessage());
|
||||
return;
|
||||
}
|
||||
|
||||
DeleteParam deleteParam = DeleteParam.newBuilder()
|
||||
.withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME)
|
||||
.withExpr(expr)
|
||||
.build();
|
||||
|
||||
R<MutationResult> response = milvusClient.delete(deleteParam);
|
||||
|
||||
if (response.getStatus() != 0) {
|
||||
logger.warn("删除旧数据时出现警告: {}", response.getMessage());
|
||||
} else {
|
||||
long deletedCount = response.getData().getDeleteCnt();
|
||||
logger.info("✓ 已删除文件的旧数据: {}, 删除记录数: {}", normalizedPath, deletedCount);
|
||||
}
|
||||
|
||||
knowledgeStore.deleteBySource(normalizedPath);
|
||||
logger.info("已删除文件的旧数据: {}", normalizedPath);
|
||||
} catch (Exception e) {
|
||||
logger.warn("删除旧数据失败（可能是首次索引）: {}", e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 构建元数据（包含文件信息）
|
||||
*/
|
||||
private Map<String, Object> buildMetadata(String filePath, DocumentChunk chunk, int totalChunks) {
|
||||
Map<String, Object> metadata = new HashMap<>();
|
||||
|
||||
// 标准化路径：使用统一的路径分隔符（正斜杠）用于存储，确保跨平台一致性
|
||||
Path path = Paths.get(filePath).normalize();
|
||||
String normalizedPath = path.toString().replace(File.separator, "/");
|
||||
|
||||
// 文件信息
|
||||
Path fileName = path.getFileName();
|
||||
String fileNameStr = fileName != null ? fileName.toString() : "";
|
||||
String extension = "";
|
||||
@@ -425,42 +285,28 @@ public class VectorIndexService {
|
||||
}
|
||||
|
||||
metadata.put("_source", normalizedPath);
|
||||
metadata.put("source", normalizedPath);
|
||||
metadata.put("_extension", extension);
|
||||
metadata.put("_file_name", fileNameStr);
|
||||
|
||||
// 提取类别（从文件路径中提取目录名）
|
||||
String category = extractCategory(normalizedPath);
|
||||
if (category != null && !category.isEmpty()) {
|
||||
metadata.put("category", category);
|
||||
}
|
||||
|
||||
// 分片信息
|
||||
metadata.put("chunkIndex", chunk.getChunkIndex());
|
||||
metadata.put("totalChunks", totalChunks);
|
||||
|
||||
// 标题信息
|
||||
if (chunk.getTitle() != null && !chunk.getTitle().isEmpty()) {
|
||||
metadata.put("title", chunk.getTitle());
|
||||
}
|
||||
|
||||
// 面包屑导航（完整标题层级路径）
|
||||
if (chunk.getBreadcrumb() != null && !chunk.getBreadcrumb().isEmpty()) {
|
||||
metadata.put("breadcrumb", chunk.getBreadcrumb());
|
||||
}
|
||||
|
||||
return metadata;
|
||||
}
|
||||
|
||||
/**
|
||||
* 从文件路径中提取类别
|
||||
* 例如：aiops-docs/api/redis-api.md → "api"
|
||||
*/
|
||||
private String extractCategory(String filePath) {
|
||||
try {
|
||||
// 标准化路径分隔符
|
||||
String normalized = filePath.replace("\\", "/");
|
||||
|
||||
// 查找 aiops-docs/ 后的第一级目录
|
||||
int docsIndex = normalized.indexOf("aiops-docs/");
|
||||
if (docsIndex >= 0) {
|
||||
String afterDocs = normalized.substring(docsIndex + "aiops-docs/".length());
|
||||
@@ -469,13 +315,10 @@ public class VectorIndexService {
|
||||
return afterDocs.substring(0, slashIndex);
|
||||
}
|
||||
}
|
||||
|
||||
// 如果没有 aiops-docs，返回第一级目录
|
||||
int firstSlash = normalized.indexOf("/");
|
||||
if (firstSlash > 0) {
|
||||
return normalized.substring(0, firstSlash);
|
||||
}
|
||||
|
||||
return null;
|
||||
} catch (Exception e) {
|
||||
logger.warn("提取类别失败，路径: {}", filePath, e);
|
||||
@@ -483,68 +326,6 @@ public class VectorIndexService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 插入向量到 Milvus
|
||||
*/
|
||||
private void insertToMilvus(String content, List<Float> vector,
|
||||
Map<String, Object> metadata, int chunkIndex) throws Exception {
|
||||
try {
|
||||
// 确保 collection 已加载
|
||||
R<RpcStatus> loadResponse = milvusClient.loadCollection(
|
||||
LoadCollectionParam.newBuilder()
|
||||
.withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME)
|
||||
.build()
|
||||
);
|
||||
|
||||
if (loadResponse.getStatus() != 0 && loadResponse.getStatus() != 65535) {
|
||||
throw new RuntimeException("加载 collection 失败: " + loadResponse.getMessage());
|
||||
}
|
||||
|
||||
// 生成唯一 ID（使用 _source + 分片索引）
|
||||
String source = (String) metadata.get("_source");
|
||||
String id = UUID.nameUUIDFromBytes((source + "_" + chunkIndex).getBytes()).toString();
|
||||
|
||||
// 构建字段数据
|
||||
List<InsertParam.Field> fields = new ArrayList<>();
|
||||
|
||||
// ID 字段
|
||||
fields.add(new InsertParam.Field("id", Collections.singletonList(id)));
|
||||
|
||||
// content 字段
|
||||
fields.add(new InsertParam.Field("content", Collections.singletonList(content)));
|
||||
|
||||
// vector 字段
|
||||
fields.add(new InsertParam.Field("vector", Collections.singletonList(vector)));
|
||||
|
||||
// metadata 字段（JSON 对象）
|
||||
com.google.gson.Gson gson = new com.google.gson.Gson();
|
||||
com.google.gson.JsonObject metadataJson = gson.toJsonTree(metadata).getAsJsonObject();
|
||||
fields.add(new InsertParam.Field("metadata", Collections.singletonList(metadataJson)));
|
||||
|
||||
// 构建插入参数
|
||||
InsertParam insertParam = InsertParam.newBuilder()
|
||||
.withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME)
|
||||
.withFields(fields)
|
||||
.build();
|
||||
|
||||
// 执行插入
|
||||
R<MutationResult> insertResponse = milvusClient.insert(insertParam);
|
||||
|
||||
if (insertResponse.getStatus() != 0) {
|
||||
throw new RuntimeException("插入向量失败: " + insertResponse.getMessage());
|
||||
}
|
||||
|
||||
logger.debug("向量插入成功: id={}, source={}, chunk={}", id, source, chunkIndex);
|
||||
|
||||
} catch (Exception e) {
|
||||
logger.error("插入向量到 Milvus 失败", e);
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 索引结果类
|
||||
*/
|
||||
@Getter
|
||||
public static class IndexingResult {
|
||||
@Setter
|
||||
|
||||
@@ -1,53 +1,23 @@
|
||||
package com.superbiz.agent.service;
|
||||
|
||||
import com.fasterxml.jackson.core.JsonProcessingException;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.constant.MilvusConstants;
|
||||
import io.milvus.client.MilvusServiceClient;
|
||||
import io.milvus.grpc.SearchResults;
|
||||
import io.milvus.param.R;
|
||||
import io.milvus.param.dml.SearchParam;
|
||||
import io.milvus.response.SearchResultsWrapper;
|
||||
import com.superbiz.agent.service.milvus.MilvusHybridKnowledgeStore;
|
||||
import lombok.Getter;
|
||||
import lombok.Setter;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.beans.factory.ObjectProvider;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Locale;
|
||||
|
||||
/**
|
||||
* L1 向量检索门面。
|
||||
* Knowledge vector retrieval facade.
|
||||
*
|
||||
* <p>对上层（{@link KnowledgeDocumentRetriever}）只暴露稳定 API：
|
||||
* {@link #searchSimilarDocuments(String, int, String)}。底层实现可切换：</p>
|
||||
* <ul>
|
||||
* <li>{@code sdk}：Milvus SDK 直连</li>
|
||||
* <li>{@code spring} / {@code spring-ai}：Spring AI VectorStore</li>
|
||||
* <li>{@code auto}（默认）：先 Spring AI，失败再 fallback 到 SDK</li>
|
||||
* </ul>
|
||||
*
|
||||
* <h3>分数兼容约定</h3>
|
||||
* 后处理 {@code KnowledgeEvidencePostProcessor} 按“L2 距离越小越相似”归一化。
|
||||
* 因此本类统一把 {@link SearchResult#score} 填成兼容 L2 距离：
|
||||
* <ul>
|
||||
* <li>SDK 路径：直接用 Milvus L2 score</li>
|
||||
* <li>Spring AI 路径：优先 metadata.distance；否则把 similarity 映射为
|
||||
* {@code (1 - similarity) * maxL2Distance}</li>
|
||||
* </ul>
|
||||
*
|
||||
* <h3>过滤</h3>
|
||||
* 可选 category + 全局 {@code retrieval.kb-scope}。两条实现路径的 filter 语法不同，
|
||||
* 但语义一致：只在对应 metadata 字段上收窄。
|
||||
* <p><b>Single backend:</b> {@link MilvusHybridKnowledgeStore} (Milvus Java SDK v2).
|
||||
* Legacy {@code MilvusServiceClient} search and Spring AI VectorStore routing for
|
||||
* {@code lookup_knowledge} have been removed.</p>
|
||||
*/
|
||||
@Service
|
||||
public class VectorSearchService {
|
||||
@@ -55,276 +25,42 @@ public class VectorSearchService {
|
||||
private static final Logger logger = LoggerFactory.getLogger(VectorSearchService.class);
|
||||
|
||||
@Autowired
|
||||
private MilvusServiceClient milvusClient;
|
||||
private MilvusHybridKnowledgeStore knowledgeStore;
|
||||
|
||||
@Autowired
|
||||
private VectorEmbeddingService embeddingService;
|
||||
|
||||
@Autowired
|
||||
private ObjectProvider<VectorStore> vectorStoreProvider;
|
||||
|
||||
@Autowired
|
||||
private ObjectMapper objectMapper;
|
||||
|
||||
/** 检索实现路由：auto / spring / spring-ai / sdk。 */
|
||||
@Value("${retrieval.vector-store.mode:auto}")
|
||||
private String vectorStoreMode = "auto";
|
||||
|
||||
/** similarity -> 兼容 L2 时使用的距离上限，需与后处理归一化配置一致。 */
|
||||
@Value("${retrieval.normalization.max-l2-distance:2.0}")
|
||||
private double maxL2Distance = 2.0;
|
||||
|
||||
/** 非空时只检索该 kb_scope 下的 chunk（多租户/多知识库隔离）。 */
|
||||
@Value("${retrieval.kb-scope:}")
|
||||
private String kbScope = "";
|
||||
/**
|
||||
* dense | hybrid
|
||||
*/
|
||||
@Value("${retrieval.search.mode:dense}")
|
||||
private String searchMode = "dense";
|
||||
|
||||
public List<SearchResult> searchSimilarDocuments(String query, int topK) {
|
||||
return searchSimilarDocuments(query, topK, null);
|
||||
}
|
||||
|
||||
/**
|
||||
* 按 query 召回 topK 相似文档片段。
|
||||
*
|
||||
* @param query 检索文本（会再 embedding）
|
||||
* @param topK 返回条数
|
||||
* @param category 可选 category 过滤；null/blank 表示不过滤
|
||||
*/
|
||||
public List<SearchResult> searchSimilarDocuments(String query, int topK, String category) {
|
||||
String mode = vectorStoreMode == null ? "auto" : vectorStoreMode.trim().toLowerCase();
|
||||
return switch (mode) {
|
||||
case "sdk" -> searchSimilarDocumentsWithSdk(query, topK, category);
|
||||
case "spring", "spring-ai" -> searchSimilarDocumentsWithVectorStore(query, topK, category);
|
||||
case "auto" -> searchWithAutoFallback(query, topK, category);
|
||||
default -> {
|
||||
logger.warn("Unknown retrieval.vector-store.mode={}, using auto mode", vectorStoreMode);
|
||||
yield searchWithAutoFallback(query, topK, category);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/** auto：Spring AI 优先，任意异常则降级 SDK（保证检索可用性）。 */
|
||||
private List<SearchResult> searchWithAutoFallback(String query, int topK, String category) {
|
||||
try {
|
||||
return searchSimilarDocumentsWithVectorStore(query, topK, category);
|
||||
} catch (Exception e) {
|
||||
logger.warn("Spring AI VectorStore retrieval failed, falling back to Milvus SDK: errorType={}",
|
||||
e.getClass().getSimpleName());
|
||||
return searchSimilarDocumentsWithSdk(query, topK, category);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Spring AI VectorStore 路径。
|
||||
* 注意：写入索引仍主要由 Milvus SDK 完成；这里是读适配，依赖双方 metadata schema 一致。
|
||||
*/
|
||||
List<SearchResult> searchSimilarDocumentsWithVectorStore(String query, int topK, String category) {
|
||||
VectorStore vectorStore = vectorStoreProvider != null ? vectorStoreProvider.getIfAvailable() : null;
|
||||
if (vectorStore == null) {
|
||||
throw new IllegalStateException("Spring AI VectorStore bean is unavailable");
|
||||
}
|
||||
|
||||
logger.info("Starting Spring AI VectorStore search: topK={}, category={}, kbScope={}",
|
||||
topK, category, effectiveKbScope());
|
||||
SearchRequest.Builder builder = SearchRequest.builder()
|
||||
.query(query)
|
||||
.topK(topK)
|
||||
// 不做框架层阈值截断，相关性判断交给后处理
|
||||
.similarityThresholdAll();
|
||||
String filterExpression = buildSpringAiFilterExpression(category);
|
||||
if (filterExpression != null) {
|
||||
builder.filterExpression(filterExpression);
|
||||
logger.info("Spring AI VectorStore metadata filter: {}", filterExpression);
|
||||
}
|
||||
|
||||
List<Document> documents = vectorStore.similaritySearch(builder.build());
|
||||
List<SearchResult> results = new ArrayList<>();
|
||||
for (Document document : documents) {
|
||||
SearchResult result = new SearchResult();
|
||||
result.setId(document.getId());
|
||||
result.setContent(document.getText());
|
||||
result.setMetadata(toJson(document.getMetadata()));
|
||||
result.setRawScore(document.getScore());
|
||||
result.setScoreLabel("similarity");
|
||||
// 下游统一按 L2 距离消费，这里做兼容映射
|
||||
result.setScore(toCompatibleL2Distance(document));
|
||||
results.add(result);
|
||||
}
|
||||
logger.info("Spring AI VectorStore search complete, candidates={}", results.size());
|
||||
return results;
|
||||
}
|
||||
|
||||
/** Milvus SDK 原生 L2 检索路径。 */
|
||||
List<SearchResult> searchSimilarDocumentsWithSdk(String query, int topK, String category) {
|
||||
try {
|
||||
logger.info("Starting Milvus SDK search: topK={}, category={}, kbScope={}",
|
||||
topK, category, effectiveKbScope());
|
||||
|
||||
String mode = searchMode == null ? "dense" : searchMode.trim().toLowerCase(Locale.ROOT);
|
||||
List<Float> queryVector = embeddingService.generateQueryVector(query);
|
||||
logger.debug("Query vector generated, dimension={}", queryVector.size());
|
||||
|
||||
SearchParam.Builder searchParamBuilder = SearchParam.newBuilder()
|
||||
.withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME)
|
||||
.withVectorFieldName("vector")
|
||||
.withVectors(Collections.singletonList(queryVector))
|
||||
.withTopK(topK)
|
||||
.withMetricType(io.milvus.param.MetricType.L2)
|
||||
.withOutFields(List.of("id", "content", "metadata"))
|
||||
.withParams("{\"nprobe\":10}");
|
||||
|
||||
String expr = buildSdkFilterExpression(category);
|
||||
if (expr != null) {
|
||||
searchParamBuilder.withExpr(expr);
|
||||
logger.info("Milvus SDK metadata filter: {}", expr);
|
||||
if ("hybrid".equals(mode)) {
|
||||
logger.info("Hybrid dense+BM25 search topK={} category={} collection={}",
|
||||
topK, category, knowledgeStore.collectionName());
|
||||
return knowledgeStore.searchHybrid(query, queryVector, topK, category);
|
||||
}
|
||||
logger.info("Dense search topK={} category={} collection={}",
|
||||
topK, category, knowledgeStore.collectionName());
|
||||
return knowledgeStore.searchDense(query, queryVector, topK, category);
|
||||
}
|
||||
|
||||
R<SearchResults> searchResponse = milvusClient.search(searchParamBuilder.build());
|
||||
if (searchResponse.getStatus() != 0) {
|
||||
throw new RuntimeException("Vector search failed: " + searchResponse.getMessage());
|
||||
}
|
||||
|
||||
SearchResultsWrapper wrapper = new SearchResultsWrapper(searchResponse.getData().getResults());
|
||||
List<SearchResult> results = new ArrayList<>();
|
||||
for (int i = 0; i < wrapper.getRowRecords(0).size(); i++) {
|
||||
SearchResult result = new SearchResult();
|
||||
result.setId((String) wrapper.getIDScore(0).get(i).get("id"));
|
||||
result.setContent((String) wrapper.getFieldData("content", 0).get(i));
|
||||
// Milvus L2：数值越小越相似
|
||||
result.setScore(wrapper.getIDScore(0).get(i).getScore());
|
||||
result.setRawScore((double) result.getScore());
|
||||
result.setScoreLabel("l2_distance");
|
||||
|
||||
Object metadataObj = wrapper.getFieldData("metadata", 0).get(i);
|
||||
if (metadataObj != null) {
|
||||
result.setMetadata(metadataObj.toString());
|
||||
}
|
||||
|
||||
results.add(result);
|
||||
}
|
||||
|
||||
logger.info("Milvus SDK search complete, candidates={}", results.size());
|
||||
return results;
|
||||
} catch (Exception e) {
|
||||
logger.error("Milvus SDK vector search failed: errorType={}", e.getClass().getSimpleName());
|
||||
throw new RuntimeException("Vector search failed: " + e.getMessage(), e);
|
||||
}
|
||||
}
|
||||
|
||||
/** 优先使用 metadata.distance；否则把 similarity 映射为兼容 L2。 */
|
||||
private float toCompatibleL2Distance(Document document) {
|
||||
Double distance = extractDistance(document.getMetadata());
|
||||
if (distance != null) {
|
||||
return distance.floatValue();
|
||||
}
|
||||
return toCompatibleL2Distance(document.getScore());
|
||||
}
|
||||
|
||||
/**
|
||||
* similarity ∈ [0,1] 越大越相似 -> 兼容 L2 距离。
|
||||
* 映射：distance = (1 - similarity) * maxL2Distance
|
||||
*/
|
||||
private float toCompatibleL2Distance(Double similarity) {
|
||||
if (similarity == null) {
|
||||
return (float) maxL2Distance;
|
||||
}
|
||||
double bounded = Math.max(0.0, Math.min(1.0, similarity));
|
||||
return (float) ((1.0 - bounded) * maxL2Distance);
|
||||
}
|
||||
|
||||
private Double extractDistance(Map<String, Object> metadata) {
|
||||
if (metadata == null) {
|
||||
return null;
|
||||
}
|
||||
Object value = metadata.get("distance");
|
||||
if (value instanceof Number number) {
|
||||
return number.doubleValue();
|
||||
}
|
||||
if (value instanceof String text) {
|
||||
try {
|
||||
return Double.parseDouble(text);
|
||||
} catch (NumberFormatException ignored) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
private String toJson(Map<String, Object> metadata) {
|
||||
if (metadata == null || metadata.isEmpty()) {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
return objectMapper.writeValueAsString(metadata);
|
||||
} catch (JsonProcessingException e) {
|
||||
return metadata.toString();
|
||||
}
|
||||
}
|
||||
|
||||
private String escapeFilterValue(String value) {
|
||||
return value.replace("'", "\\'");
|
||||
}
|
||||
|
||||
/** Spring AI filter DSL，例如：category == 'mysql' && kb_scope == 'prod' */
|
||||
String buildSpringAiFilterExpression(String category) {
|
||||
List<String> parts = new ArrayList<>();
|
||||
String categoryFilter = trimToNull(category);
|
||||
if (categoryFilter != null) {
|
||||
parts.add("category == '" + escapeFilterValue(categoryFilter) + "'");
|
||||
}
|
||||
String scopeFilter = effectiveKbScope();
|
||||
if (scopeFilter != null) {
|
||||
parts.add("kb_scope == '" + escapeFilterValue(scopeFilter) + "'");
|
||||
}
|
||||
return parts.isEmpty() ? null : String.join(" && ", parts);
|
||||
}
|
||||
|
||||
/** Milvus boolean expr，字段在 JSON metadata 内。 */
|
||||
String buildSdkFilterExpression(String category) {
|
||||
List<String> parts = new ArrayList<>();
|
||||
String categoryFilter = trimToNull(category);
|
||||
if (categoryFilter != null) {
|
||||
parts.add("metadata[\"category\"] == \"" + escapeMilvusString(categoryFilter) + "\"");
|
||||
}
|
||||
String scopeFilter = effectiveKbScope();
|
||||
if (scopeFilter != null) {
|
||||
parts.add("metadata[\"kb_scope\"] == \"" + escapeMilvusString(scopeFilter) + "\"");
|
||||
}
|
||||
return parts.isEmpty() ? null : String.join(" && ", parts);
|
||||
}
|
||||
|
||||
private String effectiveKbScope() {
|
||||
return trimToNull(kbScope);
|
||||
}
|
||||
|
||||
private String trimToNull(String value) {
|
||||
if (value == null || value.isBlank()) {
|
||||
return null;
|
||||
}
|
||||
return value.trim();
|
||||
}
|
||||
|
||||
private String escapeMilvusString(String value) {
|
||||
return value.replace("\\", "\\\\").replace("\"", "\\\"");
|
||||
}
|
||||
|
||||
/**
|
||||
* 统一检索命中结构。
|
||||
*
|
||||
* <ul>
|
||||
* <li>{@code score}：兼容 L2 距离，供后处理 normalizeL2</li>
|
||||
* <li>{@code rawScore}：底层原始分（similarity 或 l2）</li>
|
||||
* <li>{@code scoreLabel}：解释 rawScore 语义</li>
|
||||
* <li>{@code metadata}：JSON 字符串，含 docId/chunkIndex/title 等</li>
|
||||
* </ul>
|
||||
*/
|
||||
@Setter
|
||||
@Getter
|
||||
public static class SearchResult {
|
||||
private String id;
|
||||
private String content;
|
||||
/**
|
||||
* Compatibility score used by existing lookup relevance normalization.
|
||||
* SDK mode keeps L2 distance; VectorStore mode prefers the Milvus
|
||||
* distance metadata and falls back to similarity mapping.
|
||||
* Compatibility score for post-process normalizeL2.
|
||||
* Dense path: L2 distance. Hybrid path: dense L2 when available.
|
||||
*/
|
||||
private float score;
|
||||
private Double rawScore;
|
||||
|
||||
@@ -0,0 +1,426 @@
|
||||
package com.superbiz.agent.service.milvus;
|
||||
|
||||
import com.google.gson.Gson;
|
||||
import com.google.gson.JsonObject;
|
||||
import com.superbiz.agent.config.MilvusProperties;
|
||||
import com.superbiz.agent.constant.MilvusConstants;
|
||||
import com.superbiz.agent.service.VectorSearchService;
|
||||
import io.milvus.common.clientenum.FunctionType;
|
||||
import io.milvus.v2.client.ConnectConfig;
|
||||
import io.milvus.v2.client.MilvusClientV2;
|
||||
import io.milvus.v2.common.DataType;
|
||||
import io.milvus.v2.common.IndexParam;
|
||||
import io.milvus.v2.service.collection.request.AddFieldReq;
|
||||
import io.milvus.v2.service.collection.request.CreateCollectionReq;
|
||||
import io.milvus.v2.service.collection.request.HasCollectionReq;
|
||||
import io.milvus.v2.service.collection.request.LoadCollectionReq;
|
||||
import io.milvus.v2.service.index.request.CreateIndexReq;
|
||||
import io.milvus.v2.service.vector.request.AnnSearchReq;
|
||||
import io.milvus.v2.service.vector.request.DeleteReq;
|
||||
import io.milvus.v2.service.vector.request.HybridSearchReq;
|
||||
import io.milvus.v2.service.vector.request.InsertReq;
|
||||
import io.milvus.v2.service.vector.request.SearchReq;
|
||||
import io.milvus.v2.service.vector.request.data.BaseVector;
|
||||
import io.milvus.v2.service.vector.request.data.EmbeddedText;
|
||||
import io.milvus.v2.service.vector.request.data.FloatVec;
|
||||
import io.milvus.v2.service.vector.request.ranker.RRFRanker;
|
||||
import io.milvus.v2.service.vector.response.SearchResp;
|
||||
import jakarta.annotation.PreDestroy;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
|
||||
/**
|
||||
* Single knowledge vector backend (Milvus Java SDK v2).
|
||||
*
|
||||
* <p>Supports dense ANN and dense+BM25 hybrid search via {@code hybridSearch} + {@link RRFRanker}.
|
||||
* Legacy {@code MilvusServiceClient} search is not used.</p>
|
||||
*/
|
||||
@Service
|
||||
public class MilvusHybridKnowledgeStore {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(MilvusHybridKnowledgeStore.class);
|
||||
private static final Gson GSON = new Gson();
|
||||
|
||||
public static final String FIELD_ID = "id";
|
||||
public static final String FIELD_CONTENT = "content";
|
||||
public static final String FIELD_SEARCH_TEXT = "search_text";
|
||||
public static final String FIELD_SPARSE = "sparse_vector";
|
||||
public static final String FIELD_DENSE = "vector";
|
||||
public static final String FIELD_METADATA = "metadata";
|
||||
|
||||
private final MilvusProperties milvusProperties;
|
||||
|
||||
@Value("${milvus.collection:biz_hybrid}")
|
||||
private String collectionName = "biz_hybrid";
|
||||
|
||||
@Value("${retrieval.hybrid.rrf-k:60}")
|
||||
private int rrfK = 60;
|
||||
|
||||
@Value("${retrieval.normalization.max-l2-distance:2.0}")
|
||||
private double maxL2Distance = 2.0;
|
||||
|
||||
@Value("${retrieval.kb-scope:}")
|
||||
private String kbScope = "";
|
||||
|
||||
private volatile MilvusClientV2 client;
|
||||
|
||||
public MilvusHybridKnowledgeStore(MilvusProperties milvusProperties) {
|
||||
this.milvusProperties = milvusProperties;
|
||||
}
|
||||
|
||||
public synchronized MilvusClientV2 client() {
|
||||
if (client == null) {
|
||||
client = connect();
|
||||
ensureCollection(client);
|
||||
loadCollection(client);
|
||||
}
|
||||
return client;
|
||||
}
|
||||
|
||||
public String collectionName() {
|
||||
return collectionName;
|
||||
}
|
||||
|
||||
public void upsertChunk(String content,
|
||||
String searchText,
|
||||
List<Float> denseVector,
|
||||
Map<String, Object> metadata,
|
||||
int chunkIndex) {
|
||||
String source = metadata == null ? null : stringVal(metadata.get("_source"));
|
||||
if (source == null) {
|
||||
source = metadata == null ? null : stringVal(metadata.get("source"));
|
||||
}
|
||||
if (source == null) {
|
||||
source = metadata == null ? null : stringVal(metadata.get("docId"));
|
||||
}
|
||||
String idSeed = (source == null ? "chunk" : source) + "_" + chunkIndex;
|
||||
String id = UUID.nameUUIDFromBytes(idSeed.getBytes()).toString();
|
||||
|
||||
JsonObject row = new JsonObject();
|
||||
row.addProperty(FIELD_ID, id);
|
||||
row.addProperty(FIELD_CONTENT, content == null ? "" : content);
|
||||
row.addProperty(FIELD_SEARCH_TEXT, searchText == null ? "" : searchText);
|
||||
row.add(FIELD_DENSE, GSON.toJsonTree(denseVector));
|
||||
row.add(FIELD_METADATA, GSON.toJsonTree(metadata == null ? Map.of() : metadata));
|
||||
|
||||
client().insert(InsertReq.builder()
|
||||
.collectionName(collectionName)
|
||||
.data(List.of(row))
|
||||
.build());
|
||||
}
|
||||
|
||||
public void deleteByDocId(String docId) {
|
||||
if (docId == null || docId.isBlank()) {
|
||||
return;
|
||||
}
|
||||
String filter = "metadata[\"docId\"] == \"" + escapeFilter(docId) + "\"";
|
||||
client().delete(DeleteReq.builder()
|
||||
.collectionName(collectionName)
|
||||
.filter(filter)
|
||||
.build());
|
||||
}
|
||||
|
||||
public void deleteBySource(String sourcePath) {
|
||||
if (sourcePath == null || sourcePath.isBlank()) {
|
||||
return;
|
||||
}
|
||||
String normalized = sourcePath.replace('\\', '/');
|
||||
String filter = "metadata[\"_source\"] == \"" + escapeFilter(normalized) + "\"";
|
||||
client().delete(DeleteReq.builder()
|
||||
.collectionName(collectionName)
|
||||
.filter(filter)
|
||||
.build());
|
||||
}
|
||||
|
||||
public List<VectorSearchService.SearchResult> searchDense(String queryEmbeddingText,
|
||||
List<Float> queryVector,
|
||||
int topK,
|
||||
String category) {
|
||||
String filter = buildFilter(category);
|
||||
SearchReq.SearchReqBuilder builder = SearchReq.builder()
|
||||
.collectionName(collectionName)
|
||||
.annsField(FIELD_DENSE)
|
||||
.data(List.of(new FloatVec(queryVector)))
|
||||
.topK(topK)
|
||||
.outputFields(List.of(FIELD_ID, FIELD_CONTENT, FIELD_METADATA))
|
||||
.metricType(IndexParam.MetricType.L2);
|
||||
if (filter != null) {
|
||||
builder.filter(filter);
|
||||
}
|
||||
SearchResp resp = client().search(builder.build());
|
||||
return toSearchResults(resp, "l2_distance", false);
|
||||
}
|
||||
|
||||
/**
|
||||
* Dense + BM25 hybrid fused by RRF. Dense L2 scores are attached when the same id
|
||||
* appears in a parallel dense search so quality thresholds stay meaningful.
|
||||
*/
|
||||
public List<VectorSearchService.SearchResult> searchHybrid(String queryText,
|
||||
List<Float> queryVector,
|
||||
int topK,
|
||||
String category) {
|
||||
String filter = buildFilter(category);
|
||||
int pathTopK = Math.max(topK, 10);
|
||||
|
||||
AnnSearchReq.AnnSearchReqBuilder denseAnn = AnnSearchReq.builder()
|
||||
.vectorFieldName(FIELD_DENSE)
|
||||
.vectors(List.of((BaseVector) new FloatVec(queryVector)))
|
||||
.topK(pathTopK)
|
||||
.metricType(IndexParam.MetricType.L2)
|
||||
.params("{\"nprobe\":10}");
|
||||
if (filter != null) {
|
||||
denseAnn.filter(filter);
|
||||
}
|
||||
|
||||
AnnSearchReq.AnnSearchReqBuilder sparseAnn = AnnSearchReq.builder()
|
||||
.vectorFieldName(FIELD_SPARSE)
|
||||
.vectors(List.of((BaseVector) new EmbeddedText(queryText == null ? "" : queryText)))
|
||||
.topK(pathTopK)
|
||||
.metricType(IndexParam.MetricType.BM25);
|
||||
if (filter != null) {
|
||||
sparseAnn.filter(filter);
|
||||
}
|
||||
|
||||
HybridSearchReq hybridReq = HybridSearchReq.builder()
|
||||
.collectionName(collectionName)
|
||||
.searchRequests(List.of(denseAnn.build(), sparseAnn.build()))
|
||||
.ranker(new RRFRanker(Math.max(1, rrfK)))
|
||||
.topK(topK)
|
||||
.outFields(List.of(FIELD_ID, FIELD_CONTENT, FIELD_METADATA))
|
||||
.build();
|
||||
|
||||
SearchResp hybridResp = client().hybridSearch(hybridReq);
|
||||
List<VectorSearchService.SearchResult> fused = toSearchResults(hybridResp, "rrf_fused", true);
|
||||
|
||||
// Attach dense-compatible L2 when available.
|
||||
Map<String, Float> denseScores = new HashMap<>();
|
||||
try {
|
||||
for (VectorSearchService.SearchResult denseHit :
|
||||
searchDense(queryText, queryVector, pathTopK, category)) {
|
||||
if (denseHit.getId() != null) {
|
||||
denseScores.put(denseHit.getId(), denseHit.getScore());
|
||||
}
|
||||
}
|
||||
} catch (Exception e) {
|
||||
log.warn("Dense score enrichment failed: {}", e.getMessage());
|
||||
}
|
||||
for (VectorSearchService.SearchResult hit : fused) {
|
||||
Float dense = denseScores.get(hit.getId());
|
||||
if (dense != null) {
|
||||
hit.setScore(dense);
|
||||
hit.setScoreLabel("l2_distance");
|
||||
} else {
|
||||
// BM25-only hit: treat as weak for legacy thresholds
|
||||
hit.setScore((float) maxL2Distance);
|
||||
hit.setScoreLabel("bm25_only_no_dense");
|
||||
}
|
||||
}
|
||||
return fused;
|
||||
}
|
||||
|
||||
private List<VectorSearchService.SearchResult> toSearchResults(SearchResp resp,
|
||||
String scoreLabel,
|
||||
boolean fused) {
|
||||
List<VectorSearchService.SearchResult> out = new ArrayList<>();
|
||||
if (resp == null || resp.getSearchResults() == null || resp.getSearchResults().isEmpty()) {
|
||||
return out;
|
||||
}
|
||||
List<SearchResp.SearchResult> first = resp.getSearchResults().get(0);
|
||||
if (first == null) {
|
||||
return out;
|
||||
}
|
||||
for (SearchResp.SearchResult row : first) {
|
||||
VectorSearchService.SearchResult mapped = new VectorSearchService.SearchResult();
|
||||
Object id = row.getId();
|
||||
mapped.setId(id == null ? null : String.valueOf(id));
|
||||
Map<String, Object> entity = row.getEntity() == null ? Map.of() : row.getEntity();
|
||||
Object content = entity.get(FIELD_CONTENT);
|
||||
mapped.setContent(content == null ? null : String.valueOf(content));
|
||||
Object metadata = entity.get(FIELD_METADATA);
|
||||
if (metadata instanceof JsonObject jsonObject) {
|
||||
mapped.setMetadata(jsonObject.toString());
|
||||
} else if (metadata instanceof Map<?, ?> map) {
|
||||
mapped.setMetadata(GSON.toJson(map));
|
||||
} else if (metadata != null) {
|
||||
mapped.setMetadata(String.valueOf(metadata));
|
||||
}
|
||||
Float score = row.getScore();
|
||||
mapped.setRawScore(score == null ? null : score.doubleValue());
|
||||
mapped.setScoreLabel(scoreLabel);
|
||||
if (fused) {
|
||||
// temporary; may be overwritten with dense L2
|
||||
mapped.setScore(score == null ? (float) maxL2Distance : invertUnknownScore(score));
|
||||
} else {
|
||||
mapped.setScore(score == null ? (float) maxL2Distance : score);
|
||||
}
|
||||
out.add(mapped);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
private float invertUnknownScore(float score) {
|
||||
// RRF-like small scores: map higher better -> small L2-like distance
|
||||
double bounded = Math.max(0.0, Math.min(1.0, score));
|
||||
if (score > 1.0f) {
|
||||
// already distance-like
|
||||
return score;
|
||||
}
|
||||
return (float) ((1.0 - bounded) * maxL2Distance);
|
||||
}
|
||||
|
||||
private String buildFilter(String category) {
|
||||
List<String> parts = new ArrayList<>();
|
||||
String categoryFilter = trimToNull(category);
|
||||
if (categoryFilter != null) {
|
||||
parts.add("metadata[\"category\"] == \"" + escapeFilter(categoryFilter) + "\"");
|
||||
}
|
||||
String scope = trimToNull(kbScope);
|
||||
if (scope != null) {
|
||||
parts.add("metadata[\"kb_scope\"] == \"" + escapeFilter(scope) + "\"");
|
||||
}
|
||||
return parts.isEmpty() ? null : String.join(" && ", parts);
|
||||
}
|
||||
|
||||
private MilvusClientV2 connect() {
|
||||
String uri;
|
||||
if (milvusProperties.isSecure() || milvusProperties.getPort() == 443) {
|
||||
uri = "https://" + milvusProperties.getHost() + ":" + milvusProperties.getPort();
|
||||
} else {
|
||||
uri = "http://" + milvusProperties.getHost() + ":" + milvusProperties.getPort();
|
||||
}
|
||||
ConnectConfig.ConnectConfigBuilder builder = ConnectConfig.builder()
|
||||
.uri(uri)
|
||||
.connectTimeoutMs(milvusProperties.getTimeout() == null ? 10000L : milvusProperties.getTimeout());
|
||||
if (milvusProperties.getToken() != null && !milvusProperties.getToken().isBlank()) {
|
||||
builder.token(milvusProperties.getToken());
|
||||
builder.secure(true);
|
||||
} else if (milvusProperties.getUsername() != null && !milvusProperties.getUsername().isBlank()) {
|
||||
builder.username(milvusProperties.getUsername());
|
||||
builder.password(milvusProperties.getPassword());
|
||||
}
|
||||
if (milvusProperties.getDatabase() != null && !milvusProperties.getDatabase().isBlank()) {
|
||||
builder.dbName(milvusProperties.getDatabase());
|
||||
}
|
||||
log.info("Connecting MilvusClientV2 uri={} db={} collection={}",
|
||||
uri, milvusProperties.getDatabase(), collectionName);
|
||||
return new MilvusClientV2(builder.build());
|
||||
}
|
||||
|
||||
private void ensureCollection(MilvusClientV2 milvusClient) {
|
||||
Boolean exists = milvusClient.hasCollection(HasCollectionReq.builder()
|
||||
.collectionName(collectionName)
|
||||
.build());
|
||||
if (Boolean.TRUE.equals(exists)) {
|
||||
log.info("Hybrid collection '{}' already exists", collectionName);
|
||||
return;
|
||||
}
|
||||
log.info("Creating hybrid collection '{}'", collectionName);
|
||||
|
||||
CreateCollectionReq.CollectionSchema schema = milvusClient.createSchema();
|
||||
schema.setEnableDynamicField(false);
|
||||
schema.addField(AddFieldReq.builder()
|
||||
.fieldName(FIELD_ID)
|
||||
.dataType(DataType.VarChar)
|
||||
.maxLength(MilvusConstants.ID_MAX_LENGTH)
|
||||
.isPrimaryKey(true)
|
||||
.autoID(false)
|
||||
.build());
|
||||
schema.addField(AddFieldReq.builder()
|
||||
.fieldName(FIELD_CONTENT)
|
||||
.dataType(DataType.VarChar)
|
||||
.maxLength(MilvusConstants.CONTENT_MAX_LENGTH)
|
||||
.build());
|
||||
schema.addField(AddFieldReq.builder()
|
||||
.fieldName(FIELD_SEARCH_TEXT)
|
||||
.dataType(DataType.VarChar)
|
||||
.maxLength(MilvusConstants.CONTENT_MAX_LENGTH)
|
||||
.enableAnalyzer(true)
|
||||
.build());
|
||||
schema.addField(AddFieldReq.builder()
|
||||
.fieldName(FIELD_SPARSE)
|
||||
.dataType(DataType.SparseFloatVector)
|
||||
.build());
|
||||
schema.addField(AddFieldReq.builder()
|
||||
.fieldName(FIELD_DENSE)
|
||||
.dataType(DataType.FloatVector)
|
||||
.dimension(milvusProperties.getVectorDim())
|
||||
.build());
|
||||
schema.addField(AddFieldReq.builder()
|
||||
.fieldName(FIELD_METADATA)
|
||||
.dataType(DataType.JSON)
|
||||
.build());
|
||||
schema.addFunction(CreateCollectionReq.Function.builder()
|
||||
.functionType(FunctionType.BM25)
|
||||
.name("bm25_fn")
|
||||
.inputFieldNames(List.of(FIELD_SEARCH_TEXT))
|
||||
.outputFieldNames(List.of(FIELD_SPARSE))
|
||||
.build());
|
||||
|
||||
milvusClient.createCollection(CreateCollectionReq.builder()
|
||||
.collectionName(collectionName)
|
||||
.collectionSchema(schema)
|
||||
.description("Knowledge hybrid dense+BM25 collection")
|
||||
.numShards(MilvusConstants.DEFAULT_SHARD_NUMBER)
|
||||
.build());
|
||||
|
||||
List<IndexParam> indexes = List.of(
|
||||
IndexParam.builder()
|
||||
.fieldName(FIELD_DENSE)
|
||||
.indexType(IndexParam.IndexType.IVF_FLAT)
|
||||
.metricType(IndexParam.MetricType.L2)
|
||||
.extraParams(Map.of("nlist", 128))
|
||||
.build(),
|
||||
IndexParam.builder()
|
||||
.fieldName(FIELD_SPARSE)
|
||||
.indexType(IndexParam.IndexType.SPARSE_INVERTED_INDEX)
|
||||
.metricType(IndexParam.MetricType.BM25)
|
||||
.build()
|
||||
);
|
||||
milvusClient.createIndex(CreateIndexReq.builder()
|
||||
.collectionName(collectionName)
|
||||
.indexParams(indexes)
|
||||
.build());
|
||||
log.info("Hybrid collection '{}' created with dense+BM25 indexes", collectionName);
|
||||
}
|
||||
|
||||
private void loadCollection(MilvusClientV2 milvusClient) {
|
||||
milvusClient.loadCollection(LoadCollectionReq.builder()
|
||||
.collectionName(collectionName)
|
||||
.build());
|
||||
}
|
||||
|
||||
@PreDestroy
|
||||
public void close() {
|
||||
if (client != null) {
|
||||
try {
|
||||
client.close();
|
||||
} catch (Exception e) {
|
||||
log.warn("Error closing MilvusClientV2: {}", e.getMessage());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static String escapeFilter(String value) {
|
||||
return value.replace("\\", "\\\\").replace("\"", "\\\"");
|
||||
}
|
||||
|
||||
private static String trimToNull(String value) {
|
||||
if (value == null || value.isBlank()) {
|
||||
return null;
|
||||
}
|
||||
return value.trim();
|
||||
}
|
||||
|
||||
private static String stringVal(Object value) {
|
||||
return value == null ? null : String.valueOf(value);
|
||||
}
|
||||
}
|
||||
+8
-119
@@ -2,22 +2,19 @@ package com.superbiz.agent.service.retrieval;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.service.VectorSearchService;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* {@link KnowledgeSearchPort} backed by the existing vector search facade.
|
||||
* {@link KnowledgeSearchPort} adapter.
|
||||
*
|
||||
* <ul>
|
||||
* <li>{@code DENSE}: single dense path (legacy behavior)</li>
|
||||
* <li>{@code HYBRID}: dense unfiltered + optional dense filtered + lexical rank, fused by RRF</li>
|
||||
* </ul>
|
||||
* <p>Delegates to {@link VectorSearchService}, which is backed solely by
|
||||
* Milvus V2 dense / dense+BM25 hybrid store. Mode selection lives in
|
||||
* {@code retrieval.search.mode}.</p>
|
||||
*/
|
||||
@Component
|
||||
public class VectorKnowledgeSearchAdapter implements KnowledgeSearchPort {
|
||||
@@ -25,21 +22,6 @@ public class VectorKnowledgeSearchAdapter implements KnowledgeSearchPort {
|
||||
private final VectorSearchService vectorSearchService;
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
@Value("${retrieval.search.mode:dense}")
|
||||
private String configuredMode = "dense";
|
||||
|
||||
@Value("${retrieval.hybrid.rrf-k:60}")
|
||||
private int rrfK = 60;
|
||||
|
||||
@Value("${retrieval.hybrid.weight.dense-unfiltered:1.0}")
|
||||
private double weightDenseUnfiltered = 1.0;
|
||||
|
||||
@Value("${retrieval.hybrid.weight.dense-filtered:1.0}")
|
||||
private double weightDenseFiltered = 1.0;
|
||||
|
||||
@Value("${retrieval.hybrid.weight.lexical:1.0}")
|
||||
private double weightLexical = 1.0;
|
||||
|
||||
public VectorKnowledgeSearchAdapter(VectorSearchService vectorSearchService, ObjectMapper objectMapper) {
|
||||
this.vectorSearchService = vectorSearchService;
|
||||
this.objectMapper = objectMapper;
|
||||
@@ -47,100 +29,14 @@ public class VectorKnowledgeSearchAdapter implements KnowledgeSearchPort {
|
||||
|
||||
@Override
|
||||
public List<KnowledgeSearchHit> search(KnowledgeSearchRequest request) {
|
||||
KnowledgeSearchMode mode = resolveMode(request.mode());
|
||||
if (mode == KnowledgeSearchMode.HYBRID) {
|
||||
return searchHybrid(request);
|
||||
}
|
||||
return searchDense(request.query(), request.topK(), request.categoryFilter());
|
||||
}
|
||||
|
||||
private List<KnowledgeSearchHit> searchDense(String query, int topK, String categoryFilter) {
|
||||
// Mode is global on VectorSearchService; request.mode is advisory for future per-call overrides.
|
||||
List<VectorSearchService.SearchResult> results = vectorSearchService.searchSimilarDocuments(
|
||||
query, topK, categoryFilter);
|
||||
request.query(),
|
||||
request.topK(),
|
||||
request.categoryFilter());
|
||||
return toHits(results);
|
||||
}
|
||||
|
||||
private List<KnowledgeSearchHit> searchHybrid(KnowledgeSearchRequest request) {
|
||||
String query = request.query();
|
||||
int topK = request.topK();
|
||||
String category = trimToNull(request.categoryFilter());
|
||||
|
||||
List<KnowledgeSearchHit> unfiltered = searchDense(query, topK, null);
|
||||
List<KnowledgeSearchHit> filtered = category == null
|
||||
? List.of()
|
||||
: searchDense(query, topK, category);
|
||||
|
||||
Map<String, KnowledgeSearchHit> unionByKey = new LinkedHashMap<>();
|
||||
for (KnowledgeSearchHit hit : unfiltered) {
|
||||
unionByKey.putIfAbsent(hit.evidenceKey(), hit);
|
||||
}
|
||||
for (KnowledgeSearchHit hit : filtered) {
|
||||
unionByKey.putIfAbsent(hit.evidenceKey(), hit);
|
||||
}
|
||||
List<KnowledgeSearchHit> union = new ArrayList<>(unionByKey.values());
|
||||
List<KnowledgeSearchHit> lexical = LexicalRanker.rank(query, union);
|
||||
|
||||
List<RrfFusion.RankedPath<KnowledgeSearchHit>> paths = new ArrayList<>();
|
||||
paths.add(new RrfFusion.RankedPath<>("dense_unfiltered", unfiltered, weightDenseUnfiltered));
|
||||
if (!filtered.isEmpty()) {
|
||||
paths.add(new RrfFusion.RankedPath<>("dense_filtered", filtered, weightDenseFiltered));
|
||||
}
|
||||
if (!lexical.isEmpty()) {
|
||||
paths.add(new RrfFusion.RankedPath<>("lexical", lexical, weightLexical));
|
||||
}
|
||||
|
||||
List<RrfFusion.Scored<KnowledgeSearchHit>> fused = RrfFusion.fuse(
|
||||
paths, rrfK, KnowledgeSearchHit::evidenceKey);
|
||||
|
||||
List<KnowledgeSearchHit> ordered = new ArrayList<>();
|
||||
int rank = 1;
|
||||
for (RrfFusion.Scored<KnowledgeSearchHit> scored : fused) {
|
||||
if (ordered.size() >= topK) {
|
||||
break;
|
||||
}
|
||||
KnowledgeSearchHit base = scored.item();
|
||||
Map<String, String> metadata = new LinkedHashMap<>(
|
||||
base.metadata() == null ? Map.of() : base.metadata());
|
||||
metadata.put("fusedScore", Double.toString(scored.rrfScore()));
|
||||
metadata.put("fusionRanks", scored.ranks().toString());
|
||||
metadata.put("fusionRank", Integer.toString(rank));
|
||||
ordered.add(new KnowledgeSearchHit(
|
||||
base.id(),
|
||||
base.content(),
|
||||
base.score(),
|
||||
base.rawScore(),
|
||||
base.scoreLabel(),
|
||||
base.metadataJson(),
|
||||
metadata,
|
||||
base.docId(),
|
||||
base.chunkIndex(),
|
||||
base.evidenceKey(),
|
||||
base.source(),
|
||||
base.title(),
|
||||
base.breadcrumb(),
|
||||
rank
|
||||
));
|
||||
rank++;
|
||||
}
|
||||
return ordered;
|
||||
}
|
||||
|
||||
private KnowledgeSearchMode resolveMode(KnowledgeSearchMode requestMode) {
|
||||
if (requestMode == KnowledgeSearchMode.HYBRID) {
|
||||
return KnowledgeSearchMode.HYBRID;
|
||||
}
|
||||
if (requestMode == KnowledgeSearchMode.DENSE) {
|
||||
// Allow global config to force hybrid even if caller passes DENSE default.
|
||||
String configured = configuredMode == null ? "dense" : configuredMode.trim().toLowerCase(Locale.ROOT);
|
||||
if ("hybrid".equals(configured)) {
|
||||
return KnowledgeSearchMode.HYBRID;
|
||||
}
|
||||
return KnowledgeSearchMode.DENSE;
|
||||
}
|
||||
String configured = configuredMode == null ? "dense" : configuredMode.trim().toLowerCase(Locale.ROOT);
|
||||
return "hybrid".equals(configured) ? KnowledgeSearchMode.HYBRID : KnowledgeSearchMode.DENSE;
|
||||
}
|
||||
|
||||
private List<KnowledgeSearchHit> toHits(List<VectorSearchService.SearchResult> results) {
|
||||
if (results == null || results.isEmpty()) {
|
||||
return List.of();
|
||||
@@ -201,11 +97,4 @@ public class VectorKnowledgeSearchAdapter implements KnowledgeSearchPort {
|
||||
return Map.of();
|
||||
}
|
||||
}
|
||||
|
||||
private static String trimToNull(String value) {
|
||||
if (value == null || value.isBlank()) {
|
||||
return null;
|
||||
}
|
||||
return value.trim();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -25,6 +25,8 @@ milvus:
|
||||
token: ${MILVUS_TOKEN}
|
||||
secure: true
|
||||
vector-dim: 1024 # BGE-M3 = 1024，换模型时同步改
|
||||
# dense+BM25 hybrid collection (requires reindex; legacy "biz" is not used by knowledge path)
|
||||
collection: biz_hybrid
|
||||
|
||||
# =====================================================
|
||||
# 模型路由配置
|
||||
@@ -162,16 +164,24 @@ rag:
|
||||
enabled: false
|
||||
content-preview-limit: 300
|
||||
|
||||
# 检索归一化配置
|
||||
# 检索配置（单一 Milvus V2 后端；已移除 sdk/spring/auto 路由）
|
||||
retrieval:
|
||||
kb-scope: "" # empty means search all legacy documents; use rag-eval for eval seed docs
|
||||
vector-store:
|
||||
mode: auto # auto | spring-ai | sdk
|
||||
kb-scope: "" # empty means search all documents in hybrid collection
|
||||
search:
|
||||
mode: hybrid # dense | hybrid (dense + BM25 RRF)
|
||||
hybrid:
|
||||
rrf-k: 60
|
||||
normalization:
|
||||
max-l2-distance: 2.0 # L2 距离上界（BGE-M3 单位向量 = 2.0）
|
||||
highly-relevant-threshold: 0.75 # similarity >= 0.75 → HIGHLY_RELEVANT
|
||||
reference-threshold: 0.5 # similarity >= 0.5 → REFERENCE
|
||||
|
||||
# RAG 召回宽度（Delivery 1）
|
||||
rag:
|
||||
retrieve-k: 20
|
||||
return-n: 5
|
||||
max-chunks-per-document: 2
|
||||
|
||||
# Prometheus 配置
|
||||
prometheus:
|
||||
base-url: http://localhost:9090
|
||||
|
||||
@@ -1,208 +1,61 @@
|
||||
package com.superbiz.agent.service;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.service.milvus.MilvusHybridKnowledgeStore;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockito.ArgumentCaptor;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.beans.factory.ObjectProvider;
|
||||
import org.springframework.test.util.ReflectionTestUtils;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.eq;
|
||||
import static org.mockito.Mockito.doReturn;
|
||||
import static org.mockito.ArgumentMatchers.isNull;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.never;
|
||||
import static org.mockito.Mockito.spy;
|
||||
import static org.mockito.Mockito.verify;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
class VectorSearchServiceTest {
|
||||
|
||||
@Test
|
||||
void sdkModeBypassesVectorStore() {
|
||||
VectorSearchService service = spy(new VectorSearchService());
|
||||
setMode(service, "sdk");
|
||||
VectorSearchService.SearchResult expected = result("sdk-doc", 0.2f);
|
||||
doReturn(List.of(expected))
|
||||
.when(service).searchSimilarDocumentsWithSdk("query", 3, null);
|
||||
void denseModeCallsHybridStoreDenseSearch() {
|
||||
MilvusHybridKnowledgeStore store = mock(MilvusHybridKnowledgeStore.class);
|
||||
VectorEmbeddingService embeddingService = mock(VectorEmbeddingService.class);
|
||||
when(embeddingService.generateQueryVector("query")).thenReturn(List.of(0.1f, 0.2f));
|
||||
VectorSearchService.SearchResult expected = result("doc-1", 0.2f);
|
||||
when(store.searchDense(eq("query"), eq(List.of(0.1f, 0.2f)), eq(3), isNull()))
|
||||
.thenReturn(List.of(expected));
|
||||
when(store.collectionName()).thenReturn("biz_hybrid");
|
||||
|
||||
VectorSearchService service = new VectorSearchService();
|
||||
ReflectionTestUtils.setField(service, "knowledgeStore", store);
|
||||
ReflectionTestUtils.setField(service, "embeddingService", embeddingService);
|
||||
ReflectionTestUtils.setField(service, "searchMode", "dense");
|
||||
|
||||
List<VectorSearchService.SearchResult> results = service.searchSimilarDocuments("query", 3, null);
|
||||
|
||||
assertEquals(List.of(expected), results);
|
||||
verify(service, never()).searchSimilarDocumentsWithVectorStore(any(), eq(3), any());
|
||||
verify(store).searchDense(eq("query"), eq(List.of(0.1f, 0.2f)), eq(3), isNull());
|
||||
}
|
||||
|
||||
@Test
|
||||
void autoModeUsesVectorStoreWhenAvailable() {
|
||||
VectorStore vectorStore = mock(VectorStore.class);
|
||||
ObjectProvider<VectorStore> provider = mock(ObjectProvider.class);
|
||||
when(provider.getIfAvailable()).thenReturn(vectorStore);
|
||||
when(vectorStore.similaritySearch(any(SearchRequest.class))).thenReturn(List.of(
|
||||
Document.builder()
|
||||
.id("spring-doc")
|
||||
.text("spring content")
|
||||
.metadata(Map.of("_source", "spring.md", "category", "api"))
|
||||
.score(0.8)
|
||||
.build()
|
||||
));
|
||||
void hybridModeCallsHybridStoreHybridSearch() {
|
||||
MilvusHybridKnowledgeStore store = mock(MilvusHybridKnowledgeStore.class);
|
||||
VectorEmbeddingService embeddingService = mock(VectorEmbeddingService.class);
|
||||
when(embeddingService.generateQueryVector("pool")).thenReturn(List.of(0.3f));
|
||||
VectorSearchService.SearchResult expected = result("doc-h", 0.4f);
|
||||
when(store.searchHybrid(eq("pool"), eq(List.of(0.3f)), eq(5), eq("mysql")))
|
||||
.thenReturn(List.of(expected));
|
||||
when(store.collectionName()).thenReturn("biz_hybrid");
|
||||
|
||||
VectorSearchService service = new VectorSearchService();
|
||||
setMode(service, "auto");
|
||||
setVectorStore(service, provider);
|
||||
ReflectionTestUtils.setField(service, "knowledgeStore", store);
|
||||
ReflectionTestUtils.setField(service, "embeddingService", embeddingService);
|
||||
ReflectionTestUtils.setField(service, "searchMode", "hybrid");
|
||||
|
||||
List<VectorSearchService.SearchResult> results = service.searchSimilarDocuments("query", 3, null);
|
||||
List<VectorSearchService.SearchResult> results = service.searchSimilarDocuments("pool", 5, "mysql");
|
||||
|
||||
assertEquals(1, results.size());
|
||||
assertEquals("spring-doc", results.get(0).getId());
|
||||
assertEquals("similarity", results.get(0).getScoreLabel());
|
||||
assertEquals(0.8, results.get(0).getRawScore(), 0.0001);
|
||||
assertEquals(0.4f, results.get(0).getScore(), 0.0001);
|
||||
assertTrue(results.get(0).getMetadata().contains("spring.md"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void vectorStoreSearchUsesDistanceMetadataAsCompatibleScore() {
|
||||
VectorStore vectorStore = mock(VectorStore.class);
|
||||
ObjectProvider<VectorStore> provider = mock(ObjectProvider.class);
|
||||
when(provider.getIfAvailable()).thenReturn(vectorStore);
|
||||
when(vectorStore.similaritySearch(any(SearchRequest.class))).thenReturn(List.of(
|
||||
Document.builder()
|
||||
.id("spring-doc")
|
||||
.text("spring content")
|
||||
.metadata(Map.of("distance", 0.5659486, "category", "api"))
|
||||
.score(0.4340514)
|
||||
.build()
|
||||
));
|
||||
|
||||
VectorSearchService service = new VectorSearchService();
|
||||
setMode(service, "auto");
|
||||
setVectorStore(service, provider);
|
||||
|
||||
List<VectorSearchService.SearchResult> results = service.searchSimilarDocuments("query", 3, null);
|
||||
|
||||
assertEquals(1, results.size());
|
||||
assertEquals("similarity", results.get(0).getScoreLabel());
|
||||
assertEquals(0.4340514, results.get(0).getRawScore(), 0.0001);
|
||||
assertEquals(0.5659486f, results.get(0).getScore(), 0.0001);
|
||||
}
|
||||
|
||||
@Test
|
||||
void autoModeFallsBackToSdkWhenVectorStoreFails() {
|
||||
VectorStore vectorStore = mock(VectorStore.class);
|
||||
ObjectProvider<VectorStore> provider = mock(ObjectProvider.class);
|
||||
when(provider.getIfAvailable()).thenReturn(vectorStore);
|
||||
when(vectorStore.similaritySearch(any(SearchRequest.class))).thenThrow(new RuntimeException("vectorstore down"));
|
||||
|
||||
VectorSearchService service = spy(new VectorSearchService());
|
||||
setMode(service, "auto");
|
||||
setVectorStore(service, provider);
|
||||
VectorSearchService.SearchResult fallback = result("sdk-doc", 0.3f);
|
||||
doReturn(List.of(fallback))
|
||||
.when(service).searchSimilarDocumentsWithSdk("query", 3, null);
|
||||
|
||||
List<VectorSearchService.SearchResult> results = service.searchSimilarDocuments("query", 3, null);
|
||||
|
||||
assertEquals(List.of(fallback), results);
|
||||
}
|
||||
|
||||
@Test
|
||||
void autoModeFallsBackToSdkWhenVectorStoreUnavailable() {
|
||||
ObjectProvider<VectorStore> provider = mock(ObjectProvider.class);
|
||||
when(provider.getIfAvailable()).thenReturn(null);
|
||||
|
||||
VectorSearchService service = spy(new VectorSearchService());
|
||||
setMode(service, "auto");
|
||||
setVectorStore(service, provider);
|
||||
VectorSearchService.SearchResult fallback = result("sdk-doc", 0.3f);
|
||||
doReturn(List.of(fallback))
|
||||
.when(service).searchSimilarDocumentsWithSdk("query", 3, null);
|
||||
|
||||
List<VectorSearchService.SearchResult> results = service.searchSimilarDocuments("query", 3, null);
|
||||
|
||||
assertEquals(List.of(fallback), results);
|
||||
}
|
||||
|
||||
@Test
|
||||
void vectorStoreSearchUsesCategoryFilter() {
|
||||
VectorStore vectorStore = mock(VectorStore.class);
|
||||
ObjectProvider<VectorStore> provider = mock(ObjectProvider.class);
|
||||
when(provider.getIfAvailable()).thenReturn(vectorStore);
|
||||
when(vectorStore.similaritySearch(any(SearchRequest.class))).thenReturn(List.of());
|
||||
VectorSearchService service = new VectorSearchService();
|
||||
setMode(service, "spring-ai");
|
||||
setVectorStore(service, provider);
|
||||
|
||||
service.searchSimilarDocuments("query", 5, "api");
|
||||
|
||||
ArgumentCaptor<SearchRequest> requestCaptor = ArgumentCaptor.forClass(SearchRequest.class);
|
||||
verify(vectorStore).similaritySearch(requestCaptor.capture());
|
||||
SearchRequest request = requestCaptor.getValue();
|
||||
assertEquals("query", request.getQuery());
|
||||
assertEquals(5, request.getTopK());
|
||||
assertTrue(request.hasFilterExpression());
|
||||
assertTrue(request.toString().contains("category"));
|
||||
assertTrue(request.toString().contains("api"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void springModeUsesVectorStoreAlias() {
|
||||
VectorStore vectorStore = mock(VectorStore.class);
|
||||
ObjectProvider<VectorStore> provider = mock(ObjectProvider.class);
|
||||
when(provider.getIfAvailable()).thenReturn(vectorStore);
|
||||
when(vectorStore.similaritySearch(any(SearchRequest.class))).thenReturn(List.of());
|
||||
|
||||
VectorSearchService service = new VectorSearchService();
|
||||
setMode(service, "spring");
|
||||
setVectorStore(service, provider);
|
||||
|
||||
service.searchSimilarDocuments("query", 5, "api");
|
||||
|
||||
verify(vectorStore).similaritySearch(any(SearchRequest.class));
|
||||
}
|
||||
|
||||
@Test
|
||||
void defaultScopeDoesNotAddMetadataFilter() {
|
||||
VectorSearchService service = new VectorSearchService();
|
||||
setKbScope(service, "");
|
||||
|
||||
assertNull(service.buildSpringAiFilterExpression(null));
|
||||
assertNull(service.buildSdkFilterExpression(null));
|
||||
assertEquals("category == 'api'", service.buildSpringAiFilterExpression("api"));
|
||||
assertEquals("metadata[\"category\"] == \"api\"", service.buildSdkFilterExpression("api"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void configuredScopeCombinesWithCategoryFilter() {
|
||||
VectorSearchService service = new VectorSearchService();
|
||||
setKbScope(service, "rag-eval");
|
||||
|
||||
assertEquals("kb_scope == 'rag-eval'", service.buildSpringAiFilterExpression(null));
|
||||
assertEquals("category == 'api' && kb_scope == 'rag-eval'",
|
||||
service.buildSpringAiFilterExpression("api"));
|
||||
assertEquals("metadata[\"category\"] == \"api\" && metadata[\"kb_scope\"] == \"rag-eval\"",
|
||||
service.buildSdkFilterExpression("api"));
|
||||
}
|
||||
|
||||
private static void setMode(VectorSearchService service, String mode) {
|
||||
ReflectionTestUtils.setField(service, "vectorStoreMode", mode);
|
||||
}
|
||||
|
||||
private static void setVectorStore(VectorSearchService service, ObjectProvider<VectorStore> provider) {
|
||||
ReflectionTestUtils.setField(service, "vectorStoreProvider", provider);
|
||||
ReflectionTestUtils.setField(service, "objectMapper", new ObjectMapper());
|
||||
ReflectionTestUtils.setField(service, "maxL2Distance", 2.0);
|
||||
}
|
||||
|
||||
private static void setKbScope(VectorSearchService service, String kbScope) {
|
||||
ReflectionTestUtils.setField(service, "kbScope", kbScope);
|
||||
assertEquals("doc-h", results.get(0).getId());
|
||||
verify(store).searchHybrid(eq("pool"), eq(List.of(0.3f)), eq(5), eq("mysql"));
|
||||
}
|
||||
|
||||
private static VectorSearchService.SearchResult result(String id, float score) {
|
||||
|
||||
+12
-17
@@ -3,41 +3,36 @@ package com.superbiz.agent.service.retrieval;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.service.VectorSearchService;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.test.util.ReflectionTestUtils;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.verify;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
class VectorKnowledgeSearchAdapterHybridTest {
|
||||
|
||||
@Test
|
||||
void hybridFusesFilteredAndUnfilteredDensePaths() {
|
||||
void adapterMapsStoreHitsWithChunkIdentity() {
|
||||
VectorSearchService vectorSearchService = mock(VectorSearchService.class);
|
||||
when(vectorSearchService.searchSimilarDocuments("pool timeout", 3, null)).thenReturn(List.of(
|
||||
result("u1", "{\"_source\":\"a.md\",\"docId\":\"a\",\"chunkIndex\":0,\"title\":\"generic\"}", "generic pool", 0.4f),
|
||||
result("u2", "{\"_source\":\"b.md\",\"docId\":\"b\",\"chunkIndex\":0,\"title\":\"other\"}", "other", 0.5f)
|
||||
));
|
||||
when(vectorSearchService.searchSimilarDocuments("pool timeout", 3, "mysql")).thenReturn(List.of(
|
||||
result("f1", "{\"_source\":\"c.md\",\"docId\":\"c\",\"chunkIndex\":0,\"title\":\"mysql pool timeout\"}", "mysql pool timeout runbook", 0.35f)
|
||||
result("id-1",
|
||||
"{\"_source\":\"c.md\",\"docId\":\"c\",\"chunkIndex\":0,\"title\":\"mysql pool timeout\"}",
|
||||
"mysql pool timeout runbook",
|
||||
0.35f)
|
||||
));
|
||||
|
||||
VectorKnowledgeSearchAdapter adapter = new VectorKnowledgeSearchAdapter(vectorSearchService, new ObjectMapper());
|
||||
ReflectionTestUtils.setField(adapter, "configuredMode", "hybrid");
|
||||
ReflectionTestUtils.setField(adapter, "rrfK", 60);
|
||||
ReflectionTestUtils.setField(adapter, "weightDenseUnfiltered", 1.0);
|
||||
ReflectionTestUtils.setField(adapter, "weightDenseFiltered", 1.0);
|
||||
ReflectionTestUtils.setField(adapter, "weightLexical", 1.0);
|
||||
VectorKnowledgeSearchAdapter adapter =
|
||||
new VectorKnowledgeSearchAdapter(vectorSearchService, new ObjectMapper());
|
||||
|
||||
List<KnowledgeSearchHit> hits = adapter.search(
|
||||
new KnowledgeSearchRequest("pool timeout", 3, "mysql", KnowledgeSearchMode.HYBRID));
|
||||
|
||||
assertEquals(3, hits.size());
|
||||
assertTrue(hits.stream().anyMatch(hit -> "c#chunk-0".equals(hit.evidenceKey())));
|
||||
assertTrue(hits.get(0).metadata().containsKey("fusedScore"));
|
||||
assertEquals(1, hits.size());
|
||||
assertEquals("c#chunk-0", hits.get(0).evidenceKey());
|
||||
assertEquals("c.md", hits.get(0).source());
|
||||
verify(vectorSearchService).searchSimilarDocuments("pool timeout", 3, "mysql");
|
||||
}
|
||||
|
||||
private static VectorSearchService.SearchResult result(String id, String metadata, String content, float score) {
|
||||
|
||||
Reference in New Issue
Block a user