From f035538531271b296b8476822594783476e641ac Mon Sep 17 00:00:00 2001 From: zhuyongxin Date: Mon, 27 Jul 2026 18:49:20 +0800 Subject: [PATCH] feat(rag): dense+BM25 hybrid on MilvusClientV2, drop SDK path Replace legacy MilvusServiceClient knowledge search/write with a single MilvusClientV2 hybrid store (BM25 function + dense ANN + RRFRanker). Use collection biz_hybrid and require knowledge reindex. --- devflow/index.md | 1 + .../acceptance.md | 22 + .../brief.md | 3 + .../decisions.md | 7 + ...vus-hybrid-search-integration-checklist.md | 20 + .../.committed | 2 + .../.openspec.yaml | 2 + .../design.md | 57 +++ .../proposal.md | 34 ++ .../specs/rag-bm25-hybrid/spec.md | 31 ++ .../tasks.md | 10 + openspec/specs/rag-bm25-hybrid/spec.md | 33 ++ .../superbiz/agent/config/MilvusConfig.java | 43 +- .../agent/constant/MilvusConstants.java | 7 +- .../controller/MilvusCheckController.java | 37 +- .../agent/service/VectorIndexService.java | 321 +++---------- .../agent/service/VectorSearchService.java | 312 +------------ .../milvus/MilvusHybridKnowledgeStore.java | 426 ++++++++++++++++++ .../VectorKnowledgeSearchAdapter.java | 127 +----- src/main/resources/application.yml | 18 +- .../service/VectorSearchServiceTest.java | 207 ++------- ...ectorKnowledgeSearchAdapterHybridTest.java | 29 +- 22 files changed, 813 insertions(+), 936 deletions(-) create mode 100644 devflow/projects/2026-07-27-rag-bm25-hybrid-drop-sdk/acceptance.md create mode 100644 devflow/projects/2026-07-27-rag-bm25-hybrid-drop-sdk/brief.md create mode 100644 devflow/projects/2026-07-27-rag-bm25-hybrid-drop-sdk/decisions.md create mode 100644 openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/.committed create mode 100644 openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/.openspec.yaml create mode 100644 openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/design.md create mode 100644 openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/proposal.md create mode 100644 openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/specs/rag-bm25-hybrid/spec.md create mode 100644 openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/tasks.md create mode 100644 openspec/specs/rag-bm25-hybrid/spec.md create mode 100644 src/main/java/com/superbiz/agent/service/milvus/MilvusHybridKnowledgeStore.java diff --git a/devflow/index.md b/devflow/index.md index a8dfdb0..3b41ed3 100644 --- a/devflow/index.md +++ b/devflow/index.md @@ -12,6 +12,7 @@ | 日期 | slug | 说明 | 领域 | 关键词 | 关联 OpenSpec | 状态 | |---|---|---|---|---|---|---| | 2026-07-27 | rag-chunk-evidence-identity-dedup | chunk 级证据身份、去重、retrieve-k/return-n 与 SearchPort 地基，为 hybrid 铺路。 | RAG/证据身份/去重 | evidenceKey, maxChunksPerDocument, retrieve-k, return-n, KnowledgeSearchPort, document_id chunk-scoped | openspec/changes/archive/2026-07-27-rag-chunk-evidence-identity-dedup | archived | +| 2026-07-27 | rag-bm25-hybrid-drop-sdk | 真 dense+BM25 hybrid（MilvusClientV2），废弃知识路径旧 SDK 检索/写入。 | RAG/BM25/hybrid | MilvusClientV2, BM25, hybridSearch, RRFRanker, biz_hybrid, drop SDK path | openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk | archived | | 2026-07-27 | rag-hybrid-search-rrf | Delivery 2：可配置 hybrid 检索与 RRF 多路融合（不绑旧 SDK）。 | RAG/hybrid/RRF | hybrid mode, RRF, KnowledgeSearchPort, filtered+unfiltered fusion, sparse-lite lexical | openspec/changes/archive/2026-07-27-rag-hybrid-search-rrf | archived | | 2026-07-21 | single-react-tool-invocation-store | 建立统一 ToolBoundary 与 Redis canonical invocation store，集中生命周期、证据状态、TTL、容量和 Run 所有权。 | Harness/Tool boundary/Canonical store | ISS-014, ToolBoundary, canonical invocation, PROJECTING, READY, ERROR, TTL, RESULT_TOO_LARGE | openspec/changes/archive/2026-07-21-single-react-tool-invocation-store | archived | | 2026-07-21 | single-react-harness-run-context | 建立显式 RunContext、Harness Core、预算、取消、类型化重试和 Tool Store 基础。 | Harness/Run lifecycle/Budget | ISS-014, RunContext, deadline, cancellation, budget, retry, ToolCallKey | openspec/changes/archive/2026-07-21-single-react-harness-run-context | archived | diff --git a/devflow/projects/2026-07-27-rag-bm25-hybrid-drop-sdk/acceptance.md b/devflow/projects/2026-07-27-rag-bm25-hybrid-drop-sdk/acceptance.md new file mode 100644 index 0000000..c708f20 --- /dev/null +++ b/devflow/projects/2026-07-27-rag-bm25-hybrid-drop-sdk/acceptance.md @@ -0,0 +1,22 @@ +# Acceptance + +## Done + +- `MilvusHybridKnowledgeStore`: schema BM25 function + dense, hybridSearch+RRFRanker, dense search +- `VectorSearchService` only routes dense|hybrid to V2 store +- `VectorIndexService` writes via V2 store +- Removed knowledge-path `MilvusServiceClient` bean wiring +- Config: `milvus.collection=biz_hybrid`, `retrieval.search.mode=hybrid` + +## Tests + +```text +mvn -Dtest=LookupKnowledgeToolTest,KnowledgeEvidencePostProcessorTest,RagResultProjectorTest,RrfFusionTest,VectorKnowledgeSearchAdapterHybridTest,VectorSearchServiceTest,VectorIndexServiceTest test +``` + +EXIT:0 + +## Ops note + +Reindex all knowledge docs into `biz_hybrid` before production hybrid search is meaningful. +Legacy `biz` collection is unused by knowledge path. diff --git a/devflow/projects/2026-07-27-rag-bm25-hybrid-drop-sdk/brief.md b/devflow/projects/2026-07-27-rag-bm25-hybrid-drop-sdk/brief.md new file mode 100644 index 0000000..b14aa74 --- /dev/null +++ b/devflow/projects/2026-07-27-rag-bm25-hybrid-drop-sdk/brief.md @@ -0,0 +1,3 @@ +# Brief: rag-bm25-hybrid-drop-sdk + +True dense+BM25 hybrid on a single MilvusClientV2 backend. Legacy SDK search/write for knowledge path removed. New collection `biz_hybrid` requires reindex. diff --git a/devflow/projects/2026-07-27-rag-bm25-hybrid-drop-sdk/decisions.md b/devflow/projects/2026-07-27-rag-bm25-hybrid-drop-sdk/decisions.md new file mode 100644 index 0000000..f4751c0 --- /dev/null +++ b/devflow/projects/2026-07-27-rag-bm25-hybrid-drop-sdk/decisions.md @@ -0,0 +1,7 @@ +# Decisions + +- Single backend: MilvusClientV2 only for knowledge RAG +- Drop sdk/spring/auto retrieval routing +- New collection biz_hybrid to avoid mutating legacy biz schema in place +- Hybrid = dense ANN + BM25 sparse ANN + RRFRanker +- Dense L2 enrichment for threshold compatibility on hybrid hits diff --git a/docs/milvus-hybrid-search-integration-checklist.md b/docs/milvus-hybrid-search-integration-checklist.md index c1ea4be..0686010 100644 --- a/docs/milvus-hybrid-search-integration-checklist.md +++ b/docs/milvus-hybrid-search-integration-checklist.md @@ -32,6 +32,26 @@ - 分块去重与 hybrid 同规划、分里程碑交付（先共用地基，再开 hybrid） ``` +## 实现状态（2026-07-27） + +| 里程碑 | 状态 | 说明 | +|---|---|---| +| 交付 1 chunk 身份/去重/SearchPort | **已完成并归档** | `2026-07-27-rag-chunk-evidence-identity-dedup` | +| 交付 2a 应用层 multi-path+RRF | **已完成并归档** | `2026-07-27-rag-hybrid-search-rrf`（已被 2b 取代为生产路径） | +| 交付 2b 真 BM25 hybrid + 废弃 SDK | **已完成并归档** | `2026-07-27-rag-bm25-hybrid-drop-sdk` | + +**当前生产知识路径：** + +```text +VectorIndexService / VectorSearchService + -> MilvusHybridKnowledgeStore (MilvusClientV2 only) + collection: milvus.collection (default biz_hybrid) + mode: retrieval.search.mode = dense | hybrid + hybrid: dense ANN + BM25 sparse ANN + RRFRanker +``` + +**运维必做：** 全量重灌知识库到 `biz_hybrid`；旧 `biz` collection 不再被知识路径使用。 + --- ## 1.1 交付拆分：分块去重 + Hybrid 同规划 diff --git a/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/.committed b/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/.committed new file mode 100644 index 0000000..8a7161d --- /dev/null +++ b/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/.committed @@ -0,0 +1,2 @@ +committed: 2026-07-27 +authorized-apply: user-preauthorized diff --git a/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/.openspec.yaml b/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/.openspec.yaml new file mode 100644 index 0000000..8e7013b --- /dev/null +++ b/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-27 diff --git a/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/design.md b/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/design.md new file mode 100644 index 0000000..6125624 --- /dev/null +++ b/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/design.md @@ -0,0 +1,57 @@ +# Design: rag-bm25-hybrid-drop-sdk + +## Decision: single backend + +```text +KnowledgeSearchPort + -> MilvusHybridKnowledgeStore (MilvusClientV2 only) +Write + -> VectorIndexService -> MilvusHybridKnowledgeStore +``` + +No `retrieval.vector-store.mode=sdk|spring|auto` for knowledge lookup. + +## Schema (`milvus.collection`, default `biz_hybrid`) + +| field | type | notes | +|---|---|---| +| id | VarChar PK | chunk id | +| content | VarChar | original chunk body returned to agent | +| search_text | VarChar + analyzer | BM25 input (title/path/content) | +| sparse_vector | SparseFloatVector | BM25 function output | +| vector | FloatVector | dense embedding | +| metadata | JSON | docId, chunkIndex, category, kb_scope, ... | + +Function: `BM25(search_text -> sparse_vector)` + +Indexes: + +- `vector`: IVF_FLAT / L2 (or COSINE if configured later) +- `sparse_vector`: SPARSE_INVERTED_INDEX / BM25 + +## Hybrid query + +```text +AnnSearchReq(vector, FloatVec(queryEmbedding), topK, filter?) +AnnSearchReq(sparse_vector, EmbeddedText(query), topK, filter?) +HybridSearchReq + RRFRanker(k) +``` + +Dense-compatible score for thresholds: if entity lacks dense distance, map fused score conservatively or re-use dense-only probe. Preferred: keep dense distance when available from a parallel dense search metadata; for hybrid hits use inverse-rank placeholder only in metadata and set score from optional dense sub-hit map. + +Implementation approach for threshold stability: + +1. Run hybridSearch for ordering/identity +2. Build map evidenceKey -> dense L2 from a concurrent dense search (same filter/topK) +3. Attach dense score onto fused hits when present; else maxL2 (low similarity) so weak BM25-only hits don't fake PRECISE + +## Migration + +- New collection name avoids mutating legacy `biz` +- Operators re-run knowledge init / document reindex +- Document in acceptance + +## Drop SDK search + +- Delete SDK search methods usage from knowledge path +- `MilvusServiceClient` bean may remain temporarily only if other non-search utilities need it; prefer migrate write/delete fully to V2 and stop creating V1 client if unused diff --git a/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/proposal.md b/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/proposal.md new file mode 100644 index 0000000..94c5dc9 --- /dev/null +++ b/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/proposal.md @@ -0,0 +1,34 @@ +# Change: Real BM25 hybrid search and drop legacy SDK path + +## Why + +Delivery 2 shipped application-layer multi-path + sparse-lite lexical ranking. Product requirement is **true dense + BM25 hybrid retrieval**. Legacy `MilvusServiceClient` search path will be abandoned in this change; keep a single knowledge vector backend based on Milvus Java SDK v2 (`MilvusClientV2`) with BM25 function + `hybridSearch` + RRF. + +## What Changes + +- New hybrid collection schema: dense float vector + analyzed text + sparse BM25 output +- Write path inserts text for BM25 auto-sparsification and dense embedding +- Search path: single backend + - dense mode: dense ANN only + - hybrid mode: dense ANN + BM25 sparse ANN fused by RRFRanker +- Remove/disable legacy SDK `search` / `searchSimilarDocumentsWithSdk` knowledge path and mode routing (`sdk|spring|auto`) +- KnowledgeSearchPort adapter calls the single backend +- Config: collection name, search mode, rrf-k, path weights optional +- Tests for store mapping/fusion wiring; no dependency on live Zilliz in unit tests + +## Non-goals + +- Automatic full production reindex job UI +- Cross-encoder model rerank +- Keeping dual Spring-AI + SDK knowledge search modes + +## Impact + +- **Breaking for existing `biz` collection**: requires new hybrid collection + reindex +- Agent tool contract unchanged +- Interface: L2 internal retrieval backend + +## Depends on + +- Delivery 1 chunk identity +- Delivery 2 SearchPort (will replace sparse-lite hybrid implementation) diff --git a/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/specs/rag-bm25-hybrid/spec.md b/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/specs/rag-bm25-hybrid/spec.md new file mode 100644 index 0000000..4f62034 --- /dev/null +++ b/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/specs/rag-bm25-hybrid/spec.md @@ -0,0 +1,31 @@ +# rag-bm25-hybrid Specification + +## ADDED Requirements + +### Requirement: Knowledge vector backend SHALL be a single Milvus V2 store + +Knowledge indexing and retrieval SHALL use one MilvusClientV2-backed store. Legacy MilvusServiceClient SDK search modes (`sdk`, `auto` fallback to SDK) SHALL NOT be used for `lookup_knowledge`. + +#### Scenario: No SDK search mode + +- **WHEN** knowledge retrieval executes +- **THEN** it SHALL NOT call legacy SDK `search` APIs for candidate generation + +### Requirement: Hybrid mode SHALL fuse dense ANN and BM25 sparse ANN + +When search mode is hybrid, the store SHALL query dense vectors and BM25 sparse vectors and fuse results with RRF (or equivalent ranker) before returning hits. + +#### Scenario: Hybrid uses BM25 text query + +- **WHEN** hybrid search runs with a text query +- **THEN** one search leg SHALL use the BM25/sparse field with the raw query text +- **AND** another leg SHALL use the dense embedding of the query + +### Requirement: Write path SHALL populate BM25 input text and dense vectors + +Document chunk indexing SHALL write original content, BM25 input text, dense embedding, and metadata required for chunk identity. + +#### Scenario: Index writes search_text and vector + +- **WHEN** a chunk is indexed +- **THEN** the store row SHALL include searchable text for BM25 and a dense vector field diff --git a/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/tasks.md b/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/tasks.md new file mode 100644 index 0000000..c2f4675 --- /dev/null +++ b/openspec/changes/archive/2026-07-27-rag-bm25-hybrid-drop-sdk/tasks.md @@ -0,0 +1,10 @@ +# Tasks + +- [x] 1. Add MilvusClientV2 factory + hybrid collection bootstrap +- [x] 2. Implement MilvusHybridKnowledgeStore (insert/delete/dense/hybrid) +- [x] 3. Point VectorIndexService writes at V2 store +- [x] 4. Replace VectorSearchService knowledge path with V2 store (drop SDK/spring mode routing) +- [x] 5. Update VectorKnowledgeSearchAdapter to delegate single backend +- [x] 6. Config + constants for collection/mode +- [x] 7. Unit tests (store routing / adapter / no SDK search) +- [x] 8. Archive + commit diff --git a/openspec/specs/rag-bm25-hybrid/spec.md b/openspec/specs/rag-bm25-hybrid/spec.md new file mode 100644 index 0000000..739ec8c --- /dev/null +++ b/openspec/specs/rag-bm25-hybrid/spec.md @@ -0,0 +1,33 @@ +# rag-bm25-hybrid Specification + +## Purpose +TBD - created by archiving change rag-bm25-hybrid-drop-sdk. Update Purpose after archive. +## Requirements +### Requirement: Knowledge vector backend SHALL be a single Milvus V2 store + +Knowledge indexing and retrieval SHALL use one MilvusClientV2-backed store. Legacy MilvusServiceClient SDK search modes (`sdk`, `auto` fallback to SDK) SHALL NOT be used for `lookup_knowledge`. + +#### Scenario: No SDK search mode + +- **WHEN** knowledge retrieval executes +- **THEN** it SHALL NOT call legacy SDK `search` APIs for candidate generation + +### Requirement: Hybrid mode SHALL fuse dense ANN and BM25 sparse ANN + +When search mode is hybrid, the store SHALL query dense vectors and BM25 sparse vectors and fuse results with RRF (or equivalent ranker) before returning hits. + +#### Scenario: Hybrid uses BM25 text query + +- **WHEN** hybrid search runs with a text query +- **THEN** one search leg SHALL use the BM25/sparse field with the raw query text +- **AND** another leg SHALL use the dense embedding of the query + +### Requirement: Write path SHALL populate BM25 input text and dense vectors + +Document chunk indexing SHALL write original content, BM25 input text, dense embedding, and metadata required for chunk identity. + +#### Scenario: Index writes search_text and vector + +- **WHEN** a chunk is indexed +- **THEN** the store row SHALL include searchable text for BM25 and a dense vector field + diff --git a/src/main/java/com/superbiz/agent/config/MilvusConfig.java b/src/main/java/com/superbiz/agent/config/MilvusConfig.java index e35587c..a196412 100644 --- a/src/main/java/com/superbiz/agent/config/MilvusConfig.java +++ b/src/main/java/com/superbiz/agent/config/MilvusConfig.java @@ -1,51 +1,22 @@ package com.superbiz.agent.config; -import io.milvus.client.MilvusServiceClient; -import com.superbiz.agent.client.MilvusClientFactory; +import com.superbiz.agent.service.milvus.MilvusHybridKnowledgeStore; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; -import jakarta.annotation.PreDestroy; - /** - * Milvus 配置类 - * 负责创建和管理 MilvusServiceClient Bean + * Milvus configuration notes. + * + *

Knowledge RAG uses {@link MilvusHybridKnowledgeStore} (MilvusClientV2) exclusively. + * Legacy {@code MilvusServiceClient} bean is no longer created for the knowledge path.

*/ @Configuration public class MilvusConfig { private static final Logger logger = LoggerFactory.getLogger(MilvusConfig.class); - @Autowired - private MilvusClientFactory milvusClientFactory; - - private MilvusServiceClient milvusClient; - - /** - * 创建 MilvusServiceClient Bean - * - * @return MilvusServiceClient 实例 - */ - @Bean - public MilvusServiceClient milvusServiceClient() { - logger.info("正在初始化 Milvus 客户端..."); - milvusClient = milvusClientFactory.createClient(); - logger.info("Milvus 客户端初始化完成"); - return milvusClient; - } - - /** - * 应用关闭时清理资源 - */ - @PreDestroy - public void cleanup() { - if (milvusClient != null) { - logger.info("正在关闭 Milvus 客户端连接..."); - milvusClient.close(); - logger.info("Milvus 客户端连接已关闭"); - } + public MilvusConfig() { + logger.info("Milvus knowledge path: MilvusClientV2 hybrid store only (legacy SDK search disabled)"); } } diff --git a/src/main/java/com/superbiz/agent/constant/MilvusConstants.java b/src/main/java/com/superbiz/agent/constant/MilvusConstants.java index 9c4475f..a274833 100644 --- a/src/main/java/com/superbiz/agent/constant/MilvusConstants.java +++ b/src/main/java/com/superbiz/agent/constant/MilvusConstants.java @@ -8,9 +8,14 @@ public class MilvusConstants { public static final String MILVUS_DB_NAME = "default"; /** - * Milvus 集合名称 + * Legacy collection name (pre hybrid). Knowledge path uses milvus.collection (default biz_hybrid). */ public static final String MILVUS_COLLECTION_NAME = "biz"; + + /** + * Default hybrid dense+BM25 collection. + */ + public static final String MILVUS_HYBRID_COLLECTION_NAME = "biz_hybrid"; /** * 向量维度（豆包 embedding 模型的维度） diff --git a/src/main/java/com/superbiz/agent/controller/MilvusCheckController.java b/src/main/java/com/superbiz/agent/controller/MilvusCheckController.java index 8771db4..458b604 100644 --- a/src/main/java/com/superbiz/agent/controller/MilvusCheckController.java +++ b/src/main/java/com/superbiz/agent/controller/MilvusCheckController.java @@ -1,47 +1,36 @@ package com.superbiz.agent.controller; -import io.milvus.client.MilvusServiceClient; -import io.milvus.grpc.ShowCollectionsResponse; -import io.milvus.param.R; -import io.milvus.param.collection.ShowCollectionsParam; +import com.superbiz.agent.service.milvus.MilvusHybridKnowledgeStore; +import io.milvus.v2.service.collection.response.ListCollectionsResp; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.http.ResponseEntity; -import org.springframework.web.bind.annotation.*; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RestController; import java.util.HashMap; import java.util.Map; /** - * Milvus 测试控制器 - * 用于测试数据库连接和数据读取 + * Milvus health check using the single V2 knowledge backend. */ @RestController @RequestMapping("/milvus") public class MilvusCheckController { @Autowired - private MilvusServiceClient milvusClient; + private MilvusHybridKnowledgeStore knowledgeStore; - /** - * 简单的健康检查 - */ @GetMapping("/health") public ResponseEntity> simpleHealth() { Map result = new HashMap<>(); - try { - R response = milvusClient.showCollections( - ShowCollectionsParam.newBuilder().build() - ); - - if (response.getStatus() == 0) { - result.put("message", "ok"); - result.put("collections", response.getData().getCollectionNamesList()); - return ResponseEntity.ok(result); - } else { - result.put("message", response.getMessage()); - return ResponseEntity.status(503).body(result); - } + ListCollectionsResp response = knowledgeStore.client().listCollections(); + result.put("message", "ok"); + result.put("backend", "milvus-client-v2"); + result.put("knowledgeCollection", knowledgeStore.collectionName()); + result.put("collections", response == null ? null : response.getCollectionNames()); + return ResponseEntity.ok(result); } catch (Exception e) { result.put("error", e.getMessage()); return ResponseEntity.status(503).body(result); diff --git a/src/main/java/com/superbiz/agent/service/VectorIndexService.java b/src/main/java/com/superbiz/agent/service/VectorIndexService.java index 11cc1e7..a4e9ce3 100644 --- a/src/main/java/com/superbiz/agent/service/VectorIndexService.java +++ b/src/main/java/com/superbiz/agent/service/VectorIndexService.java @@ -1,17 +1,10 @@ package com.superbiz.agent.service; -import io.milvus.client.MilvusServiceClient; -import io.milvus.grpc.MutationResult; -import io.milvus.param.R; -import io.milvus.param.RpcStatus; -import io.milvus.param.collection.LoadCollectionParam; -import io.milvus.param.dml.DeleteParam; -import io.milvus.param.dml.InsertParam; -import lombok.Getter; -import lombok.Setter; -import com.superbiz.agent.constant.MilvusConstants; import com.superbiz.agent.dto.DocumentChunk; import com.superbiz.agent.dto.Frontmatter; +import com.superbiz.agent.service.milvus.MilvusHybridKnowledgeStore; +import lombok.Getter; +import lombok.Setter; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Autowired; @@ -23,20 +16,15 @@ import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.time.LocalDateTime; -import java.util.*; +import java.util.HashMap; +import java.util.List; +import java.util.Map; /** * 向量索引写入服务（RAG 入库侧）。 * - *

负责：读文件/文档块 -> 切片 -> embedding -> 写入 Milvus。 - * 检索读取走 {@link VectorSearchService}，双方通过 collection + metadata 约定衔接。

- * - *

metadata 关键字段

- * docId / _source / chunkIndex / totalChunks / title / breadcrumb / category / kb_scope - * - *

embedding 文本

- * 见 {@link #buildEmbeddingText(DocumentChunk)}：会把 title、breadcrumb 拼进向量文本， - * 而入库 content 字段仍保存原始 chunk 正文（检索返回的是 content，不是 embedding 拼接串）。 + *

写入单一后端 {@link MilvusHybridKnowledgeStore}（dense + BM25 search_text）。 + * 不再使用 legacy {@code MilvusServiceClient} insert/delete。

*/ @Service public class VectorIndexService { @@ -44,7 +32,7 @@ public class VectorIndexService { private static final Logger logger = LoggerFactory.getLogger(VectorIndexService.class); @Autowired - private MilvusServiceClient milvusClient; + private MilvusHybridKnowledgeStore knowledgeStore; @Autowired private VectorEmbeddingService embeddingService; @@ -55,33 +43,25 @@ public class VectorIndexService { @Value("${file.upload.path}") private String uploadPath; - /** - * 索引指定目录下的所有文件 - * - * @param directoryPath 目录路径（可选，默认使用配置的上传目录） - * @return 索引结果 这里可以优化：定时重建目录下所有文件的索引 - */ public IndexingResult indexDirectory(String directoryPath) { IndexingResult result = new IndexingResult(); result.setStartTime(LocalDateTime.now()); try { - // 使用指定目录或默认上传目录 - String targetPath = (directoryPath != null && !directoryPath.trim().isEmpty()) + String targetPath = (directoryPath != null && !directoryPath.trim().isEmpty()) ? directoryPath : uploadPath; - + Path dirPath = Paths.get(targetPath).normalize(); File directory = dirPath.toFile(); - + if (!directory.exists() || !directory.isDirectory()) { throw new IllegalArgumentException("目录不存在或不是有效目录: " + targetPath); } result.setDirectoryPath(directory.getAbsolutePath()); - // 获取所有支持的文件 - File[] files = directory.listFiles((dir, name) -> - name.endsWith(".txt") || name.endsWith(".md") + File[] files = directory.listFiles((dir, name) -> + name.endsWith(".txt") || name.endsWith(".md") ); if (files == null || files.length == 0) { @@ -95,25 +75,20 @@ public class VectorIndexService { result.setTotalFiles(files.length); logger.info("开始索引目录: {}, 找到 {} 个文件", targetPath, files.length); - // 遍历并索引每个文件 for (File file : files) { try { indexSingleFile(file.getAbsolutePath()); result.incrementSuccessCount(); - logger.info("✓ 文件索引成功: {}", file.getName()); + logger.info("文件索引成功: {}", file.getName()); } catch (Exception e) { result.incrementFailCount(); result.addFailedFile(file.getAbsolutePath(), e.getMessage()); - logger.error("✗ 文件索引失败: {}", file.getName(), e); + logger.error("文件索引失败: {}", file.getName(), e); } } result.setSuccess(result.getFailCount() == 0); result.setEndTime(LocalDateTime.now()); - - logger.info("目录索引完成: 总数={}, 成功={}, 失败={}", - result.getTotalFiles(), result.getSuccessCount(), result.getFailCount()); - return result; } catch (Exception e) { @@ -125,148 +100,85 @@ public class VectorIndexService { } } - /** - * 索引单个文件 - * - * @param filePath 文件路径 - * @throws Exception 索引失败时抛出异常 - */ public void indexSingleFile(String filePath) throws Exception { Path path = Paths.get(filePath).normalize(); File file = path.toFile(); - + if (!file.exists() || !file.isFile()) { throw new IllegalArgumentException("文件不存在: " + filePath); } logger.info("开始索引文件: {}", path); - - // 1. 读取文件内容 String content = Files.readString(path); - logger.info("读取文件: {}, 内容长度: {} 字符", path, content.length()); - - // 2. 删除该文件的旧数据（如果存在） deleteExistingData(path.toString()); - // 3. 文档分片 List chunks = chunkService.chunkDocument(content, path.toString()); logger.info("文档分片完成: {} -> {} 个分片", filePath, chunks.size()); - // 4. 为每个分片生成向量并插入 Milvus for (int i = 0; i < chunks.size(); i++) { DocumentChunk chunk = chunks.get(i); - try { - // 生成向量 List vector = embeddingService.generateEmbedding(buildEmbeddingText(chunk)); - - // 构建元数据（包含文件信息） Map metadata = buildMetadata(path.toString(), chunk, chunks.size()); - - // 插入到 Milvus - insertToMilvus(chunk.getContent(), vector, metadata, chunk.getChunkIndex()); - - logger.info("✓ 分片 {}/{} 索引成功", i + 1, chunks.size()); - + knowledgeStore.upsertChunk( + chunk.getContent(), + buildSearchText(chunk), + vector, + metadata, + chunk.getChunkIndex()); + logger.info("分片 {}/{} 索引成功", i + 1, chunks.size()); } catch (Exception e) { - logger.error("✗ 分片 {}/{} 索引失败", i + 1, chunks.size(), e); + logger.error("分片 {}/{} 索引失败", i + 1, chunks.size(), e); throw new RuntimeException("分片索引失败: " + e.getMessage(), e); } } - logger.info("文件索引完成: {}, 共 {} 个分片", filePath, chunks.size()); } - /** - * 索引文档分块（用于上传文档的向量化） - * - * @param docId 文档ID - * @param chunks 文档分块列表 - * @param category 文档类别（api、domain、troubleshoot 等） - * @throws Exception 索引失败时抛出异常 - */ public void indexDocumentChunks(String docId, List chunks, String category) throws Exception { indexDocumentChunks(docId, chunks, category, null); } - public void indexDocumentChunks(String docId, List chunks, String category, Frontmatter frontmatter) throws Exception { + public void indexDocumentChunks(String docId, + List chunks, + String category, + Frontmatter frontmatter) throws Exception { if (chunks == null || chunks.isEmpty()) { throw new IllegalArgumentException("文档分块列表为空"); } logger.info("开始索引文档分块，docId: {}, 分块数: {}, 类别: {}", docId, chunks.size(), category); - - // 1. 删除该文档的旧数据（如果存在） deleteDocumentChunks(docId); - // 2. 为每个分块生成向量并插入 Milvus for (int i = 0; i < chunks.size(); i++) { DocumentChunk chunk = chunks.get(i); - try { - // 生成向量 List vector = embeddingService.generateEmbedding(buildEmbeddingText(chunk)); - - // 构建元数据（使用 docId 和 category） Map metadata = buildDocumentMetadata(docId, chunk, chunks.size(), category, frontmatter); - - // 插入到 Milvus - insertToMilvus(chunk.getContent(), vector, metadata, chunk.getChunkIndex()); - - logger.info("✓ 文档分块 {}/{} 索引成功，docId: {}", i + 1, chunks.size(), docId); - + knowledgeStore.upsertChunk( + chunk.getContent(), + buildSearchText(chunk), + vector, + metadata, + chunk.getChunkIndex()); + logger.info("文档分块 {}/{} 索引成功，docId: {}", i + 1, chunks.size(), docId); } catch (Exception e) { - logger.error("✗ 文档分块 {}/{} 索引失败，docId: {}", i + 1, chunks.size(), docId, e); + logger.error("文档分块 {}/{} 索引失败，docId: {}", i + 1, chunks.size(), docId, e); throw new RuntimeException("文档分块索引失败: " + e.getMessage(), e); } } - logger.info("文档索引完成，docId: {}, 共 {} 个分块，类别: {}", docId, chunks.size(), category); } - /** - * 删除文档的所有分块（根据 docId） - */ public void deleteDocumentChunks(String docId) { try { - // 构建删除表达式：metadata["docId"] == "xxx" - String expr = String.format("metadata[\"docId\"] == \"%s\"", docId); - - logger.info("准备删除文档旧数据，docId: {}, 表达式: {}", docId, expr); - - // 确保 collection 已加载 - R loadResponse = milvusClient.loadCollection( - LoadCollectionParam.newBuilder() - .withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME) - .build() - ); - - if (loadResponse.getStatus() != 0 && loadResponse.getStatus() != 65535) { - logger.warn("加载 collection 失败: {}", loadResponse.getMessage()); - return; - } - - DeleteParam deleteParam = DeleteParam.newBuilder() - .withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME) - .withExpr(expr) - .build(); - - R deleteResponse = milvusClient.delete(deleteParam); - - if (deleteResponse.getStatus() == 0) { - logger.info("删除文档旧数据成功，docId: {}", docId); - } else { - logger.warn("删除文档旧数据失败，docId: {}, 原因: {}", docId, deleteResponse.getMessage()); - } - + knowledgeStore.deleteByDocId(docId); + logger.info("删除文档旧数据成功，docId: {}", docId); } catch (Exception e) { logger.warn("删除文档旧数据异常，docId: {}", docId, e); } } - /** - * 构建文档元数据（用于上传文档） - */ static Map buildDocumentMetadata(String docId, DocumentChunk chunk, int totalChunks, String category) { return buildDocumentMetadata(docId, chunk, totalChunks, category, null); } @@ -277,47 +189,37 @@ public class VectorIndexService { String category, Frontmatter frontmatter) { Map metadata = new HashMap<>(); - - // 文档标识 String source = firstNonBlank(frontmatter != null ? frontmatter.getSource() : null, "upload:" + docId); metadata.put("docId", docId); - metadata.put("_source", source); // 区分文件索引和上传文档 + metadata.put("_source", source); metadata.put("source", source); - - // 分片信息 metadata.put("chunkIndex", chunk.getChunkIndex()); metadata.put("totalChunks", totalChunks); - // 标题信息 String title = firstNonBlank(chunk.getTitle(), frontmatter != null ? frontmatter.getTitle() : null); if (title != null) { metadata.put("title", title); } - - // 面包屑导航（完整标题层级路径） String breadcrumb = firstNonBlank(frontmatter != null ? frontmatter.getBreadcrumb() : null, chunk.getBreadcrumb()); if (breadcrumb != null) { metadata.put("breadcrumb", breadcrumb); } - - // 文档类别 metadata.put("category", category != null && !category.isBlank() ? category : "upload"); - String kbScope = trimToNull(frontmatter != null ? frontmatter.getKbScope() : null); if (kbScope != null) { metadata.put("kb_scope", kbScope); } - return metadata; } - /** - * 构造送入 embedding 模型的文本。 - * - *

把标题链路注入向量语义，缓解“正文片段缺上下文”导致的召回漂移。 - * 注意：这里只影响向量，不影响 Milvus content 字段存储的原文。

- */ static String buildEmbeddingText(DocumentChunk chunk) { + return buildSearchText(chunk); + } + + /** + * Text used for BM25 {@code search_text} and dense embedding. + */ + static String buildSearchText(DocumentChunk chunk) { String content = trimToEmpty(chunk.getContent()); String title = trimToEmpty(chunk.getTitle()); String breadcrumb = trimToEmpty(chunk.getBreadcrumb()); @@ -358,64 +260,22 @@ public class VectorIndexService { return null; } - /** - * 删除文件的旧数据（根据 metadata._source） - */ private void deleteExistingData(String filePath) { try { - // 使用统一的路径分隔符（正斜杠）用于Milvus存储，避免表达式解析错误 - // 将系统路径转换为统一格式 Path path = Paths.get(filePath).normalize(); String normalizedPath = path.toString().replace(File.separator, "/"); - - // 构建删除表达式：metadata["_source"] == "xxx" - String expr = String.format("metadata[\"_source\"] == \"%s\"", normalizedPath); - - logger.info("准备删除旧数据，路径: {}, 表达式: {}", normalizedPath, expr); - - // 确保 collection 已加载（删除操作需要集合已加载） - R loadResponse = milvusClient.loadCollection( - LoadCollectionParam.newBuilder() - .withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME) - .build() - ); - - // 状态码 65535 表示集合已经加载，这不是错误 - if (loadResponse.getStatus() != 0 && loadResponse.getStatus() != 65535) { - logger.warn("加载 collection 失败: {}", loadResponse.getMessage()); - return; - } - - DeleteParam deleteParam = DeleteParam.newBuilder() - .withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME) - .withExpr(expr) - .build(); - - R response = milvusClient.delete(deleteParam); - - if (response.getStatus() != 0) { - logger.warn("删除旧数据时出现警告: {}", response.getMessage()); - } else { - long deletedCount = response.getData().getDeleteCnt(); - logger.info("✓ 已删除文件的旧数据: {}, 删除记录数: {}", normalizedPath, deletedCount); - } - + knowledgeStore.deleteBySource(normalizedPath); + logger.info("已删除文件的旧数据: {}", normalizedPath); } catch (Exception e) { logger.warn("删除旧数据失败（可能是首次索引）: {}", e.getMessage()); } } - /** - * 构建元数据（包含文件信息） - */ private Map buildMetadata(String filePath, DocumentChunk chunk, int totalChunks) { Map metadata = new HashMap<>(); - - // 标准化路径：使用统一的路径分隔符（正斜杠）用于存储，确保跨平台一致性 Path path = Paths.get(filePath).normalize(); String normalizedPath = path.toString().replace(File.separator, "/"); - - // 文件信息 + Path fileName = path.getFileName(); String fileNameStr = fileName != null ? fileName.toString() : ""; String extension = ""; @@ -425,42 +285,28 @@ public class VectorIndexService { } metadata.put("_source", normalizedPath); + metadata.put("source", normalizedPath); metadata.put("_extension", extension); metadata.put("_file_name", fileNameStr); - // 提取类别（从文件路径中提取目录名） String category = extractCategory(normalizedPath); if (category != null && !category.isEmpty()) { metadata.put("category", category); } - - // 分片信息 metadata.put("chunkIndex", chunk.getChunkIndex()); metadata.put("totalChunks", totalChunks); - - // 标题信息 if (chunk.getTitle() != null && !chunk.getTitle().isEmpty()) { metadata.put("title", chunk.getTitle()); } - - // 面包屑导航（完整标题层级路径） if (chunk.getBreadcrumb() != null && !chunk.getBreadcrumb().isEmpty()) { metadata.put("breadcrumb", chunk.getBreadcrumb()); } - return metadata; } - /** - * 从文件路径中提取类别 - * 例如：aiops-docs/api/redis-api.md → "api" - */ private String extractCategory(String filePath) { try { - // 标准化路径分隔符 String normalized = filePath.replace("\\", "/"); - - // 查找 aiops-docs/ 后的第一级目录 int docsIndex = normalized.indexOf("aiops-docs/"); if (docsIndex >= 0) { String afterDocs = normalized.substring(docsIndex + "aiops-docs/".length()); @@ -469,13 +315,10 @@ public class VectorIndexService { return afterDocs.substring(0, slashIndex); } } - - // 如果没有 aiops-docs，返回第一级目录 int firstSlash = normalized.indexOf("/"); if (firstSlash > 0) { return normalized.substring(0, firstSlash); } - return null; } catch (Exception e) { logger.warn("提取类别失败，路径: {}", filePath, e); @@ -483,68 +326,6 @@ public class VectorIndexService { } } - /** - * 插入向量到 Milvus - */ - private void insertToMilvus(String content, List vector, - Map metadata, int chunkIndex) throws Exception { - try { - // 确保 collection 已加载 - R loadResponse = milvusClient.loadCollection( - LoadCollectionParam.newBuilder() - .withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME) - .build() - ); - - if (loadResponse.getStatus() != 0 && loadResponse.getStatus() != 65535) { - throw new RuntimeException("加载 collection 失败: " + loadResponse.getMessage()); - } - - // 生成唯一 ID（使用 _source + 分片索引） - String source = (String) metadata.get("_source"); - String id = UUID.nameUUIDFromBytes((source + "_" + chunkIndex).getBytes()).toString(); - - // 构建字段数据 - List fields = new ArrayList<>(); - - // ID 字段 - fields.add(new InsertParam.Field("id", Collections.singletonList(id))); - - // content 字段 - fields.add(new InsertParam.Field("content", Collections.singletonList(content))); - - // vector 字段 - fields.add(new InsertParam.Field("vector", Collections.singletonList(vector))); - - // metadata 字段（JSON 对象） - com.google.gson.Gson gson = new com.google.gson.Gson(); - com.google.gson.JsonObject metadataJson = gson.toJsonTree(metadata).getAsJsonObject(); - fields.add(new InsertParam.Field("metadata", Collections.singletonList(metadataJson))); - - // 构建插入参数 - InsertParam insertParam = InsertParam.newBuilder() - .withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME) - .withFields(fields) - .build(); - - // 执行插入 - R insertResponse = milvusClient.insert(insertParam); - - if (insertResponse.getStatus() != 0) { - throw new RuntimeException("插入向量失败: " + insertResponse.getMessage()); - } - - logger.debug("向量插入成功: id={}, source={}, chunk={}", id, source, chunkIndex); - - } catch (Exception e) { - logger.error("插入向量到 Milvus 失败", e); - throw e; - } - } - - /** - * 索引结果类 - */ @Getter public static class IndexingResult { @Setter diff --git a/src/main/java/com/superbiz/agent/service/VectorSearchService.java b/src/main/java/com/superbiz/agent/service/VectorSearchService.java index c717993..3aa7105 100644 --- a/src/main/java/com/superbiz/agent/service/VectorSearchService.java +++ b/src/main/java/com/superbiz/agent/service/VectorSearchService.java @@ -1,53 +1,23 @@ package com.superbiz.agent.service; -import com.fasterxml.jackson.core.JsonProcessingException; -import com.fasterxml.jackson.databind.ObjectMapper; -import com.superbiz.agent.constant.MilvusConstants; -import io.milvus.client.MilvusServiceClient; -import io.milvus.grpc.SearchResults; -import io.milvus.param.R; -import io.milvus.param.dml.SearchParam; -import io.milvus.response.SearchResultsWrapper; +import com.superbiz.agent.service.milvus.MilvusHybridKnowledgeStore; import lombok.Getter; import lombok.Setter; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import org.springframework.ai.document.Document; -import org.springframework.ai.vectorstore.SearchRequest; -import org.springframework.ai.vectorstore.VectorStore; -import org.springframework.beans.factory.ObjectProvider; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Service; -import java.util.ArrayList; -import java.util.Collections; import java.util.List; -import java.util.Map; +import java.util.Locale; /** - * L1 向量检索门面。 + * Knowledge vector retrieval facade. * - *

对上层（{@link KnowledgeDocumentRetriever}）只暴露稳定 API： - * {@link #searchSimilarDocuments(String, int, String)}。底层实现可切换：

- *
    - *
  • {@code sdk}：Milvus SDK 直连
    • - *
  • {@code spring} / {@code spring-ai}：Spring AI VectorStore
    • - *
  • {@code auto}（默认）：先 Spring AI，失败再 fallback 到 SDK
    • - *
- * - *

分数兼容约定

- * 后处理 {@code KnowledgeEvidencePostProcessor} 按“L2 距离越小越相似”归一化。 - * 因此本类统一把 {@link SearchResult#score} 填成兼容 L2 距离： - *
    - *
  • SDK 路径：直接用 Milvus L2 score
    • - *
  • Spring AI 路径：优先 metadata.distance；否则把 similarity 映射为 - * {@code (1 - similarity) * maxL2Distance}
    • - *
- * - *

过滤

- * 可选 category + 全局 {@code retrieval.kb-scope}。两条实现路径的 filter 语法不同， - * 但语义一致：只在对应 metadata 字段上收窄。 + *

Single backend: {@link MilvusHybridKnowledgeStore} (Milvus Java SDK v2). + * Legacy {@code MilvusServiceClient} search and Spring AI VectorStore routing for + * {@code lookup_knowledge} have been removed.

*/ @Service public class VectorSearchService { @@ -55,276 +25,42 @@ public class VectorSearchService { private static final Logger logger = LoggerFactory.getLogger(VectorSearchService.class); @Autowired - private MilvusServiceClient milvusClient; + private MilvusHybridKnowledgeStore knowledgeStore; @Autowired private VectorEmbeddingService embeddingService; - @Autowired - private ObjectProvider vectorStoreProvider; - - @Autowired - private ObjectMapper objectMapper; - - /** 检索实现路由：auto / spring / spring-ai / sdk。 */ - @Value("${retrieval.vector-store.mode:auto}") - private String vectorStoreMode = "auto"; - - /** similarity -> 兼容 L2 时使用的距离上限，需与后处理归一化配置一致。 */ - @Value("${retrieval.normalization.max-l2-distance:2.0}") - private double maxL2Distance = 2.0; - - /** 非空时只检索该 kb_scope 下的 chunk（多租户/多知识库隔离）。 */ - @Value("${retrieval.kb-scope:}") - private String kbScope = ""; + /** + * dense | hybrid + */ + @Value("${retrieval.search.mode:dense}") + private String searchMode = "dense"; public List searchSimilarDocuments(String query, int topK) { return searchSimilarDocuments(query, topK, null); } - /** - * 按 query 召回 topK 相似文档片段。 - * - * @param query 检索文本（会再 embedding） - * @param topK 返回条数 - * @param category 可选 category 过滤；null/blank 表示不过滤 - */ public List searchSimilarDocuments(String query, int topK, String category) { - String mode = vectorStoreMode == null ? "auto" : vectorStoreMode.trim().toLowerCase(); - return switch (mode) { - case "sdk" -> searchSimilarDocumentsWithSdk(query, topK, category); - case "spring", "spring-ai" -> searchSimilarDocumentsWithVectorStore(query, topK, category); - case "auto" -> searchWithAutoFallback(query, topK, category); - default -> { - logger.warn("Unknown retrieval.vector-store.mode={}, using auto mode", vectorStoreMode); - yield searchWithAutoFallback(query, topK, category); - } - }; + String mode = searchMode == null ? "dense" : searchMode.trim().toLowerCase(Locale.ROOT); + List queryVector = embeddingService.generateQueryVector(query); + if ("hybrid".equals(mode)) { + logger.info("Hybrid dense+BM25 search topK={} category={} collection={}", + topK, category, knowledgeStore.collectionName()); + return knowledgeStore.searchHybrid(query, queryVector, topK, category); + } + logger.info("Dense search topK={} category={} collection={}", + topK, category, knowledgeStore.collectionName()); + return knowledgeStore.searchDense(query, queryVector, topK, category); } - /** auto：Spring AI 优先，任意异常则降级 SDK（保证检索可用性）。 */ - private List searchWithAutoFallback(String query, int topK, String category) { - try { - return searchSimilarDocumentsWithVectorStore(query, topK, category); - } catch (Exception e) { - logger.warn("Spring AI VectorStore retrieval failed, falling back to Milvus SDK: errorType={}", - e.getClass().getSimpleName()); - return searchSimilarDocumentsWithSdk(query, topK, category); - } - } - - /** - * Spring AI VectorStore 路径。 - * 注意：写入索引仍主要由 Milvus SDK 完成；这里是读适配，依赖双方 metadata schema 一致。 - */ - List searchSimilarDocumentsWithVectorStore(String query, int topK, String category) { - VectorStore vectorStore = vectorStoreProvider != null ? vectorStoreProvider.getIfAvailable() : null; - if (vectorStore == null) { - throw new IllegalStateException("Spring AI VectorStore bean is unavailable"); - } - - logger.info("Starting Spring AI VectorStore search: topK={}, category={}, kbScope={}", - topK, category, effectiveKbScope()); - SearchRequest.Builder builder = SearchRequest.builder() - .query(query) - .topK(topK) - // 不做框架层阈值截断，相关性判断交给后处理 - .similarityThresholdAll(); - String filterExpression = buildSpringAiFilterExpression(category); - if (filterExpression != null) { - builder.filterExpression(filterExpression); - logger.info("Spring AI VectorStore metadata filter: {}", filterExpression); - } - - List documents = vectorStore.similaritySearch(builder.build()); - List results = new ArrayList<>(); - for (Document document : documents) { - SearchResult result = new SearchResult(); - result.setId(document.getId()); - result.setContent(document.getText()); - result.setMetadata(toJson(document.getMetadata())); - result.setRawScore(document.getScore()); - result.setScoreLabel("similarity"); - // 下游统一按 L2 距离消费，这里做兼容映射 - result.setScore(toCompatibleL2Distance(document)); - results.add(result); - } - logger.info("Spring AI VectorStore search complete, candidates={}", results.size()); - return results; - } - - /** Milvus SDK 原生 L2 检索路径。 */ - List searchSimilarDocumentsWithSdk(String query, int topK, String category) { - try { - logger.info("Starting Milvus SDK search: topK={}, category={}, kbScope={}", - topK, category, effectiveKbScope()); - - List queryVector = embeddingService.generateQueryVector(query); - logger.debug("Query vector generated, dimension={}", queryVector.size()); - - SearchParam.Builder searchParamBuilder = SearchParam.newBuilder() - .withCollectionName(MilvusConstants.MILVUS_COLLECTION_NAME) - .withVectorFieldName("vector") - .withVectors(Collections.singletonList(queryVector)) - .withTopK(topK) - .withMetricType(io.milvus.param.MetricType.L2) - .withOutFields(List.of("id", "content", "metadata")) - .withParams("{\"nprobe\":10}"); - - String expr = buildSdkFilterExpression(category); - if (expr != null) { - searchParamBuilder.withExpr(expr); - logger.info("Milvus SDK metadata filter: {}", expr); - } - - R searchResponse = milvusClient.search(searchParamBuilder.build()); - if (searchResponse.getStatus() != 0) { - throw new RuntimeException("Vector search failed: " + searchResponse.getMessage()); - } - - SearchResultsWrapper wrapper = new SearchResultsWrapper(searchResponse.getData().getResults()); - List results = new ArrayList<>(); - for (int i = 0; i < wrapper.getRowRecords(0).size(); i++) { - SearchResult result = new SearchResult(); - result.setId((String) wrapper.getIDScore(0).get(i).get("id")); - result.setContent((String) wrapper.getFieldData("content", 0).get(i)); - // Milvus L2：数值越小越相似 - result.setScore(wrapper.getIDScore(0).get(i).getScore()); - result.setRawScore((double) result.getScore()); - result.setScoreLabel("l2_distance"); - - Object metadataObj = wrapper.getFieldData("metadata", 0).get(i); - if (metadataObj != null) { - result.setMetadata(metadataObj.toString()); - } - - results.add(result); - } - - logger.info("Milvus SDK search complete, candidates={}", results.size()); - return results; - } catch (Exception e) { - logger.error("Milvus SDK vector search failed: errorType={}", e.getClass().getSimpleName()); - throw new RuntimeException("Vector search failed: " + e.getMessage(), e); - } - } - - /** 优先使用 metadata.distance；否则把 similarity 映射为兼容 L2。 */ - private float toCompatibleL2Distance(Document document) { - Double distance = extractDistance(document.getMetadata()); - if (distance != null) { - return distance.floatValue(); - } - return toCompatibleL2Distance(document.getScore()); - } - - /** - * similarity ∈ [0,1] 越大越相似 -> 兼容 L2 距离。 - * 映射：distance = (1 - similarity) * maxL2Distance - */ - private float toCompatibleL2Distance(Double similarity) { - if (similarity == null) { - return (float) maxL2Distance; - } - double bounded = Math.max(0.0, Math.min(1.0, similarity)); - return (float) ((1.0 - bounded) * maxL2Distance); - } - - private Double extractDistance(Map metadata) { - if (metadata == null) { - return null; - } - Object value = metadata.get("distance"); - if (value instanceof Number number) { - return number.doubleValue(); - } - if (value instanceof String text) { - try { - return Double.parseDouble(text); - } catch (NumberFormatException ignored) { - return null; - } - } - return null; - } - - private String toJson(Map metadata) { - if (metadata == null || metadata.isEmpty()) { - return null; - } - try { - return objectMapper.writeValueAsString(metadata); - } catch (JsonProcessingException e) { - return metadata.toString(); - } - } - - private String escapeFilterValue(String value) { - return value.replace("'", "\\'"); - } - - /** Spring AI filter DSL，例如：category == 'mysql' && kb_scope == 'prod' */ - String buildSpringAiFilterExpression(String category) { - List parts = new ArrayList<>(); - String categoryFilter = trimToNull(category); - if (categoryFilter != null) { - parts.add("category == '" + escapeFilterValue(categoryFilter) + "'"); - } - String scopeFilter = effectiveKbScope(); - if (scopeFilter != null) { - parts.add("kb_scope == '" + escapeFilterValue(scopeFilter) + "'"); - } - return parts.isEmpty() ? null : String.join(" && ", parts); - } - - /** Milvus boolean expr，字段在 JSON metadata 内。 */ - String buildSdkFilterExpression(String category) { - List parts = new ArrayList<>(); - String categoryFilter = trimToNull(category); - if (categoryFilter != null) { - parts.add("metadata[\"category\"] == \"" + escapeMilvusString(categoryFilter) + "\""); - } - String scopeFilter = effectiveKbScope(); - if (scopeFilter != null) { - parts.add("metadata[\"kb_scope\"] == \"" + escapeMilvusString(scopeFilter) + "\""); - } - return parts.isEmpty() ? null : String.join(" && ", parts); - } - - private String effectiveKbScope() { - return trimToNull(kbScope); - } - - private String trimToNull(String value) { - if (value == null || value.isBlank()) { - return null; - } - return value.trim(); - } - - private String escapeMilvusString(String value) { - return value.replace("\\", "\\\\").replace("\"", "\\\""); - } - - /** - * 统一检索命中结构。 - * - *
    - *
  • {@code score}：兼容 L2 距离，供后处理 normalizeL2
    • - *
  • {@code rawScore}：底层原始分（similarity 或 l2）
    • - *
  • {@code scoreLabel}：解释 rawScore 语义
    • - *
  • {@code metadata}：JSON 字符串，含 docId/chunkIndex/title 等
    • - *
- */ @Setter @Getter public static class SearchResult { private String id; private String content; /** - * Compatibility score used by existing lookup relevance normalization. - * SDK mode keeps L2 distance; VectorStore mode prefers the Milvus - * distance metadata and falls back to similarity mapping. + * Compatibility score for post-process normalizeL2. + * Dense path: L2 distance. Hybrid path: dense L2 when available. */ private float score; private Double rawScore; diff --git a/src/main/java/com/superbiz/agent/service/milvus/MilvusHybridKnowledgeStore.java b/src/main/java/com/superbiz/agent/service/milvus/MilvusHybridKnowledgeStore.java new file mode 100644 index 0000000..1622157 --- /dev/null +++ b/src/main/java/com/superbiz/agent/service/milvus/MilvusHybridKnowledgeStore.java @@ -0,0 +1,426 @@ +package com.superbiz.agent.service.milvus; + +import com.google.gson.Gson; +import com.google.gson.JsonObject; +import com.superbiz.agent.config.MilvusProperties; +import com.superbiz.agent.constant.MilvusConstants; +import com.superbiz.agent.service.VectorSearchService; +import io.milvus.common.clientenum.FunctionType; +import io.milvus.v2.client.ConnectConfig; +import io.milvus.v2.client.MilvusClientV2; +import io.milvus.v2.common.DataType; +import io.milvus.v2.common.IndexParam; +import io.milvus.v2.service.collection.request.AddFieldReq; +import io.milvus.v2.service.collection.request.CreateCollectionReq; +import io.milvus.v2.service.collection.request.HasCollectionReq; +import io.milvus.v2.service.collection.request.LoadCollectionReq; +import io.milvus.v2.service.index.request.CreateIndexReq; +import io.milvus.v2.service.vector.request.AnnSearchReq; +import io.milvus.v2.service.vector.request.DeleteReq; +import io.milvus.v2.service.vector.request.HybridSearchReq; +import io.milvus.v2.service.vector.request.InsertReq; +import io.milvus.v2.service.vector.request.SearchReq; +import io.milvus.v2.service.vector.request.data.BaseVector; +import io.milvus.v2.service.vector.request.data.EmbeddedText; +import io.milvus.v2.service.vector.request.data.FloatVec; +import io.milvus.v2.service.vector.request.ranker.RRFRanker; +import io.milvus.v2.service.vector.response.SearchResp; +import jakarta.annotation.PreDestroy; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Value; +import org.springframework.stereotype.Service; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.UUID; + +/** + * Single knowledge vector backend (Milvus Java SDK v2). + * + *

Supports dense ANN and dense+BM25 hybrid search via {@code hybridSearch} + {@link RRFRanker}. + * Legacy {@code MilvusServiceClient} search is not used.

+ */ +@Service +public class MilvusHybridKnowledgeStore { + + private static final Logger log = LoggerFactory.getLogger(MilvusHybridKnowledgeStore.class); + private static final Gson GSON = new Gson(); + + public static final String FIELD_ID = "id"; + public static final String FIELD_CONTENT = "content"; + public static final String FIELD_SEARCH_TEXT = "search_text"; + public static final String FIELD_SPARSE = "sparse_vector"; + public static final String FIELD_DENSE = "vector"; + public static final String FIELD_METADATA = "metadata"; + + private final MilvusProperties milvusProperties; + + @Value("${milvus.collection:biz_hybrid}") + private String collectionName = "biz_hybrid"; + + @Value("${retrieval.hybrid.rrf-k:60}") + private int rrfK = 60; + + @Value("${retrieval.normalization.max-l2-distance:2.0}") + private double maxL2Distance = 2.0; + + @Value("${retrieval.kb-scope:}") + private String kbScope = ""; + + private volatile MilvusClientV2 client; + + public MilvusHybridKnowledgeStore(MilvusProperties milvusProperties) { + this.milvusProperties = milvusProperties; + } + + public synchronized MilvusClientV2 client() { + if (client == null) { + client = connect(); + ensureCollection(client); + loadCollection(client); + } + return client; + } + + public String collectionName() { + return collectionName; + } + + public void upsertChunk(String content, + String searchText, + List denseVector, + Map metadata, + int chunkIndex) { + String source = metadata == null ? null : stringVal(metadata.get("_source")); + if (source == null) { + source = metadata == null ? null : stringVal(metadata.get("source")); + } + if (source == null) { + source = metadata == null ? null : stringVal(metadata.get("docId")); + } + String idSeed = (source == null ? "chunk" : source) + "_" + chunkIndex; + String id = UUID.nameUUIDFromBytes(idSeed.getBytes()).toString(); + + JsonObject row = new JsonObject(); + row.addProperty(FIELD_ID, id); + row.addProperty(FIELD_CONTENT, content == null ? "" : content); + row.addProperty(FIELD_SEARCH_TEXT, searchText == null ? "" : searchText); + row.add(FIELD_DENSE, GSON.toJsonTree(denseVector)); + row.add(FIELD_METADATA, GSON.toJsonTree(metadata == null ? Map.of() : metadata)); + + client().insert(InsertReq.builder() + .collectionName(collectionName) + .data(List.of(row)) + .build()); + } + + public void deleteByDocId(String docId) { + if (docId == null || docId.isBlank()) { + return; + } + String filter = "metadata[\"docId\"] == \"" + escapeFilter(docId) + "\""; + client().delete(DeleteReq.builder() + .collectionName(collectionName) + .filter(filter) + .build()); + } + + public void deleteBySource(String sourcePath) { + if (sourcePath == null || sourcePath.isBlank()) { + return; + } + String normalized = sourcePath.replace('\\', '/'); + String filter = "metadata[\"_source\"] == \"" + escapeFilter(normalized) + "\""; + client().delete(DeleteReq.builder() + .collectionName(collectionName) + .filter(filter) + .build()); + } + + public List searchDense(String queryEmbeddingText, + List queryVector, + int topK, + String category) { + String filter = buildFilter(category); + SearchReq.SearchReqBuilder builder = SearchReq.builder() + .collectionName(collectionName) + .annsField(FIELD_DENSE) + .data(List.of(new FloatVec(queryVector))) + .topK(topK) + .outputFields(List.of(FIELD_ID, FIELD_CONTENT, FIELD_METADATA)) + .metricType(IndexParam.MetricType.L2); + if (filter != null) { + builder.filter(filter); + } + SearchResp resp = client().search(builder.build()); + return toSearchResults(resp, "l2_distance", false); + } + + /** + * Dense + BM25 hybrid fused by RRF. Dense L2 scores are attached when the same id + * appears in a parallel dense search so quality thresholds stay meaningful. + */ + public List searchHybrid(String queryText, + List queryVector, + int topK, + String category) { + String filter = buildFilter(category); + int pathTopK = Math.max(topK, 10); + + AnnSearchReq.AnnSearchReqBuilder denseAnn = AnnSearchReq.builder() + .vectorFieldName(FIELD_DENSE) + .vectors(List.of((BaseVector) new FloatVec(queryVector))) + .topK(pathTopK) + .metricType(IndexParam.MetricType.L2) + .params("{\"nprobe\":10}"); + if (filter != null) { + denseAnn.filter(filter); + } + + AnnSearchReq.AnnSearchReqBuilder sparseAnn = AnnSearchReq.builder() + .vectorFieldName(FIELD_SPARSE) + .vectors(List.of((BaseVector) new EmbeddedText(queryText == null ? "" : queryText))) + .topK(pathTopK) + .metricType(IndexParam.MetricType.BM25); + if (filter != null) { + sparseAnn.filter(filter); + } + + HybridSearchReq hybridReq = HybridSearchReq.builder() + .collectionName(collectionName) + .searchRequests(List.of(denseAnn.build(), sparseAnn.build())) + .ranker(new RRFRanker(Math.max(1, rrfK))) + .topK(topK) + .outFields(List.of(FIELD_ID, FIELD_CONTENT, FIELD_METADATA)) + .build(); + + SearchResp hybridResp = client().hybridSearch(hybridReq); + List fused = toSearchResults(hybridResp, "rrf_fused", true); + + // Attach dense-compatible L2 when available. + Map denseScores = new HashMap<>(); + try { + for (VectorSearchService.SearchResult denseHit : + searchDense(queryText, queryVector, pathTopK, category)) { + if (denseHit.getId() != null) { + denseScores.put(denseHit.getId(), denseHit.getScore()); + } + } + } catch (Exception e) { + log.warn("Dense score enrichment failed: {}", e.getMessage()); + } + for (VectorSearchService.SearchResult hit : fused) { + Float dense = denseScores.get(hit.getId()); + if (dense != null) { + hit.setScore(dense); + hit.setScoreLabel("l2_distance"); + } else { + // BM25-only hit: treat as weak for legacy thresholds + hit.setScore((float) maxL2Distance); + hit.setScoreLabel("bm25_only_no_dense"); + } + } + return fused; + } + + private List toSearchResults(SearchResp resp, + String scoreLabel, + boolean fused) { + List out = new ArrayList<>(); + if (resp == null || resp.getSearchResults() == null || resp.getSearchResults().isEmpty()) { + return out; + } + List first = resp.getSearchResults().get(0); + if (first == null) { + return out; + } + for (SearchResp.SearchResult row : first) { + VectorSearchService.SearchResult mapped = new VectorSearchService.SearchResult(); + Object id = row.getId(); + mapped.setId(id == null ? null : String.valueOf(id)); + Map entity = row.getEntity() == null ? Map.of() : row.getEntity(); + Object content = entity.get(FIELD_CONTENT); + mapped.setContent(content == null ? null : String.valueOf(content)); + Object metadata = entity.get(FIELD_METADATA); + if (metadata instanceof JsonObject jsonObject) { + mapped.setMetadata(jsonObject.toString()); + } else if (metadata instanceof Map map) { + mapped.setMetadata(GSON.toJson(map)); + } else if (metadata != null) { + mapped.setMetadata(String.valueOf(metadata)); + } + Float score = row.getScore(); + mapped.setRawScore(score == null ? null : score.doubleValue()); + mapped.setScoreLabel(scoreLabel); + if (fused) { + // temporary; may be overwritten with dense L2 + mapped.setScore(score == null ? (float) maxL2Distance : invertUnknownScore(score)); + } else { + mapped.setScore(score == null ? (float) maxL2Distance : score); + } + out.add(mapped); + } + return out; + } + + private float invertUnknownScore(float score) { + // RRF-like small scores: map higher better -> small L2-like distance + double bounded = Math.max(0.0, Math.min(1.0, score)); + if (score > 1.0f) { + // already distance-like + return score; + } + return (float) ((1.0 - bounded) * maxL2Distance); + } + + private String buildFilter(String category) { + List parts = new ArrayList<>(); + String categoryFilter = trimToNull(category); + if (categoryFilter != null) { + parts.add("metadata[\"category\"] == \"" + escapeFilter(categoryFilter) + "\""); + } + String scope = trimToNull(kbScope); + if (scope != null) { + parts.add("metadata[\"kb_scope\"] == \"" + escapeFilter(scope) + "\""); + } + return parts.isEmpty() ? null : String.join(" && ", parts); + } + + private MilvusClientV2 connect() { + String uri; + if (milvusProperties.isSecure() || milvusProperties.getPort() == 443) { + uri = "https://" + milvusProperties.getHost() + ":" + milvusProperties.getPort(); + } else { + uri = "http://" + milvusProperties.getHost() + ":" + milvusProperties.getPort(); + } + ConnectConfig.ConnectConfigBuilder builder = ConnectConfig.builder() + .uri(uri) + .connectTimeoutMs(milvusProperties.getTimeout() == null ? 10000L : milvusProperties.getTimeout()); + if (milvusProperties.getToken() != null && !milvusProperties.getToken().isBlank()) { + builder.token(milvusProperties.getToken()); + builder.secure(true); + } else if (milvusProperties.getUsername() != null && !milvusProperties.getUsername().isBlank()) { + builder.username(milvusProperties.getUsername()); + builder.password(milvusProperties.getPassword()); + } + if (milvusProperties.getDatabase() != null && !milvusProperties.getDatabase().isBlank()) { + builder.dbName(milvusProperties.getDatabase()); + } + log.info("Connecting MilvusClientV2 uri={} db={} collection={}", + uri, milvusProperties.getDatabase(), collectionName); + return new MilvusClientV2(builder.build()); + } + + private void ensureCollection(MilvusClientV2 milvusClient) { + Boolean exists = milvusClient.hasCollection(HasCollectionReq.builder() + .collectionName(collectionName) + .build()); + if (Boolean.TRUE.equals(exists)) { + log.info("Hybrid collection '{}' already exists", collectionName); + return; + } + log.info("Creating hybrid collection '{}'", collectionName); + + CreateCollectionReq.CollectionSchema schema = milvusClient.createSchema(); + schema.setEnableDynamicField(false); + schema.addField(AddFieldReq.builder() + .fieldName(FIELD_ID) + .dataType(DataType.VarChar) + .maxLength(MilvusConstants.ID_MAX_LENGTH) + .isPrimaryKey(true) + .autoID(false) + .build()); + schema.addField(AddFieldReq.builder() + .fieldName(FIELD_CONTENT) + .dataType(DataType.VarChar) + .maxLength(MilvusConstants.CONTENT_MAX_LENGTH) + .build()); + schema.addField(AddFieldReq.builder() + .fieldName(FIELD_SEARCH_TEXT) + .dataType(DataType.VarChar) + .maxLength(MilvusConstants.CONTENT_MAX_LENGTH) + .enableAnalyzer(true) + .build()); + schema.addField(AddFieldReq.builder() + .fieldName(FIELD_SPARSE) + .dataType(DataType.SparseFloatVector) + .build()); + schema.addField(AddFieldReq.builder() + .fieldName(FIELD_DENSE) + .dataType(DataType.FloatVector) + .dimension(milvusProperties.getVectorDim()) + .build()); + schema.addField(AddFieldReq.builder() + .fieldName(FIELD_METADATA) + .dataType(DataType.JSON) + .build()); + schema.addFunction(CreateCollectionReq.Function.builder() + .functionType(FunctionType.BM25) + .name("bm25_fn") + .inputFieldNames(List.of(FIELD_SEARCH_TEXT)) + .outputFieldNames(List.of(FIELD_SPARSE)) + .build()); + + milvusClient.createCollection(CreateCollectionReq.builder() + .collectionName(collectionName) + .collectionSchema(schema) + .description("Knowledge hybrid dense+BM25 collection") + .numShards(MilvusConstants.DEFAULT_SHARD_NUMBER) + .build()); + + List indexes = List.of( + IndexParam.builder() + .fieldName(FIELD_DENSE) + .indexType(IndexParam.IndexType.IVF_FLAT) + .metricType(IndexParam.MetricType.L2) + .extraParams(Map.of("nlist", 128)) + .build(), + IndexParam.builder() + .fieldName(FIELD_SPARSE) + .indexType(IndexParam.IndexType.SPARSE_INVERTED_INDEX) + .metricType(IndexParam.MetricType.BM25) + .build() + ); + milvusClient.createIndex(CreateIndexReq.builder() + .collectionName(collectionName) + .indexParams(indexes) + .build()); + log.info("Hybrid collection '{}' created with dense+BM25 indexes", collectionName); + } + + private void loadCollection(MilvusClientV2 milvusClient) { + milvusClient.loadCollection(LoadCollectionReq.builder() + .collectionName(collectionName) + .build()); + } + + @PreDestroy + public void close() { + if (client != null) { + try { + client.close(); + } catch (Exception e) { + log.warn("Error closing MilvusClientV2: {}", e.getMessage()); + } + } + } + + private static String escapeFilter(String value) { + return value.replace("\\", "\\\\").replace("\"", "\\\""); + } + + private static String trimToNull(String value) { + if (value == null || value.isBlank()) { + return null; + } + return value.trim(); + } + + private static String stringVal(Object value) { + return value == null ? null : String.valueOf(value); + } +} diff --git a/src/main/java/com/superbiz/agent/service/retrieval/VectorKnowledgeSearchAdapter.java b/src/main/java/com/superbiz/agent/service/retrieval/VectorKnowledgeSearchAdapter.java index fca46dc..c7d2245 100644 --- a/src/main/java/com/superbiz/agent/service/retrieval/VectorKnowledgeSearchAdapter.java +++ b/src/main/java/com/superbiz/agent/service/retrieval/VectorKnowledgeSearchAdapter.java @@ -2,22 +2,19 @@ package com.superbiz.agent.service.retrieval; import com.fasterxml.jackson.databind.ObjectMapper; import com.superbiz.agent.service.VectorSearchService; -import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component; import java.util.ArrayList; import java.util.LinkedHashMap; import java.util.List; -import java.util.Locale; import java.util.Map; /** - * {@link KnowledgeSearchPort} backed by the existing vector search facade. + * {@link KnowledgeSearchPort} adapter. * - *
    - *
  • {@code DENSE}: single dense path (legacy behavior)
    • - *
  • {@code HYBRID}: dense unfiltered + optional dense filtered + lexical rank, fused by RRF
    • - *
+ *

Delegates to {@link VectorSearchService}, which is backed solely by + * Milvus V2 dense / dense+BM25 hybrid store. Mode selection lives in + * {@code retrieval.search.mode}.

*/ @Component public class VectorKnowledgeSearchAdapter implements KnowledgeSearchPort { @@ -25,21 +22,6 @@ public class VectorKnowledgeSearchAdapter implements KnowledgeSearchPort { private final VectorSearchService vectorSearchService; private final ObjectMapper objectMapper; - @Value("${retrieval.search.mode:dense}") - private String configuredMode = "dense"; - - @Value("${retrieval.hybrid.rrf-k:60}") - private int rrfK = 60; - - @Value("${retrieval.hybrid.weight.dense-unfiltered:1.0}") - private double weightDenseUnfiltered = 1.0; - - @Value("${retrieval.hybrid.weight.dense-filtered:1.0}") - private double weightDenseFiltered = 1.0; - - @Value("${retrieval.hybrid.weight.lexical:1.0}") - private double weightLexical = 1.0; - public VectorKnowledgeSearchAdapter(VectorSearchService vectorSearchService, ObjectMapper objectMapper) { this.vectorSearchService = vectorSearchService; this.objectMapper = objectMapper; @@ -47,100 +29,14 @@ public class VectorKnowledgeSearchAdapter implements KnowledgeSearchPort { @Override public List search(KnowledgeSearchRequest request) { - KnowledgeSearchMode mode = resolveMode(request.mode()); - if (mode == KnowledgeSearchMode.HYBRID) { - return searchHybrid(request); - } - return searchDense(request.query(), request.topK(), request.categoryFilter()); - } - - private List searchDense(String query, int topK, String categoryFilter) { + // Mode is global on VectorSearchService; request.mode is advisory for future per-call overrides. List results = vectorSearchService.searchSimilarDocuments( - query, topK, categoryFilter); + request.query(), + request.topK(), + request.categoryFilter()); return toHits(results); } - private List searchHybrid(KnowledgeSearchRequest request) { - String query = request.query(); - int topK = request.topK(); - String category = trimToNull(request.categoryFilter()); - - List unfiltered = searchDense(query, topK, null); - List filtered = category == null - ? List.of() - : searchDense(query, topK, category); - - Map unionByKey = new LinkedHashMap<>(); - for (KnowledgeSearchHit hit : unfiltered) { - unionByKey.putIfAbsent(hit.evidenceKey(), hit); - } - for (KnowledgeSearchHit hit : filtered) { - unionByKey.putIfAbsent(hit.evidenceKey(), hit); - } - List union = new ArrayList<>(unionByKey.values()); - List lexical = LexicalRanker.rank(query, union); - - List> paths = new ArrayList<>(); - paths.add(new RrfFusion.RankedPath<>("dense_unfiltered", unfiltered, weightDenseUnfiltered)); - if (!filtered.isEmpty()) { - paths.add(new RrfFusion.RankedPath<>("dense_filtered", filtered, weightDenseFiltered)); - } - if (!lexical.isEmpty()) { - paths.add(new RrfFusion.RankedPath<>("lexical", lexical, weightLexical)); - } - - List> fused = RrfFusion.fuse( - paths, rrfK, KnowledgeSearchHit::evidenceKey); - - List ordered = new ArrayList<>(); - int rank = 1; - for (RrfFusion.Scored scored : fused) { - if (ordered.size() >= topK) { - break; - } - KnowledgeSearchHit base = scored.item(); - Map metadata = new LinkedHashMap<>( - base.metadata() == null ? Map.of() : base.metadata()); - metadata.put("fusedScore", Double.toString(scored.rrfScore())); - metadata.put("fusionRanks", scored.ranks().toString()); - metadata.put("fusionRank", Integer.toString(rank)); - ordered.add(new KnowledgeSearchHit( - base.id(), - base.content(), - base.score(), - base.rawScore(), - base.scoreLabel(), - base.metadataJson(), - metadata, - base.docId(), - base.chunkIndex(), - base.evidenceKey(), - base.source(), - base.title(), - base.breadcrumb(), - rank - )); - rank++; - } - return ordered; - } - - private KnowledgeSearchMode resolveMode(KnowledgeSearchMode requestMode) { - if (requestMode == KnowledgeSearchMode.HYBRID) { - return KnowledgeSearchMode.HYBRID; - } - if (requestMode == KnowledgeSearchMode.DENSE) { - // Allow global config to force hybrid even if caller passes DENSE default. - String configured = configuredMode == null ? "dense" : configuredMode.trim().toLowerCase(Locale.ROOT); - if ("hybrid".equals(configured)) { - return KnowledgeSearchMode.HYBRID; - } - return KnowledgeSearchMode.DENSE; - } - String configured = configuredMode == null ? "dense" : configuredMode.trim().toLowerCase(Locale.ROOT); - return "hybrid".equals(configured) ? KnowledgeSearchMode.HYBRID : KnowledgeSearchMode.DENSE; - } - private List toHits(List results) { if (results == null || results.isEmpty()) { return List.of(); @@ -201,11 +97,4 @@ public class VectorKnowledgeSearchAdapter implements KnowledgeSearchPort { return Map.of(); } } - - private static String trimToNull(String value) { - if (value == null || value.isBlank()) { - return null; - } - return value.trim(); - } } diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml index 648d355..f3cf67e 100644 --- a/src/main/resources/application.yml +++ b/src/main/resources/application.yml @@ -25,6 +25,8 @@ milvus: token: ${MILVUS_TOKEN} secure: true vector-dim: 1024 # BGE-M3 = 1024，换模型时同步改 + # dense+BM25 hybrid collection (requires reindex; legacy "biz" is not used by knowledge path) + collection: biz_hybrid # ===================================================== # 模型路由配置 @@ -162,16 +164,24 @@ rag: enabled: false content-preview-limit: 300 -# 检索归一化配置 +# 检索配置（单一 Milvus V2 后端；已移除 sdk/spring/auto 路由） retrieval: - kb-scope: "" # empty means search all legacy documents; use rag-eval for eval seed docs - vector-store: - mode: auto # auto | spring-ai | sdk + kb-scope: "" # empty means search all documents in hybrid collection + search: + mode: hybrid # dense | hybrid (dense + BM25 RRF) + hybrid: + rrf-k: 60 normalization: max-l2-distance: 2.0 # L2 距离上界（BGE-M3 单位向量 = 2.0） highly-relevant-threshold: 0.75 # similarity >= 0.75 → HIGHLY_RELEVANT reference-threshold: 0.5 # similarity >= 0.5 → REFERENCE +# RAG 召回宽度（Delivery 1） +rag: + retrieve-k: 20 + return-n: 5 + max-chunks-per-document: 2 + # Prometheus 配置 prometheus: base-url: http://localhost:9090 diff --git a/src/test/java/com/superbiz/agent/service/VectorSearchServiceTest.java b/src/test/java/com/superbiz/agent/service/VectorSearchServiceTest.java index 5a5cc5f..9945b21 100644 --- a/src/test/java/com/superbiz/agent/service/VectorSearchServiceTest.java +++ b/src/test/java/com/superbiz/agent/service/VectorSearchServiceTest.java @@ -1,208 +1,61 @@ package com.superbiz.agent.service; -import com.fasterxml.jackson.databind.ObjectMapper; +import com.superbiz.agent.service.milvus.MilvusHybridKnowledgeStore; import org.junit.jupiter.api.Test; -import org.mockito.ArgumentCaptor; -import org.springframework.ai.document.Document; -import org.springframework.ai.vectorstore.SearchRequest; -import org.springframework.ai.vectorstore.VectorStore; -import org.springframework.beans.factory.ObjectProvider; import org.springframework.test.util.ReflectionTestUtils; import java.util.List; -import java.util.Map; import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertNull; -import static org.junit.jupiter.api.Assertions.assertTrue; -import static org.mockito.ArgumentMatchers.any; import static org.mockito.ArgumentMatchers.eq; -import static org.mockito.Mockito.doReturn; +import static org.mockito.ArgumentMatchers.isNull; import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.never; -import static org.mockito.Mockito.spy; import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; class VectorSearchServiceTest { @Test - void sdkModeBypassesVectorStore() { - VectorSearchService service = spy(new VectorSearchService()); - setMode(service, "sdk"); - VectorSearchService.SearchResult expected = result("sdk-doc", 0.2f); - doReturn(List.of(expected)) - .when(service).searchSimilarDocumentsWithSdk("query", 3, null); + void denseModeCallsHybridStoreDenseSearch() { + MilvusHybridKnowledgeStore store = mock(MilvusHybridKnowledgeStore.class); + VectorEmbeddingService embeddingService = mock(VectorEmbeddingService.class); + when(embeddingService.generateQueryVector("query")).thenReturn(List.of(0.1f, 0.2f)); + VectorSearchService.SearchResult expected = result("doc-1", 0.2f); + when(store.searchDense(eq("query"), eq(List.of(0.1f, 0.2f)), eq(3), isNull())) + .thenReturn(List.of(expected)); + when(store.collectionName()).thenReturn("biz_hybrid"); + + VectorSearchService service = new VectorSearchService(); + ReflectionTestUtils.setField(service, "knowledgeStore", store); + ReflectionTestUtils.setField(service, "embeddingService", embeddingService); + ReflectionTestUtils.setField(service, "searchMode", "dense"); List results = service.searchSimilarDocuments("query", 3, null); assertEquals(List.of(expected), results); - verify(service, never()).searchSimilarDocumentsWithVectorStore(any(), eq(3), any()); + verify(store).searchDense(eq("query"), eq(List.of(0.1f, 0.2f)), eq(3), isNull()); } @Test - void autoModeUsesVectorStoreWhenAvailable() { - VectorStore vectorStore = mock(VectorStore.class); - ObjectProvider provider = mock(ObjectProvider.class); - when(provider.getIfAvailable()).thenReturn(vectorStore); - when(vectorStore.similaritySearch(any(SearchRequest.class))).thenReturn(List.of( - Document.builder() - .id("spring-doc") - .text("spring content") - .metadata(Map.of("_source", "spring.md", "category", "api")) - .score(0.8) - .build() - )); + void hybridModeCallsHybridStoreHybridSearch() { + MilvusHybridKnowledgeStore store = mock(MilvusHybridKnowledgeStore.class); + VectorEmbeddingService embeddingService = mock(VectorEmbeddingService.class); + when(embeddingService.generateQueryVector("pool")).thenReturn(List.of(0.3f)); + VectorSearchService.SearchResult expected = result("doc-h", 0.4f); + when(store.searchHybrid(eq("pool"), eq(List.of(0.3f)), eq(5), eq("mysql"))) + .thenReturn(List.of(expected)); + when(store.collectionName()).thenReturn("biz_hybrid"); VectorSearchService service = new VectorSearchService(); - setMode(service, "auto"); - setVectorStore(service, provider); + ReflectionTestUtils.setField(service, "knowledgeStore", store); + ReflectionTestUtils.setField(service, "embeddingService", embeddingService); + ReflectionTestUtils.setField(service, "searchMode", "hybrid"); - List results = service.searchSimilarDocuments("query", 3, null); + List results = service.searchSimilarDocuments("pool", 5, "mysql"); assertEquals(1, results.size()); - assertEquals("spring-doc", results.get(0).getId()); - assertEquals("similarity", results.get(0).getScoreLabel()); - assertEquals(0.8, results.get(0).getRawScore(), 0.0001); - assertEquals(0.4f, results.get(0).getScore(), 0.0001); - assertTrue(results.get(0).getMetadata().contains("spring.md")); - } - - @Test - void vectorStoreSearchUsesDistanceMetadataAsCompatibleScore() { - VectorStore vectorStore = mock(VectorStore.class); - ObjectProvider provider = mock(ObjectProvider.class); - when(provider.getIfAvailable()).thenReturn(vectorStore); - when(vectorStore.similaritySearch(any(SearchRequest.class))).thenReturn(List.of( - Document.builder() - .id("spring-doc") - .text("spring content") - .metadata(Map.of("distance", 0.5659486, "category", "api")) - .score(0.4340514) - .build() - )); - - VectorSearchService service = new VectorSearchService(); - setMode(service, "auto"); - setVectorStore(service, provider); - - List results = service.searchSimilarDocuments("query", 3, null); - - assertEquals(1, results.size()); - assertEquals("similarity", results.get(0).getScoreLabel()); - assertEquals(0.4340514, results.get(0).getRawScore(), 0.0001); - assertEquals(0.5659486f, results.get(0).getScore(), 0.0001); - } - - @Test - void autoModeFallsBackToSdkWhenVectorStoreFails() { - VectorStore vectorStore = mock(VectorStore.class); - ObjectProvider provider = mock(ObjectProvider.class); - when(provider.getIfAvailable()).thenReturn(vectorStore); - when(vectorStore.similaritySearch(any(SearchRequest.class))).thenThrow(new RuntimeException("vectorstore down")); - - VectorSearchService service = spy(new VectorSearchService()); - setMode(service, "auto"); - setVectorStore(service, provider); - VectorSearchService.SearchResult fallback = result("sdk-doc", 0.3f); - doReturn(List.of(fallback)) - .when(service).searchSimilarDocumentsWithSdk("query", 3, null); - - List results = service.searchSimilarDocuments("query", 3, null); - - assertEquals(List.of(fallback), results); - } - - @Test - void autoModeFallsBackToSdkWhenVectorStoreUnavailable() { - ObjectProvider provider = mock(ObjectProvider.class); - when(provider.getIfAvailable()).thenReturn(null); - - VectorSearchService service = spy(new VectorSearchService()); - setMode(service, "auto"); - setVectorStore(service, provider); - VectorSearchService.SearchResult fallback = result("sdk-doc", 0.3f); - doReturn(List.of(fallback)) - .when(service).searchSimilarDocumentsWithSdk("query", 3, null); - - List results = service.searchSimilarDocuments("query", 3, null); - - assertEquals(List.of(fallback), results); - } - - @Test - void vectorStoreSearchUsesCategoryFilter() { - VectorStore vectorStore = mock(VectorStore.class); - ObjectProvider provider = mock(ObjectProvider.class); - when(provider.getIfAvailable()).thenReturn(vectorStore); - when(vectorStore.similaritySearch(any(SearchRequest.class))).thenReturn(List.of()); - VectorSearchService service = new VectorSearchService(); - setMode(service, "spring-ai"); - setVectorStore(service, provider); - - service.searchSimilarDocuments("query", 5, "api"); - - ArgumentCaptor requestCaptor = ArgumentCaptor.forClass(SearchRequest.class); - verify(vectorStore).similaritySearch(requestCaptor.capture()); - SearchRequest request = requestCaptor.getValue(); - assertEquals("query", request.getQuery()); - assertEquals(5, request.getTopK()); - assertTrue(request.hasFilterExpression()); - assertTrue(request.toString().contains("category")); - assertTrue(request.toString().contains("api")); - } - - @Test - void springModeUsesVectorStoreAlias() { - VectorStore vectorStore = mock(VectorStore.class); - ObjectProvider provider = mock(ObjectProvider.class); - when(provider.getIfAvailable()).thenReturn(vectorStore); - when(vectorStore.similaritySearch(any(SearchRequest.class))).thenReturn(List.of()); - - VectorSearchService service = new VectorSearchService(); - setMode(service, "spring"); - setVectorStore(service, provider); - - service.searchSimilarDocuments("query", 5, "api"); - - verify(vectorStore).similaritySearch(any(SearchRequest.class)); - } - - @Test - void defaultScopeDoesNotAddMetadataFilter() { - VectorSearchService service = new VectorSearchService(); - setKbScope(service, ""); - - assertNull(service.buildSpringAiFilterExpression(null)); - assertNull(service.buildSdkFilterExpression(null)); - assertEquals("category == 'api'", service.buildSpringAiFilterExpression("api")); - assertEquals("metadata[\"category\"] == \"api\"", service.buildSdkFilterExpression("api")); - } - - @Test - void configuredScopeCombinesWithCategoryFilter() { - VectorSearchService service = new VectorSearchService(); - setKbScope(service, "rag-eval"); - - assertEquals("kb_scope == 'rag-eval'", service.buildSpringAiFilterExpression(null)); - assertEquals("category == 'api' && kb_scope == 'rag-eval'", - service.buildSpringAiFilterExpression("api")); - assertEquals("metadata[\"category\"] == \"api\" && metadata[\"kb_scope\"] == \"rag-eval\"", - service.buildSdkFilterExpression("api")); - } - - private static void setMode(VectorSearchService service, String mode) { - ReflectionTestUtils.setField(service, "vectorStoreMode", mode); - } - - private static void setVectorStore(VectorSearchService service, ObjectProvider provider) { - ReflectionTestUtils.setField(service, "vectorStoreProvider", provider); - ReflectionTestUtils.setField(service, "objectMapper", new ObjectMapper()); - ReflectionTestUtils.setField(service, "maxL2Distance", 2.0); - } - - private static void setKbScope(VectorSearchService service, String kbScope) { - ReflectionTestUtils.setField(service, "kbScope", kbScope); + assertEquals("doc-h", results.get(0).getId()); + verify(store).searchHybrid(eq("pool"), eq(List.of(0.3f)), eq(5), eq("mysql")); } private static VectorSearchService.SearchResult result(String id, float score) { diff --git a/src/test/java/com/superbiz/agent/service/retrieval/VectorKnowledgeSearchAdapterHybridTest.java b/src/test/java/com/superbiz/agent/service/retrieval/VectorKnowledgeSearchAdapterHybridTest.java index 5af03d1..4709633 100644 --- a/src/test/java/com/superbiz/agent/service/retrieval/VectorKnowledgeSearchAdapterHybridTest.java +++ b/src/test/java/com/superbiz/agent/service/retrieval/VectorKnowledgeSearchAdapterHybridTest.java @@ -3,41 +3,36 @@ package com.superbiz.agent.service.retrieval; import com.fasterxml.jackson.databind.ObjectMapper; import com.superbiz.agent.service.VectorSearchService; import org.junit.jupiter.api.Test; -import org.springframework.test.util.ReflectionTestUtils; import java.util.List; import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; class VectorKnowledgeSearchAdapterHybridTest { @Test - void hybridFusesFilteredAndUnfilteredDensePaths() { + void adapterMapsStoreHitsWithChunkIdentity() { VectorSearchService vectorSearchService = mock(VectorSearchService.class); - when(vectorSearchService.searchSimilarDocuments("pool timeout", 3, null)).thenReturn(List.of( - result("u1", "{\"_source\":\"a.md\",\"docId\":\"a\",\"chunkIndex\":0,\"title\":\"generic\"}", "generic pool", 0.4f), - result("u2", "{\"_source\":\"b.md\",\"docId\":\"b\",\"chunkIndex\":0,\"title\":\"other\"}", "other", 0.5f) - )); when(vectorSearchService.searchSimilarDocuments("pool timeout", 3, "mysql")).thenReturn(List.of( - result("f1", "{\"_source\":\"c.md\",\"docId\":\"c\",\"chunkIndex\":0,\"title\":\"mysql pool timeout\"}", "mysql pool timeout runbook", 0.35f) + result("id-1", + "{\"_source\":\"c.md\",\"docId\":\"c\",\"chunkIndex\":0,\"title\":\"mysql pool timeout\"}", + "mysql pool timeout runbook", + 0.35f) )); - VectorKnowledgeSearchAdapter adapter = new VectorKnowledgeSearchAdapter(vectorSearchService, new ObjectMapper()); - ReflectionTestUtils.setField(adapter, "configuredMode", "hybrid"); - ReflectionTestUtils.setField(adapter, "rrfK", 60); - ReflectionTestUtils.setField(adapter, "weightDenseUnfiltered", 1.0); - ReflectionTestUtils.setField(adapter, "weightDenseFiltered", 1.0); - ReflectionTestUtils.setField(adapter, "weightLexical", 1.0); + VectorKnowledgeSearchAdapter adapter = + new VectorKnowledgeSearchAdapter(vectorSearchService, new ObjectMapper()); List hits = adapter.search( new KnowledgeSearchRequest("pool timeout", 3, "mysql", KnowledgeSearchMode.HYBRID)); - assertEquals(3, hits.size()); - assertTrue(hits.stream().anyMatch(hit -> "c#chunk-0".equals(hit.evidenceKey()))); - assertTrue(hits.get(0).metadata().containsKey("fusedScore")); + assertEquals(1, hits.size()); + assertEquals("c#chunk-0", hits.get(0).evidenceKey()); + assertEquals("c.md", hits.get(0).source()); + verify(vectorSearchService).searchSimilarDocuments("pool timeout", 3, "mysql"); } private static VectorSearchService.SearchResult result(String id, String metadata, String content, float score) {