diff --git a/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/.openspec.yaml b/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/.openspec.yaml new file mode 100644 index 0000000..d86f152 --- /dev/null +++ b/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-04 diff --git a/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/design.md b/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/design.md new file mode 100644 index 0000000..df084f2 --- /dev/null +++ b/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/design.md @@ -0,0 +1,71 @@ +## Context + +The project already uses Spring AI/Spring AI Alibaba for model and agent capabilities, but RAG vector retrieval still uses the Milvus Java SDK directly. The current main path is now observable and covered by golden retrieval cases, so the next migration step should compare framework retrieval behavior without changing Chat or AIOps runtime behavior. + +## Goals / Non-Goals + +**Goals:** + +- Introduce a Spring AI VectorStore sidecar behind configuration. +- Keep `lookup_knowledge` and `VectorSearchService` as the default production path. +- Normalize sidecar results into the same comparable shape as current `VectorSearchService.SearchResult`. +- Add an offline or developer-triggered comparison report that runs golden cases through both retrieval paths. +- Capture schema and scoring differences before deciding whether to replace the current implementation. + +**Non-Goals:** + +- Do not replace `VectorSearchService` in this change. +- Do not change document upload, chunking, or Milvus collection schema. +- Do not introduce query transformer, multi-query, RRF, or rerank behavior. +- Do not make Spring AI Advisor the RAG entry point. + +## Decisions + +### Decision: Sidecar over replacement + +Add a separate sidecar service/adapter instead of changing the existing retrieval service. + +Rationale: the current path is already used by Chat and AIOps, and framework behavior may differ in score semantics, metadata filtering, or expected schema. A sidecar lets us compare before cutting over. + +Alternative considered: replace `VectorSearchService` immediately. Rejected because it would conflate dependency integration with retrieval behavior migration. + +### Decision: Preserve explicit tool boundary + +The sidecar will be called by evaluation or diagnostic code, not by implicit Chat Advisor behavior. + +Rationale: the interview value of the project is Agent engineering observability: explicit tool calls, evidence blocks, and `tool_invocation` traces. + +Alternative considered: use Spring AI Advisor directly. Rejected for now because it hides the decision point where the Agent chooses retrieval. + +### Decision: Compare normalized results + +Both retrieval paths should be mapped into a small comparable result shape containing source/doc id, title, breadcrumb, category, score/distance, rank, and content preview. + +Rationale: direct score equality is unlikely because the current path uses Milvus L2 distance while Spring AI abstractions may expose similarity scores or provider-specific values. The first useful comparison is source/rank/metadata coverage. + +### Decision: Keep dependency risk isolated + +If the current dependency set does not expose a compatible Milvus VectorStore, the first implementation should add a narrow optional dependency/config class and keep it disabled by default. + +Rationale: Spring AI version compatibility is a migration risk. The project should still build and run with the current main path if sidecar configuration is absent. + +## Risks / Trade-offs + +- Spring AI Milvus schema may not match the existing collection -> keep sidecar disabled by default and report incompatibility rather than failing the app. +- Score semantics may differ from current L2 distance -> compare rank/source metadata first and label score fields by retrieval path. +- Adding framework dependencies may affect startup auto-configuration -> guard sidecar beans behind properties or conditions. +- Sidecar evaluation may require live Milvus unlike the baseline fixture evaluator -> make live comparison opt-in and keep offline baseline unchanged. + +## Migration Plan + +1. Add sidecar configuration and adapter behind `rag.sidecar.spring-ai.enabled=false`. +2. Add comparison command/script/service that runs golden cases through current retrieval plus sidecar when enabled. +3. Store comparison reports separately from the offline baseline reports. +4. Use report differences to decide whether a later change should replace `VectorSearchService` internals. +5. Rollback is disabling the sidecar property or reverting the sidecar dependency/config only; the main path remains unchanged. + +## Open Questions + +- Which exact Spring AI Milvus VectorStore artifact is compatible with the existing Spring AI/Spring AI Alibaba BOM versions? +- Can the current Milvus collection be queried by Spring AI VectorStore without schema migration, or do we need a second collection for sidecar experiments? +- Should sidecar comparison run from Java tests, a script, or a developer-only endpoint/runner? diff --git a/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/proposal.md b/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/proposal.md new file mode 100644 index 0000000..952c320 --- /dev/null +++ b/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/proposal.md @@ -0,0 +1,27 @@ +## Why + +The current RAG retrieval path talks to Milvus through the raw Java SDK, so framework-level retrieval behavior cannot be compared safely. Before replacing the main path, we need a Spring AI VectorStore sidecar that can run the same golden cases and expose differences without affecting `lookup_knowledge`. + +## What Changes + +- Add a disabled-by-default Spring AI VectorStore sidecar retrieval path. +- Keep the current `VectorSearchService` as the production path for Chat and AIOps. +- Add an adapter/reporting surface that can run golden retrieval cases against both current and sidecar paths. +- Record comparable fields: result id/source, title, breadcrumb, score/distance, category, and metadata. +- Document incompatibilities between the current Milvus schema and Spring AI VectorStore behavior. +- No breaking changes. + +## Capabilities + +### New Capabilities +- None. + +### Modified Capabilities +- `rag-knowledge-retrieval`: Add requirements for sidecar Spring AI retrieval comparison while preserving the explicit `lookup_knowledge` tool boundary. +- `rag-retrieval-evaluation`: Add requirements for comparing baseline retrieval with the sidecar retriever on the golden case set. + +## Impact + +- Affects retrieval service wiring, configuration, and evaluation scripts. +- May add Spring AI VectorStore dependency/configuration if the current dependency set does not already expose it. +- Does not change document upload, chunking, Milvus collection schema, Agent prompts, AIOps diagnosis flow, or the default `lookup_knowledge` runtime path. diff --git a/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/specs/rag-knowledge-retrieval/spec.md b/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/specs/rag-knowledge-retrieval/spec.md new file mode 100644 index 0000000..85ec080 --- /dev/null +++ b/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/specs/rag-knowledge-retrieval/spec.md @@ -0,0 +1,25 @@ +## ADDED Requirements + +### Requirement: Knowledge retrieval SHALL support a disabled-by-default Spring AI sidecar +The retrieval system SHALL allow a Spring AI VectorStore retrieval path to be wired as a sidecar without changing the default `lookup_knowledge` runtime path. + +#### Scenario: Sidecar disabled by default +- **WHEN** the application starts without explicit sidecar enablement +- **THEN** `lookup_knowledge` SHALL continue using the existing retrieval path +- **AND** Chat and AIOps runtime behavior SHALL not depend on the sidecar + +#### Scenario: Sidecar failure does not break main retrieval +- **WHEN** the Spring AI sidecar is enabled but cannot initialize or query successfully +- **THEN** the existing retrieval path SHALL remain usable +- **AND** the failure SHALL be reported as sidecar status rather than as a main retrieval failure + +### Requirement: Knowledge retrieval SHALL normalize sidecar results for comparison +The sidecar retrieval path SHALL expose results in a comparable structure aligned with the current retrieval result shape. + +#### Scenario: Comparable result metadata +- **WHEN** sidecar retrieval returns candidates +- **THEN** each comparable result SHALL include source or doc id, title when available, breadcrumb when available, category when available, rank, content preview, and the sidecar score label/value + +#### Scenario: Score semantics are explicit +- **WHEN** current retrieval and sidecar retrieval scores are compared +- **THEN** the report SHALL label score semantics by path instead of assuming direct numeric equivalence diff --git a/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/specs/rag-retrieval-evaluation/spec.md b/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/specs/rag-retrieval-evaluation/spec.md new file mode 100644 index 0000000..fd31f6b --- /dev/null +++ b/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/specs/rag-retrieval-evaluation/spec.md @@ -0,0 +1,21 @@ +## ADDED Requirements + +### Requirement: Retrieval evaluation SHALL compare current and sidecar retrieval paths +The retrieval evaluation system SHALL provide an opt-in comparison between the existing retrieval path and the Spring AI sidecar retrieval path. + +#### Scenario: Sidecar comparison report +- **WHEN** sidecar comparison is run for the golden case set +- **THEN** the report SHALL include per-case current-path top candidates and sidecar top candidates +- **AND** it SHALL highlight source, breadcrumb, category, rank, and score-label differences + +#### Scenario: Offline baseline remains unchanged +- **WHEN** the fixture-based offline baseline evaluator is run +- **THEN** it SHALL not require live Milvus, Spring Boot, or Spring AI sidecar configuration + +### Requirement: Retrieval evaluation SHALL make sidecar readiness visible +The sidecar comparison report SHALL show whether the Spring AI sidecar was runnable for the current environment. + +#### Scenario: Sidecar unavailable +- **WHEN** sidecar comparison is requested but the sidecar is disabled or unavailable +- **THEN** the report SHALL mark sidecar status as unavailable +- **AND** it SHALL keep current-path baseline results available for review diff --git a/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/tasks.md b/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/tasks.md new file mode 100644 index 0000000..8751c6a --- /dev/null +++ b/openspec/changes/archive/2026-07-04-rag-spring-ai-vectorstore-sidecar/tasks.md @@ -0,0 +1,24 @@ +## 1. Dependency And Configuration + +- [x] 1.1 Inspect available Spring AI VectorStore/Milvus classes for the current dependency set. +- [x] 1.2 Add the narrow dependency or optional configuration needed for the sidecar path. +- [x] 1.3 Add disabled-by-default sidecar properties under RAG configuration. + +## 2. Sidecar Retrieval Adapter + +- [x] 2.1 Define a comparable retrieval result DTO for current and sidecar paths. +- [x] 2.2 Implement a Spring AI sidecar retrieval service that reports readiness and failures without breaking the main path. +- [x] 2.3 Normalize sidecar metadata into source/doc id, title, breadcrumb, category, rank, content preview, and score label/value. + +## 3. Comparison Evaluation + +- [x] 3.1 Add a comparison service or script that runs golden cases against current retrieval and the sidecar path when enabled. +- [x] 3.2 Write sidecar comparison JSON/Markdown reports separate from the offline baseline reports. +- [x] 3.3 Preserve the existing offline evaluator behavior without requiring live Spring AI/Milvus services. + +## 4. Tests And Validation + +- [x] 4.1 Add tests for disabled sidecar fallback/readiness behavior. +- [x] 4.2 Add tests for comparable result normalization and report generation. +- [x] 4.3 Run targeted tests, the offline RAG retrieval baseline evaluator, and OpenSpec validation. +- [x] 4.4 Review git diff to confirm the default `lookup_knowledge` runtime path is unchanged. diff --git a/openspec/specs/rag-knowledge-retrieval/spec.md b/openspec/specs/rag-knowledge-retrieval/spec.md index 926a701..e1b520a 100644 --- a/openspec/specs/rag-knowledge-retrieval/spec.md +++ b/openspec/specs/rag-knowledge-retrieval/spec.md @@ -82,3 +82,27 @@ The system SHALL persist compact evidence block summaries in `tool_invocation.re #### Scenario: Full content is not duplicated into retrieval details - **WHEN** evidence block summaries are persisted - **THEN** full evidence content SHALL be omitted or truncated so the trace record remains compact + +### Requirement: Knowledge retrieval SHALL support a disabled-by-default Spring AI sidecar +The retrieval system SHALL allow a Spring AI VectorStore retrieval path to be wired as a sidecar without changing the default `lookup_knowledge` runtime path. + +#### Scenario: Sidecar disabled by default +- **WHEN** the application starts without explicit sidecar enablement +- **THEN** `lookup_knowledge` SHALL continue using the existing retrieval path +- **AND** Chat and AIOps runtime behavior SHALL not depend on the sidecar + +#### Scenario: Sidecar failure does not break main retrieval +- **WHEN** the Spring AI sidecar is enabled but cannot initialize or query successfully +- **THEN** the existing retrieval path SHALL remain usable +- **AND** the failure SHALL be reported as sidecar status rather than as a main retrieval failure + +### Requirement: Knowledge retrieval SHALL normalize sidecar results for comparison +The sidecar retrieval path SHALL expose results in a comparable structure aligned with the current retrieval result shape. + +#### Scenario: Comparable result metadata +- **WHEN** sidecar retrieval returns candidates +- **THEN** each comparable result SHALL include source or doc id, title when available, breadcrumb when available, category when available, rank, content preview, and the sidecar score label/value + +#### Scenario: Score semantics are explicit +- **WHEN** current retrieval and sidecar retrieval scores are compared +- **THEN** the report SHALL label score semantics by path instead of assuming direct numeric equivalence diff --git a/openspec/specs/rag-retrieval-evaluation/spec.md b/openspec/specs/rag-retrieval-evaluation/spec.md index 6a4861c..5f81cea 100644 --- a/openspec/specs/rag-retrieval-evaluation/spec.md +++ b/openspec/specs/rag-retrieval-evaluation/spec.md @@ -62,3 +62,23 @@ The system SHALL preserve generated baseline reports in JSON and Markdown format #### Scenario: Baseline regeneration is documented - **WHEN** a developer changes golden cases, fixtures, or evaluator logic - **THEN** the repository SHALL explain how to regenerate the retrieval baseline reports + +### Requirement: Retrieval evaluation SHALL compare current and sidecar retrieval paths +The retrieval evaluation system SHALL provide an opt-in comparison between the existing retrieval path and the Spring AI sidecar retrieval path. + +#### Scenario: Sidecar comparison report +- **WHEN** sidecar comparison is run for the golden case set +- **THEN** the report SHALL include per-case current-path top candidates and sidecar top candidates +- **AND** it SHALL highlight source, breadcrumb, category, rank, and score-label differences + +#### Scenario: Offline baseline remains unchanged +- **WHEN** the fixture-based offline baseline evaluator is run +- **THEN** it SHALL not require live Milvus, Spring Boot, or Spring AI sidecar configuration + +### Requirement: Retrieval evaluation SHALL make sidecar readiness visible +The sidecar comparison report SHALL show whether the Spring AI sidecar was runnable for the current environment. + +#### Scenario: Sidecar unavailable +- **WHEN** sidecar comparison is requested but the sidecar is disabled or unavailable +- **THEN** the report SHALL mark sidecar status as unavailable +- **AND** it SHALL keep current-path baseline results available for review diff --git a/src/main/java/com/superbiz/agent/config/RagSidecarProperties.java b/src/main/java/com/superbiz/agent/config/RagSidecarProperties.java new file mode 100644 index 0000000..eb8dd34 --- /dev/null +++ b/src/main/java/com/superbiz/agent/config/RagSidecarProperties.java @@ -0,0 +1,23 @@ +package com.superbiz.agent.config; + +import lombok.Getter; +import org.springframework.boot.context.properties.ConfigurationProperties; +import org.springframework.context.annotation.Configuration; + +@Getter +@Configuration +@ConfigurationProperties(prefix = "rag.sidecar.spring-ai") +public class RagSidecarProperties { + + private boolean enabled = false; + + private int contentPreviewLimit = 300; + + public void setEnabled(boolean enabled) { + this.enabled = enabled; + } + + public void setContentPreviewLimit(int contentPreviewLimit) { + this.contentPreviewLimit = contentPreviewLimit; + } +} diff --git a/src/main/java/com/superbiz/agent/dto/ComparableRetrievalResult.java b/src/main/java/com/superbiz/agent/dto/ComparableRetrievalResult.java new file mode 100644 index 0000000..8aa5794 --- /dev/null +++ b/src/main/java/com/superbiz/agent/dto/ComparableRetrievalResult.java @@ -0,0 +1,31 @@ +package com.superbiz.agent.dto; + +import lombok.Builder; +import lombok.Data; + +@Data +@Builder +public class ComparableRetrievalResult { + + private String path; + + private Integer rank; + + private String id; + + private String source; + + private String docId; + + private String title; + + private String breadcrumb; + + private String category; + + private String contentPreview; + + private String scoreLabel; + + private Double scoreValue; +} diff --git a/src/main/java/com/superbiz/agent/dto/RetrievalComparisonCase.java b/src/main/java/com/superbiz/agent/dto/RetrievalComparisonCase.java new file mode 100644 index 0000000..de837d4 --- /dev/null +++ b/src/main/java/com/superbiz/agent/dto/RetrievalComparisonCase.java @@ -0,0 +1,17 @@ +package com.superbiz.agent.dto; + +import lombok.Builder; +import lombok.Data; + +@Data +@Builder +public class RetrievalComparisonCase { + + private String caseId; + + private String scenario; + + private String query; + + private String category; +} diff --git a/src/main/java/com/superbiz/agent/dto/RetrievalComparisonReport.java b/src/main/java/com/superbiz/agent/dto/RetrievalComparisonReport.java new file mode 100644 index 0000000..1b96e9c --- /dev/null +++ b/src/main/java/com/superbiz/agent/dto/RetrievalComparisonReport.java @@ -0,0 +1,21 @@ +package com.superbiz.agent.dto; + +import lombok.Builder; +import lombok.Data; + +import java.util.List; + +@Data +@Builder +public class RetrievalComparisonReport { + + private String generatedAt; + + private int caseCount; + + private int topK; + + private String sidecarStatus; + + private List results; +} diff --git a/src/main/java/com/superbiz/agent/dto/RetrievalComparisonResult.java b/src/main/java/com/superbiz/agent/dto/RetrievalComparisonResult.java new file mode 100644 index 0000000..56ecc10 --- /dev/null +++ b/src/main/java/com/superbiz/agent/dto/RetrievalComparisonResult.java @@ -0,0 +1,25 @@ +package com.superbiz.agent.dto; + +import lombok.Builder; +import lombok.Data; + +import java.util.List; + +@Data +@Builder +public class RetrievalComparisonResult { + + private String caseId; + + private String scenario; + + private String query; + + private String category; + + private List currentResults; + + private SidecarRetrievalResponse sidecar; + + private List differences; +} diff --git a/src/main/java/com/superbiz/agent/dto/SidecarRetrievalResponse.java b/src/main/java/com/superbiz/agent/dto/SidecarRetrievalResponse.java new file mode 100644 index 0000000..4f72427 --- /dev/null +++ b/src/main/java/com/superbiz/agent/dto/SidecarRetrievalResponse.java @@ -0,0 +1,21 @@ +package com.superbiz.agent.dto; + +import lombok.Builder; +import lombok.Data; + +import java.util.List; + +@Data +@Builder +public class SidecarRetrievalResponse { + + private boolean enabled; + + private boolean available; + + private String status; + + private String errorMessage; + + private List results; +} diff --git a/src/main/java/com/superbiz/agent/service/RagRetrievalSidecarComparisonService.java b/src/main/java/com/superbiz/agent/service/RagRetrievalSidecarComparisonService.java new file mode 100644 index 0000000..2a8ef0d --- /dev/null +++ b/src/main/java/com/superbiz/agent/service/RagRetrievalSidecarComparisonService.java @@ -0,0 +1,179 @@ +package com.superbiz.agent.service; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.superbiz.agent.config.RagSidecarProperties; +import com.superbiz.agent.dto.ComparableRetrievalResult; +import com.superbiz.agent.dto.RetrievalComparisonCase; +import com.superbiz.agent.dto.RetrievalComparisonReport; +import com.superbiz.agent.dto.RetrievalComparisonResult; +import com.superbiz.agent.dto.SidecarRetrievalResponse; +import org.springframework.stereotype.Service; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.OffsetDateTime; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; + +@Service +public class RagRetrievalSidecarComparisonService { + + private final VectorSearchService vectorSearchService; + private final SpringAiVectorStoreSidecarService sidecarService; + private final RetrievalResultNormalizer normalizer; + private final RagSidecarProperties properties; + private final ObjectMapper objectMapper; + + public RagRetrievalSidecarComparisonService(VectorSearchService vectorSearchService, + SpringAiVectorStoreSidecarService sidecarService, + RetrievalResultNormalizer normalizer, + RagSidecarProperties properties, + ObjectMapper objectMapper) { + this.vectorSearchService = vectorSearchService; + this.sidecarService = sidecarService; + this.normalizer = normalizer; + this.properties = properties; + this.objectMapper = objectMapper; + } + + public RetrievalComparisonReport compare(List cases, int topK) { + List results = new ArrayList<>(); + String sidecarStatus = "not_run"; + for (RetrievalComparisonCase comparisonCase : cases) { + List currentResults = normalizeCurrentResults( + vectorSearchService.searchSimilarDocuments( + comparisonCase.getQuery(), + topK, + comparisonCase.getCategory() + ) + ); + SidecarRetrievalResponse sidecar = sidecarService.search( + comparisonCase.getQuery(), + topK, + comparisonCase.getCategory() + ); + sidecarStatus = sidecar.getStatus(); + results.add(RetrievalComparisonResult.builder() + .caseId(comparisonCase.getCaseId()) + .scenario(comparisonCase.getScenario()) + .query(comparisonCase.getQuery()) + .category(comparisonCase.getCategory()) + .currentResults(currentResults) + .sidecar(sidecar) + .differences(compareDifferences(currentResults, sidecar.getResults())) + .build()); + } + + return RetrievalComparisonReport.builder() + .generatedAt(OffsetDateTime.now(ZoneOffset.UTC).toString()) + .caseCount(cases.size()) + .topK(topK) + .sidecarStatus(sidecarStatus) + .results(results) + .build(); + } + + public RetrievalComparisonReport compareGoldenCases(Path caseFile) throws IOException { + var root = objectMapper.readTree(caseFile.toFile()); + int topK = root.path("topK").asInt(5); + List cases = new ArrayList<>(); + for (var node : root.path("cases")) { + cases.add(RetrievalComparisonCase.builder() + .caseId(node.path("caseId").asText()) + .scenario(node.path("scenario").asText()) + .query(node.path("query").asText()) + .build()); + } + return compare(cases, topK); + } + + public void writeReports(RetrievalComparisonReport report, Path jsonPath, Path markdownPath) throws IOException { + createParentDirectories(jsonPath); + createParentDirectories(markdownPath); + objectMapper.writerWithDefaultPrettyPrinter().writeValue(jsonPath.toFile(), report); + Files.writeString(markdownPath, renderMarkdown(report)); + } + + private void createParentDirectories(Path path) throws IOException { + Path parent = path.getParent(); + if (parent != null) { + Files.createDirectories(parent); + } + } + + private List normalizeCurrentResults(List rawResults) { + List results = new ArrayList<>(); + for (int i = 0; i < rawResults.size(); i++) { + results.add(normalizer.fromCurrent(rawResults.get(i), i + 1, properties.getContentPreviewLimit())); + } + return results; + } + + private List compareDifferences(List currentResults, + List sidecarResults) { + if (sidecarResults == null || sidecarResults.isEmpty()) { + return List.of("sidecar_unavailable_or_empty"); + } + List differences = new ArrayList<>(); + String currentTopSource = currentResults.isEmpty() ? null : currentResults.get(0).getSource(); + String sidecarTopSource = sidecarResults.get(0).getSource(); + if (!Objects.equals(currentTopSource, sidecarTopSource)) { + differences.add("top_source_differs"); + } + String currentTopBreadcrumb = currentResults.isEmpty() ? null : currentResults.get(0).getBreadcrumb(); + String sidecarTopBreadcrumb = sidecarResults.get(0).getBreadcrumb(); + if (!Objects.equals(currentTopBreadcrumb, sidecarTopBreadcrumb)) { + differences.add("top_breadcrumb_differs"); + } + String currentScoreLabel = currentResults.isEmpty() ? null : currentResults.get(0).getScoreLabel(); + String sidecarScoreLabel = sidecarResults.get(0).getScoreLabel(); + if (!Objects.equals(currentScoreLabel, sidecarScoreLabel)) { + differences.add("score_label_differs"); + } + return differences; + } + + private String renderMarkdown(RetrievalComparisonReport report) { + StringBuilder builder = new StringBuilder(); + builder.append("# RAG Sidecar Retrieval Comparison\n\n"); + builder.append("Generated at: `").append(report.getGeneratedAt()).append("`\n\n"); + builder.append("- Cases: ").append(report.getCaseCount()).append("\n"); + builder.append("- Top K: ").append(report.getTopK()).append("\n"); + builder.append("- Sidecar status: `").append(report.getSidecarStatus()).append("`\n\n"); + builder.append("| Case | Query | Current Top | Sidecar Top | Differences |\n"); + builder.append("|---|---|---|---|---|\n"); + for (RetrievalComparisonResult result : report.getResults()) { + builder.append("| ") + .append(nullToBlank(result.getCaseId())) + .append(" | ") + .append(escapePipe(result.getQuery())) + .append(" | ") + .append(formatTop(result.getCurrentResults())) + .append(" | ") + .append(formatTop(result.getSidecar() != null ? result.getSidecar().getResults() : List.of())) + .append(" | ") + .append(String.join("
", result.getDifferences())) + .append(" |\n"); + } + return builder.toString(); + } + + private String formatTop(List results) { + if (results == null || results.isEmpty()) { + return ""; + } + ComparableRetrievalResult top = results.get(0); + return escapePipe(nullToBlank(top.getSource())) + " (" + nullToBlank(top.getScoreLabel()) + ")"; + } + + private String escapePipe(String value) { + return nullToBlank(value).replace("|", "\\|"); + } + + private String nullToBlank(String value) { + return value == null ? "" : value; + } +} diff --git a/src/main/java/com/superbiz/agent/service/RetrievalResultNormalizer.java b/src/main/java/com/superbiz/agent/service/RetrievalResultNormalizer.java new file mode 100644 index 0000000..3424b32 --- /dev/null +++ b/src/main/java/com/superbiz/agent/service/RetrievalResultNormalizer.java @@ -0,0 +1,96 @@ +package com.superbiz.agent.service; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.superbiz.agent.dto.ComparableRetrievalResult; +import org.springframework.ai.document.Document; +import org.springframework.stereotype.Component; + +import java.util.LinkedHashMap; +import java.util.Map; + +@Component +public class RetrievalResultNormalizer { + + private final ObjectMapper objectMapper; + + public RetrievalResultNormalizer(ObjectMapper objectMapper) { + this.objectMapper = objectMapper; + } + + public ComparableRetrievalResult fromCurrent(VectorSearchService.SearchResult result, int rank, int previewLimit) { + Map metadata = parseMetadata(result.getMetadata()); + String source = firstNonBlank(metadata.get("_source"), metadata.get("source"), result.getMetadata(), result.getId()); + return ComparableRetrievalResult.builder() + .path("current") + .rank(rank) + .id(result.getId()) + .source(source) + .docId(metadata.get("docId")) + .title(metadata.get("title")) + .breadcrumb(metadata.get("breadcrumb")) + .category(metadata.get("category")) + .contentPreview(truncate(result.getContent(), previewLimit)) + .scoreLabel("l2_distance") + .scoreValue((double) result.getScore()) + .build(); + } + + public ComparableRetrievalResult fromSidecar(Document document, int rank, int previewLimit) { + Map metadata = stringifyMetadata(document.getMetadata()); + String source = firstNonBlank(metadata.get("_source"), metadata.get("source"), metadata.get("docId"), document.getId()); + return ComparableRetrievalResult.builder() + .path("sidecar") + .rank(rank) + .id(document.getId()) + .source(source) + .docId(metadata.get("docId")) + .title(metadata.get("title")) + .breadcrumb(metadata.get("breadcrumb")) + .category(metadata.get("category")) + .contentPreview(truncate(document.getText(), previewLimit)) + .scoreLabel("similarity") + .scoreValue(document.getScore()) + .build(); + } + + private Map parseMetadata(String metadata) { + if (metadata == null || metadata.isBlank()) { + return Map.of(); + } + try { + Map raw = objectMapper.readValue(metadata, Map.class); + return stringifyMetadata(raw); + } catch (Exception e) { + return Map.of(); + } + } + + private Map stringifyMetadata(Map raw) { + if (raw == null || raw.isEmpty()) { + return Map.of(); + } + Map result = new LinkedHashMap<>(); + for (Map.Entry entry : raw.entrySet()) { + if (entry.getKey() != null && entry.getValue() != null) { + result.put(String.valueOf(entry.getKey()), String.valueOf(entry.getValue())); + } + } + return result; + } + + private String firstNonBlank(String... values) { + for (String value : values) { + if (value != null && !value.isBlank()) { + return value; + } + } + return null; + } + + private String truncate(String text, int maxLength) { + if (text == null || text.length() <= maxLength) { + return text; + } + return text.substring(0, maxLength) + "..."; + } +} diff --git a/src/main/java/com/superbiz/agent/service/SpringAiVectorStoreSidecarService.java b/src/main/java/com/superbiz/agent/service/SpringAiVectorStoreSidecarService.java new file mode 100644 index 0000000..362976b --- /dev/null +++ b/src/main/java/com/superbiz/agent/service/SpringAiVectorStoreSidecarService.java @@ -0,0 +1,81 @@ +package com.superbiz.agent.service; + +import com.superbiz.agent.config.RagSidecarProperties; +import com.superbiz.agent.dto.ComparableRetrievalResult; +import com.superbiz.agent.dto.SidecarRetrievalResponse; +import lombok.extern.slf4j.Slf4j; +import org.springframework.ai.document.Document; +import org.springframework.ai.vectorstore.SearchRequest; +import org.springframework.ai.vectorstore.VectorStore; +import org.springframework.beans.factory.ObjectProvider; +import org.springframework.stereotype.Service; + +import java.util.ArrayList; +import java.util.List; + +@Slf4j +@Service +public class SpringAiVectorStoreSidecarService { + + private final RagSidecarProperties properties; + private final ObjectProvider vectorStoreProvider; + private final RetrievalResultNormalizer normalizer; + + public SpringAiVectorStoreSidecarService(RagSidecarProperties properties, + ObjectProvider vectorStoreProvider, + RetrievalResultNormalizer normalizer) { + this.properties = properties; + this.vectorStoreProvider = vectorStoreProvider; + this.normalizer = normalizer; + } + + public SidecarRetrievalResponse search(String query, int topK, String category) { + if (!properties.isEnabled()) { + return unavailable("disabled", null); + } + + VectorStore vectorStore = vectorStoreProvider.getIfAvailable(); + if (vectorStore == null) { + return unavailable("missing_vector_store", "No Spring AI VectorStore bean is available"); + } + + try { + SearchRequest.Builder builder = SearchRequest.builder() + .query(query) + .topK(topK) + .similarityThresholdAll(); + if (category != null && !category.isBlank()) { + builder.filterExpression("category == '" + escapeFilterValue(category) + "'"); + } + + List documents = vectorStore.similaritySearch(builder.build()); + List results = new ArrayList<>(); + for (int i = 0; i < documents.size(); i++) { + results.add(normalizer.fromSidecar(documents.get(i), i + 1, properties.getContentPreviewLimit())); + } + return SidecarRetrievalResponse.builder() + .enabled(true) + .available(true) + .status("available") + .results(results) + .build(); + } catch (Exception e) { + log.warn("Spring AI sidecar retrieval failed: {}", e.getMessage()); + return unavailable("query_failed", e.getMessage()); + } + } + + private SidecarRetrievalResponse unavailable(String status, String errorMessage) { + return SidecarRetrievalResponse.builder() + .enabled(properties.isEnabled()) + .available(false) + .status(status) + .errorMessage(errorMessage) + .results(List.of()) + .build(); + } + + private String escapeFilterValue(String value) { + return value.replace("'", "\\'"); + } +} diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml index 312ba48..f6a255d 100644 --- a/src/main/resources/application.yml +++ b/src/main/resources/application.yml @@ -126,6 +126,10 @@ document: # RAG 配置 rag: top-k: 3 # 检索返回的最相似文档数量 + sidecar: + spring-ai: + enabled: false + content-preview-limit: 300 # 检索归一化配置 retrieval: diff --git a/src/test/java/com/superbiz/agent/service/RagRetrievalSidecarComparisonServiceTest.java b/src/test/java/com/superbiz/agent/service/RagRetrievalSidecarComparisonServiceTest.java new file mode 100644 index 0000000..63c7fe5 --- /dev/null +++ b/src/test/java/com/superbiz/agent/service/RagRetrievalSidecarComparisonServiceTest.java @@ -0,0 +1,117 @@ +package com.superbiz.agent.service; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.superbiz.agent.config.RagSidecarProperties; +import com.superbiz.agent.dto.ComparableRetrievalResult; +import com.superbiz.agent.dto.RetrievalComparisonCase; +import com.superbiz.agent.dto.RetrievalComparisonReport; +import com.superbiz.agent.dto.SidecarRetrievalResponse; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +class RagRetrievalSidecarComparisonServiceTest { + + @TempDir + Path tempDir; + + @Test + void compareWritesSeparateSidecarReports() throws Exception { + VectorSearchService vectorSearchService = mock(VectorSearchService.class); + SpringAiVectorStoreSidecarService sidecarService = mock(SpringAiVectorStoreSidecarService.class); + RagSidecarProperties properties = new RagSidecarProperties(); + RetrievalResultNormalizer normalizer = new RetrievalResultNormalizer(new ObjectMapper()); + RagRetrievalSidecarComparisonService comparisonService = new RagRetrievalSidecarComparisonService( + vectorSearchService, + sidecarService, + normalizer, + properties, + new ObjectMapper() + ); + + VectorSearchService.SearchResult current = new VectorSearchService.SearchResult(); + current.setId("current-1"); + current.setMetadata("{\"_source\":\"current.md\",\"breadcrumb\":\"A\",\"category\":\"api\"}"); + current.setContent("current content"); + current.setScore(0.1f); + when(vectorSearchService.searchSimilarDocuments("timeout", 3, "api")) + .thenReturn(List.of(current)); + when(sidecarService.search("timeout", 3, "api")) + .thenReturn(SidecarRetrievalResponse.builder() + .enabled(true) + .available(true) + .status("available") + .results(List.of(ComparableRetrievalResult.builder() + .path("sidecar") + .rank(1) + .source("sidecar.md") + .breadcrumb("B") + .scoreLabel("similarity") + .scoreValue(0.9) + .build())) + .build()); + + RetrievalComparisonReport report = comparisonService.compare(List.of( + RetrievalComparisonCase.builder() + .caseId("case-1") + .scenario("aiops") + .query("timeout") + .category("api") + .build() + ), 3); + + assertEquals(1, report.getCaseCount()); + assertEquals("available", report.getSidecarStatus()); + assertTrue(report.getResults().get(0).getDifferences().contains("top_source_differs")); + Path json = tempDir.resolve("sidecar.json"); + Path markdown = tempDir.resolve("sidecar.md"); + comparisonService.writeReports(report, json, markdown); + + assertTrue(Files.readString(json).contains("\"sidecarStatus\"")); + assertTrue(Files.readString(markdown).contains("RAG Sidecar Retrieval Comparison")); + } + + @Test + void compareGoldenCasesLoadsExistingCaseShape() throws Exception { + VectorSearchService vectorSearchService = mock(VectorSearchService.class); + SpringAiVectorStoreSidecarService sidecarService = mock(SpringAiVectorStoreSidecarService.class); + RagRetrievalSidecarComparisonService comparisonService = new RagRetrievalSidecarComparisonService( + vectorSearchService, + sidecarService, + new RetrievalResultNormalizer(new ObjectMapper()), + new RagSidecarProperties(), + new ObjectMapper() + ); + when(vectorSearchService.searchSimilarDocuments("query", 2, null)).thenReturn(List.of()); + when(sidecarService.search("query", 2, null)) + .thenReturn(SidecarRetrievalResponse.builder() + .enabled(false) + .available(false) + .status("disabled") + .results(List.of()) + .build()); + Path cases = tempDir.resolve("cases.json"); + Files.writeString(cases, """ + { + "topK": 2, + "cases": [ + {"caseId": "case-1", "scenario": "chat", "query": "query"} + ] + } + """); + + RetrievalComparisonReport report = comparisonService.compareGoldenCases(cases); + + assertEquals(1, report.getCaseCount()); + assertEquals(2, report.getTopK()); + assertEquals("disabled", report.getSidecarStatus()); + } +} diff --git a/src/test/java/com/superbiz/agent/service/RetrievalResultNormalizerTest.java b/src/test/java/com/superbiz/agent/service/RetrievalResultNormalizerTest.java new file mode 100644 index 0000000..2c3e62f --- /dev/null +++ b/src/test/java/com/superbiz/agent/service/RetrievalResultNormalizerTest.java @@ -0,0 +1,60 @@ +package com.superbiz.agent.service; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.superbiz.agent.dto.ComparableRetrievalResult; +import org.junit.jupiter.api.Test; +import org.springframework.ai.document.Document; + +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +class RetrievalResultNormalizerTest { + + private final RetrievalResultNormalizer normalizer = new RetrievalResultNormalizer(new ObjectMapper()); + + @Test + void fromCurrentParsesMetadataAndLabelsDistanceScore() { + VectorSearchService.SearchResult result = new VectorSearchService.SearchResult(); + result.setId("vec-1"); + result.setMetadata("{\"docId\":\"doc-1\",\"_source\":\"docs/api.md\",\"title\":\"API\",\"breadcrumb\":\"A > B\",\"category\":\"api\"}"); + result.setContent("abcdef"); + result.setScore(0.25f); + + ComparableRetrievalResult comparable = normalizer.fromCurrent(result, 1, 3); + + assertEquals("current", comparable.getPath()); + assertEquals("docs/api.md", comparable.getSource()); + assertEquals("doc-1", comparable.getDocId()); + assertEquals("API", comparable.getTitle()); + assertEquals("A > B", comparable.getBreadcrumb()); + assertEquals("api", comparable.getCategory()); + assertEquals("abc...", comparable.getContentPreview()); + assertEquals("l2_distance", comparable.getScoreLabel()); + assertEquals(0.25, comparable.getScoreValue(), 0.0001); + } + + @Test + void fromSidecarNormalizesDocumentMetadataAndLabelsSimilarityScore() { + Document document = Document.builder() + .id("doc-vector") + .text("sidecar content") + .metadata(Map.of( + "docId", "doc-2", + "_source", "docs/sidecar.md", + "title", "Sidecar", + "breadcrumb", "Root > Sidecar", + "category", "rag" + )) + .score(0.91) + .build(); + + ComparableRetrievalResult comparable = normalizer.fromSidecar(document, 2, 100); + + assertEquals("sidecar", comparable.getPath()); + assertEquals(2, comparable.getRank()); + assertEquals("docs/sidecar.md", comparable.getSource()); + assertEquals("similarity", comparable.getScoreLabel()); + assertEquals(0.91, comparable.getScoreValue(), 0.0001); + } +} diff --git a/src/test/java/com/superbiz/agent/service/SpringAiVectorStoreSidecarServiceTest.java b/src/test/java/com/superbiz/agent/service/SpringAiVectorStoreSidecarServiceTest.java new file mode 100644 index 0000000..309ae85 --- /dev/null +++ b/src/test/java/com/superbiz/agent/service/SpringAiVectorStoreSidecarServiceTest.java @@ -0,0 +1,55 @@ +package com.superbiz.agent.service; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.superbiz.agent.config.RagSidecarProperties; +import com.superbiz.agent.dto.SidecarRetrievalResponse; +import org.junit.jupiter.api.Test; +import org.springframework.ai.vectorstore.VectorStore; +import org.springframework.beans.factory.ObjectProvider; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class SpringAiVectorStoreSidecarServiceTest { + + @Test + void disabledSidecarDoesNotRequestVectorStore() { + RagSidecarProperties properties = new RagSidecarProperties(); + ObjectProvider provider = mock(ObjectProvider.class); + SpringAiVectorStoreSidecarService service = new SpringAiVectorStoreSidecarService( + properties, + provider, + new RetrievalResultNormalizer(new ObjectMapper()) + ); + + SidecarRetrievalResponse response = service.search("query", 3, null); + + assertFalse(response.isEnabled()); + assertFalse(response.isAvailable()); + assertEquals("disabled", response.getStatus()); + verify(provider, never()).getIfAvailable(); + } + + @Test + void enabledSidecarReportsMissingVectorStore() { + RagSidecarProperties properties = new RagSidecarProperties(); + properties.setEnabled(true); + ObjectProvider provider = mock(ObjectProvider.class); + when(provider.getIfAvailable()).thenReturn(null); + SpringAiVectorStoreSidecarService service = new SpringAiVectorStoreSidecarService( + properties, + provider, + new RetrievalResultNormalizer(new ObjectMapper()) + ); + + SidecarRetrievalResponse response = service.search("query", 3, "api"); + + assertEquals("missing_vector_store", response.getStatus()); + assertFalse(response.isAvailable()); + assertEquals(0, response.getResults().size()); + } +}