From 93764488047362fbac246ee031e1ea082298e5d5 Mon Sep 17 00:00:00 2001 From: aruo <40362743+zyongxin@users.noreply.github.com> Date: Sun, 5 Jul 2026 11:13:50 +0800 Subject: [PATCH] docs: add rag vectorstore interview notes --- interview/rag-vectorstore-interview-notes.md | 167 ++++++++++++++++ interview/rag-vectorstore-live-acceptance.md | 199 +++++++++++++++++++ 2 files changed, 366 insertions(+) create mode 100644 interview/rag-vectorstore-interview-notes.md create mode 100644 interview/rag-vectorstore-live-acceptance.md diff --git a/interview/rag-vectorstore-interview-notes.md b/interview/rag-vectorstore-interview-notes.md new file mode 100644 index 0000000..8dc871b --- /dev/null +++ b/interview/rag-vectorstore-interview-notes.md @@ -0,0 +1,167 @@ +# RAG VectorStore Interview Notes + +## 60-Second Explanation + +I refactored the RAG retrieval path from a direct Milvus SDK-only implementation to a Spring AI `VectorStore` main path, while keeping the SDK path as a fallback. + +The important part is not just the dependency change. I kept `VectorSearchService` as the boundary, so `lookup_knowledge` and the Agent workflow did not need to change. The system now supports three modes: + +```text +auto -> try Spring AI VectorStore, fallback to SDK +spring-ai -> force VectorStore +sdk -> force SDK +``` + +During live verification, the first run found a real config mismatch: VectorStore was pointed at `business_knowledge`, but the real Zilliz collection was `biz`. The fallback worked, so the system still returned results through SDK. After aligning the collection name, the same query went through Spring AI VectorStore successfully. + +I also fixed score compatibility. Spring AI Milvus exposes similarity as the document score, but the old `lookup_knowledge` logic expects L2 distance. So I preserve `rawScore` and `scoreLabel`, and use Milvus `metadata.distance` as the compatibility `score` when available. + +## Architecture Answer + +```text +Agent / API + -> lookup_knowledge or /api/search/similar + -> VectorSearchService + -> Spring AI VectorStore + -> Milvus SDK fallback + -> Milvus/Zilliz collection: biz +``` + +The key design choice is that `VectorSearchService` remains the retrieval facade. This avoids spreading framework-specific code into the Agent tool layer. + +## Why Keep The SDK Path? + +I kept SDK fallback for three reasons: + +- Migration safety: the existing SDK path was already proven against the live collection. +- Runtime resilience: if VectorStore schema mapping or filtering fails, retrieval still works. +- Interview/demo stability: a retrieval abstraction change should not break the main Agent diagnosis demo. + +This was validated in practice. When VectorStore pointed at the wrong collection, `auto` mode fell back to SDK and still returned results. + +## Why Use Spring AI VectorStore At All? + +Using Spring AI `VectorStore` moves the project closer to a standard RAG abstraction: + +- Retrieval code no longer needs to own all Milvus-specific search details. +- Later features such as query transformers, document postprocessors, advisors, or retrievers can be introduced more naturally. +- The code becomes easier to compare with common Spring AI RAG patterns in an interview. + +But I did not blindly replace everything. Writes/indexing still use SDK because changing read and write paths at the same time would make failures harder to isolate. + +## Why Keep L0? + +L0 is no longer treated as the final source of truth. It is a deterministic hint layer: + +- It extracts domain/entity hints from indexed metadata. +- It helps constrain L1 retrieval by category when possible. +- It gives the Agent a stable clue even when semantic retrieval is weak. + +The current design is: + +```text +L0 = domain/entity hint +L1 = semantic retrieval through VectorStore/SDK +postprocess = evidence trace and relevance normalization +``` + +This is easier to defend than saying "we only use vector search." Real incident diagnosis often has exact identifiers, error codes, service names, and alert names. L0 is useful for those. + +## Why Not Use Hidden Spring AI Advisors Directly? + +For this project, `lookup_knowledge` remains an explicit tool. + +Reason: + +- The Agent trace needs to show when knowledge was retrieved. +- `tool_invocation` records input, output preview, relevance level, and evidence metadata. +- The interview story is about auditable Agent execution, not only answer quality. + +Spring AI Advisors may be useful later, but hiding retrieval inside an advisor would make the evidence chain less visible unless we rebuild trace hooks around it. + +## Score Design + +The result object intentionally separates these fields: + +```text +score -> compatibility score used by old relevance normalization +rawScore -> raw score from the retrieval implementation +scoreLabel -> semantic label for rawScore +``` + +For SDK: + +```text +score = L2 distance +rawScore = L2 distance +scoreLabel = l2_distance +``` + +For VectorStore: + +```text +score = metadata.distance if present +rawScore = Spring AI document score +scoreLabel = similarity +``` + +This prevents a subtle bug: if we treat Spring AI similarity as L2 distance, relevance becomes wrong. If we only expose distance, we lose the ability to compare Spring AI behavior. Keeping both makes the migration inspectable. + +## How I Verified It + +I verified at three levels: + +- Unit tests: SDK mode, auto VectorStore mode, fallback mode, category filter, distance metadata mapping. +- Live API: `/api/search/similar?query=ERR_TIMEOUT&topK=3`. +- Logs: confirmed whether the path was VectorStore success or SDK fallback. + +The live API returned: + +```text +scoreLabel = similarity +rawScore = Spring AI similarity +score = Milvus distance metadata +``` + +That means the main path was Spring AI VectorStore and compatibility scoring remained stable. + +## What I Would Do Next + +I would not immediately migrate indexing writes. The next responsible steps are: + +- Add a small live acceptance report for several golden queries. +- Compare `sdk` and `spring-ai` mode side by side for topK overlap. +- Decide whether `VectorIndexService` should move to `VectorStore.add(...)`. +- Add query transformation or hybrid retrieval only after we have baseline metrics. + +This staged approach is intentional: first stabilize the read path, then evaluate retrieval quality, then migrate writes if the abstraction proves reliable. + +## Interview Questions And Short Answers + +### Why did you not remove the SDK? + +Because this is a migration, not a rewrite. SDK fallback gives rollback safety and proved useful when VectorStore config was initially wrong. + +### What changed for `lookup_knowledge`? + +The public contract did not change. It still calls `VectorSearchService.searchSimilarDocuments(...)`. The implementation behind that facade changed. + +### How do you know VectorStore is actually used? + +The logs show `Starting Spring AI VectorStore search` followed by `Spring AI VectorStore search complete`. The API response also has `scoreLabel=similarity`, which only comes from the VectorStore path. + +### What was the main bug found during live validation? + +The configured collection name was wrong. Spring AI looked for `business_knowledge`, but the actual Milvus collection was `biz`. + +### What did fallback prove? + +It proved that `auto` mode is resilient: VectorStore failed, SDK search still returned valid results, and the API did not fail. + +### Why is `metadata.distance` important? + +Because `lookup_knowledge` uses L2 distance normalization. Spring AI returns similarity as the main document score, but the Milvus distance is available in metadata. Using it preserves old relevance behavior. + +### Is this full Spring AI RAG now? + +Not yet. It uses Spring AI VectorStore for the main read path, but keeps explicit tools, custom evidence trace, L0 hints, and SDK indexing. That is deliberate because the project values auditability and staged migration. diff --git a/interview/rag-vectorstore-live-acceptance.md b/interview/rag-vectorstore-live-acceptance.md new file mode 100644 index 0000000..8b61a8c --- /dev/null +++ b/interview/rag-vectorstore-live-acceptance.md @@ -0,0 +1,199 @@ +# RAG VectorStore Live Acceptance + +## Purpose + +This note records the live acceptance result for the RAG retrieval refactor. + +The goal of this refactor was not only to add a Spring AI abstraction, but to prove that the production retrieval path can: + +- Prefer Spring AI `VectorStore` for Milvus retrieval. +- Preserve the existing Milvus SDK path as fallback. +- Keep the `lookup_knowledge` tool contract stable. +- Keep L2-distance based relevance normalization compatible. + +## Current Retrieval Shape + +```text +lookup_knowledge / /api/search/similar + -> VectorSearchService.searchSimilarDocuments(...) + -> retrieval.vector-store.mode + -> auto + -> Spring AI VectorStore + -> fallback to Milvus SDK if VectorStore fails + -> spring-ai + -> Spring AI VectorStore only + -> sdk + -> Milvus SDK only +``` + +## Configuration Verified + +The live Milvus/Zilliz database contains the collection: + +```text +biz +``` + +The Spring AI VectorStore configuration was aligned with the existing SDK collection: + +```yaml +spring: + ai: + vectorstore: + type: milvus + milvus: + initialize-schema: false + database-name: ${milvus.database} + collection-name: biz + embedding-dimension: ${milvus.vector-dim} + metric-type: L2 + id-field-name: id + content-field-name: content + metadata-field-name: metadata + embedding-field-name: vector +``` + +Why this matters: the earlier config used `business_knowledge`, but the SDK path and real collection use `biz`. That mismatch proved the fallback worked, but it also meant VectorStore was not the successful main path until the config was corrected. + +## Commands Used + +Health check: + +```powershell +Invoke-RestMethod ` + -Uri "http://127.0.0.1:9900/milvus/health" ` + -Method Get +``` + +Observed result: + +```json +{ + "collections": ["biz"], + "message": "ok" +} +``` + +Direct retrieval check: + +```powershell +Invoke-RestMethod ` + -Uri "http://127.0.0.1:9900/api/search/similar?query=ERR_TIMEOUT&topK=3" ` + -Method Get +``` + +Observed result shape: + +```json +{ + "code": 200, + "message": "success", + "data": [ + { + "id": "f7dff7c8-5665-3145-9f75-ef741528b914", + "content": "### ERR_TIMEOUT ...", + "score": 0.5662, + "rawScore": 0.4337, + "scoreLabel": "similarity", + "metadata": { + "distance": 0.5662, + "title": "ERR_TIMEOUT", + "category": "api" + } + } + ] +} +``` + +## What The Logs Proved + +Before collection alignment: + +```text +Starting Spring AI VectorStore search +SearchRequest collectionName:business_knowledge failed +Spring AI VectorStore retrieval failed, falling back to Milvus SDK +Starting Milvus SDK search +``` + +After collection alignment: + +```text +Starting Spring AI VectorStore search: query=ERR_TIMEOUT +Spring AI VectorStore search complete, candidates=3 +``` + +This proves: + +- `auto` mode really attempts VectorStore first. +- The fallback is functional when VectorStore fails. +- After config alignment, the main path is Spring AI VectorStore rather than SDK fallback. + +## Score Semantics + +The project keeps three score fields intentionally: + +```text +rawScore -> the raw score from the active retrieval implementation +scoreLabel -> the semantic meaning of rawScore +score -> compatibility score used by existing lookup relevance normalization +``` + +For SDK retrieval: + +```text +rawScore = L2 distance +scoreLabel = l2_distance +score = L2 distance +``` + +For Spring AI VectorStore retrieval: + +```text +rawScore = Spring AI similarity score +scoreLabel = similarity +score = Milvus distance metadata when available +``` + +Why use `metadata.distance` for `score`: `LookupKnowledgeTool` already normalizes relevance from L2 distance. Spring AI Milvus returns similarity as the document score, but also includes the Milvus distance in metadata. Using distance preserves the old relevance behavior while still exposing the new VectorStore score semantics through `rawScore` and `scoreLabel`. + +## Regression Checks + +Targeted tests: + +```powershell +mvn -q "-Dtest=VectorSearchServiceTest,LookupKnowledgeToolTest" test +``` + +Spec validation: + +```powershell +openspec.cmd validate rag-knowledge-retrieval --specs +openspec.cmd validate rag-retrieval-evaluation --specs +``` + +Whitespace check: + +```powershell +git diff --check +``` + +Observed result: + +```text +All targeted tests passed. +All related specs passed. +No diff-check errors. +``` + +## Acceptance Conclusion + +The VectorStore refactor is accepted for the read path: + +- Spring AI VectorStore is integrated and selected in `auto` mode. +- The SDK path remains available and was proven by fallback behavior. +- The live collection configuration is aligned with the existing Milvus collection. +- The `lookup_knowledge` public contract remains stable. +- Existing L2-based relevance normalization remains compatible. + +The write/indexing path still uses the Milvus SDK. That is an intentional staged migration decision, not a failed acceptance item.