SuperBizAgent-java/interview/rag-vectorstore-live-acceptance.md

RAG VectorStore Live Acceptance

Purpose

This note records the live acceptance result for the RAG retrieval refactor.

The goal of this refactor was not only to add a Spring AI abstraction, but to prove that the production retrieval path can:

• Prefer Spring AI VectorStore for Milvus retrieval.
• Preserve the existing Milvus SDK path as fallback.
• Keep the lookup_knowledge tool contract stable.
• Keep L2-distance based relevance normalization compatible.

Current Retrieval Shape

lookup_knowledge / /api/search/similar
  -> VectorSearchService.searchSimilarDocuments(...)
      -> retrieval.vector-store.mode
          -> auto
              -> Spring AI VectorStore
              -> fallback to Milvus SDK if VectorStore fails
          -> spring-ai
              -> Spring AI VectorStore only
          -> sdk
              -> Milvus SDK only

Configuration Verified

The live Milvus/Zilliz database contains the collection:

biz

The Spring AI VectorStore configuration was aligned with the existing SDK collection:

spring:
  ai:
    vectorstore:
      type: milvus
      milvus:
        initialize-schema: false
        database-name: ${milvus.database}
        collection-name: biz
        embedding-dimension: ${milvus.vector-dim}
        metric-type: L2
        id-field-name: id
        content-field-name: content
        metadata-field-name: metadata
        embedding-field-name: vector

Why this matters: the earlier config used business_knowledge, but the SDK path and real collection use biz. That mismatch proved the fallback worked, but it also meant VectorStore was not the successful main path until the config was corrected.

Commands Used

Health check:

Invoke-RestMethod `
  -Uri "http://127.0.0.1:9900/milvus/health" `
  -Method Get

Observed result:

{
  "collections": ["biz"],
  "message": "ok"
}

Direct retrieval check:

Invoke-RestMethod `
  -Uri "http://127.0.0.1:9900/api/search/similar?query=ERR_TIMEOUT&topK=3" `
  -Method Get

Observed result shape:

{
  "code": 200,
  "message": "success",
  "data": [
    {
      "id": "f7dff7c8-5665-3145-9f75-ef741528b914",
      "content": "### ERR_TIMEOUT ...",
      "score": 0.5662,
      "rawScore": 0.4337,
      "scoreLabel": "similarity",
      "metadata": {
        "distance": 0.5662,
        "title": "ERR_TIMEOUT",
        "category": "api"
      }
    }
  ]
}

What The Logs Proved

Before collection alignment:

Starting Spring AI VectorStore search
SearchRequest collectionName:business_knowledge failed
Spring AI VectorStore retrieval failed, falling back to Milvus SDK
Starting Milvus SDK search

After collection alignment:

Starting Spring AI VectorStore search: query=ERR_TIMEOUT
Spring AI VectorStore search complete, candidates=3

This proves:

• auto mode really attempts VectorStore first.
• The fallback is functional when VectorStore fails.
• After config alignment, the main path is Spring AI VectorStore rather than SDK fallback.

Score Semantics

The project keeps three score fields intentionally:

rawScore    -> the raw score from the active retrieval implementation
scoreLabel  -> the semantic meaning of rawScore
score       -> compatibility score used by existing lookup relevance normalization

For SDK retrieval:

rawScore = L2 distance
scoreLabel = l2_distance
score = L2 distance

For Spring AI VectorStore retrieval:

rawScore = Spring AI similarity score
scoreLabel = similarity
score = Milvus distance metadata when available

Why use metadata.distance for score: LookupKnowledgeTool already normalizes relevance from L2 distance. Spring AI Milvus returns similarity as the document score, but also includes the Milvus distance in metadata. Using distance preserves the old relevance behavior while still exposing the new VectorStore score semantics through rawScore and scoreLabel.

Regression Checks

Targeted tests:

mvn -q "-Dtest=VectorSearchServiceTest,LookupKnowledgeToolTest" test

Spec validation:

openspec.cmd validate rag-knowledge-retrieval --specs
openspec.cmd validate rag-retrieval-evaluation --specs

Whitespace check:

git diff --check

Observed result:

All targeted tests passed.
All related specs passed.
No diff-check errors.

Acceptance Conclusion

The VectorStore refactor is accepted for the read path:

• Spring AI VectorStore is integrated and selected in auto mode.
• The SDK path remains available and was proven by fallback behavior.
• The live collection configuration is aligned with the existing Milvus collection.
• The lookup_knowledge public contract remains stable.
• Existing L2-based relevance normalization remains compatible.

The write/indexing path still uses the Milvus SDK. That is an intentional staged migration decision, not a failed acceptance item.