SuperBizAgent-java/openspec/changes/archive/2026-07-28-rag-eval-hybrid-baseline/proposal.md

Change: Align RAG offline eval with hybrid + qualityScore era

Why

eval/rag-retrieval already matches the offline model (golden × fixture × key-field checks × baseline/diff), but it is stuck on the pre-hybrid narrative:

• Snapshot generator still passes dead retrieval.vector-store.mode=spring|sdk.
• Fixtures lack searchMode / scope meta; content still shows boost-style hitReasons and old score story.
• No first-class dense vs hybrid fixture split for recall comparison.
• Golden lacks optional hard-negatives / chunk keys / tags that the design discussion called out.

Without this, offline eval cannot gate the current main path (retrieval.search.mode=hybrid, V2 store, qualityScore post-process).

What Changes

Knife 1 (must) — make offline eval reflect current main path

1. Generator wiring

   • Replace retrieval.vector-store.mode with retrieval.search.mode (hybrid default; dense allowed).
   • Keep -Dretrieval.kb-scope=rag-eval (or configurable).
   • Update scripts/generate_rag_lookup_snapshots.ps1 and any Java system-property docs/comments.

2. Fixture meta

   • Each fixture SHALL record at least: caseId, query, retrievedAt, searchMode, kbScope (when set), plus lookupResult payload.
   • Snapshot writer emits current LookupResult shape (evidence identity fields if already present on blocks).

3. Refresh path

   • Document and support: prepare seed → generate fixtures (hybrid) → eval_rag_retrieval.py → update baseline.
   • Refresh committed fixtures/baseline when live generation is available; if environment blocks live run, ship wiring + docs and record gap in acceptance.

4. Docs

   • Update eval/rag-retrieval/README.md to hybrid/quality narrative; remove spring vector-store as default.

Out of scope this change (confirmed grill)

• Knife 2: fixtures/hybrid vs fixtures/dense dual layout and comparison report.
• Golden extensions: tags / mustNot* / chunk keys / expectedRelevanceLevel hard gates.

Non-goals

• Rewriting eval into a new framework or LLM-as-judge.
• Full threshold calibration productization.
• Neighbor chunks / query rewrite / cross-encoder.
• Changing production retrieval code paths (except eval generator test harness props).
• Forcing live Milvus E2E / fixture refresh when embedding/Milvus unavailable (wiring+docs still complete; refresh recorded as unverified).
• Dense/hybrid dual fixture directories (later change).

Context constraints

• Continues rag-quality-score-unify, rag-bm25-hybrid-drop-sdk, rag-chunk-evidence-identity-dedup.
• Offline checker must remain dependency-free (no Milvus/LLM in eval_rag_retrieval.py).
• Seed isolation via kb_scope=rag-eval stays.

Impact

• Interface: L1/L2 docs + eval artifacts only; no Agent ACI change.
• Risk: refreshed fixtures may change pass/fail vs old baseline — expect intentional baseline update with diff review.
• Scale: micro→standard lean — multi-file scripts/docs/fixtures; no production architecture change. Use standard artifacts for clarity (design + specs + tasks).

Success

• Generator defaults to hybrid search mode; dead vector-store mode flag gone.
• Fixtures carry searchMode meta.
• README describes correct offline/live loop.
• Offline eval runs on refreshed or existing fixtures without requiring removed config keys.
• (If knife 2) dual fixture roots documented and runnable.