SuperBizAgent-java/devflow/projects/2026-07-06-rag-eval-pipeline-closure/brief.md

Brief: rag-eval-pipeline-closure

Background

The modular RAG pipeline now returns LookupResult with evidenceBlocks, contextPack, retrievalTrace, and rerankTrace. The RAG retrieval baseline must validate that full contract, so it can detect regressions in fallback behavior, context packing, or rerank trace.

Goals

1. Reuse the existing offline RAG retrieval baseline.
2. Extend it to support modular LookupResult fixtures.
3. Add golden assertions for selected attempt, fallback reason, evidence status, context sources, and rerank top source.
4. Add a RAG baseline diff path for regression detection.
5. Add a snapshot generator that calls the real LookupKnowledgeTool.
6. Document how RAG baseline and diagnosis baseline form a quality loop.

Non-Goals

• No new production API.
• No LLM-as-judge scoring.
• No production API behavior changes.
• No replacement for diagnosis eval.