test: add rag retrieval baseline

2026-07-05 02:02:27 +08:00
parent 79feed3314
commit 9a2a44d1b5
19 changed files with 946 additions and 0 deletions
@@ -0,0 +1,27 @@
+## Why
+
+The RAG refactor needs a repeatable baseline before changing L0, metadata filtering, post-processing, or Spring AI retriever integration. Without fixed retrieval cases and measurable output, later changes can look cleaner architecturally while silently degrading recall or evidence quality.
+
+## What Changes
+
+- Add a retrieval evaluation baseline for RAG queries, separate from full diagnosis evaluation.
+- Define golden retrieval cases covering Chat-style knowledge lookup and AIOps-style alert diagnosis retrieval.
+- Add a lightweight offline evaluator that compares retrieved candidates against expected documents, breadcrumbs, and evidence keywords.
+- Preserve baseline JSON and Markdown reports so future changes can compare retrieval behavior.
+- No production retrieval behavior changes in this change.
+
+## Capabilities
+
+### New Capabilities
+
+- `rag-retrieval-evaluation`: Defines fixed retrieval golden cases, deterministic retrieval evaluation, and baseline report preservation.
+
+### Modified Capabilities
+
+- None.
+
+## Impact
+
+- Adds retrieval evaluation fixtures, documentation, and scripts.
+- May read existing retrieval/tool trace output or saved fixtures, but does not require live LLM calls.
+- Does not change the `lookup_knowledge` runtime behavior, Milvus schema, document upload API, or Agent flow.