SuperBizAgent-java/openspec/specs/rag-retrieval-evaluation/spec.md

rag-retrieval-evaluation Specification

Purpose

Provide a repeatable offline evaluation baseline for RAG retrieval behavior, so L0, query augmentation, evidence post-processing, and vector store changes can be checked against fixed golden retrieval cases before they affect Agent diagnosis quality.

Requirements

Requirement: Retrieval evaluation SHALL define fixed golden cases

The system SHALL provide a fixed set of RAG retrieval golden cases that can be evaluated deterministically.

Scenario: Golden case includes expected retrieval evidence

• WHEN a retrieval golden case is defined
• THEN it SHALL include a case id, query, expected document identifiers or labels, and expected evidence keywords or breadcrumbs

Scenario: Golden case distinguishes scenario type

• WHEN a retrieval golden case is defined
• THEN it SHALL indicate whether it covers Chat-style knowledge lookup, AIOps alert diagnosis retrieval, or another explicit retrieval scenario type

Requirement: Retrieval evaluation SHALL run offline against fixtures

The evaluator SHALL run without requiring live MySQL, Redis, Milvus, LLM, or Spring Boot services.

Scenario: Fixture evaluation

• WHEN the evaluator is run with a golden case file and retrieval fixture directory
• THEN it SHALL evaluate each case against its matching fixture file
• AND it SHALL not call external services

Scenario: Missing fixture is reported

• WHEN a golden case has no matching retrieval fixture
• THEN the evaluator SHALL report the case as failed or not run with a clear reason

Requirement: Retrieval evaluation SHALL classify hit quality

The evaluator SHALL classify each case into a deterministic hit level.

Scenario: Strong hit classification

• WHEN retrieved candidates include an expected document and satisfy expected breadcrumb or evidence keyword coverage
• THEN the evaluator SHALL classify the case as strong

Scenario: Medium hit classification

• WHEN retrieved candidates include an expected document but do not satisfy expected breadcrumb or evidence keyword coverage
• THEN the evaluator SHALL classify the case as medium

Scenario: Miss classification

• WHEN retrieved candidates do not include expected documents or expected evidence
• THEN the evaluator SHALL classify the case as miss

Requirement: Retrieval evaluation SHALL report ranking signals

The evaluator SHALL report ranking and aggregate retrieval signals suitable for future regression checks.

Scenario: Per-case ranking output

• WHEN a case is evaluated
• THEN the report SHALL include hit level, first expected document rank when available, top candidate labels, and failed checks

Scenario: Aggregate metrics output

• WHEN multiple cases are evaluated
• THEN the report SHALL include case count, strong hit count, medium hit count, miss count, recall at configured K, and average first hit rank when available

Requirement: Retrieval evaluation SHALL preserve baseline reports

The system SHALL preserve generated baseline reports in JSON and Markdown formats.

Scenario: Baseline report generation

• WHEN the baseline evaluator is run for the fixed golden case set
• THEN it SHALL write a JSON report and a Markdown report under the retrieval evaluation documentation area

Scenario: Baseline regeneration is documented

• WHEN a developer changes golden cases, fixtures, or evaluator logic
• THEN the repository SHALL explain how to regenerate the retrieval baseline reports