62 lines
3.2 KiB
Markdown
62 lines
3.2 KiB
Markdown
## ADDED Requirements
|
|
|
|
### Requirement: Retrieval evaluation SHALL define fixed golden cases
|
|
The system SHALL provide a fixed set of RAG retrieval golden cases that can be evaluated deterministically.
|
|
|
|
#### Scenario: Golden case includes expected retrieval evidence
|
|
- **WHEN** a retrieval golden case is defined
|
|
- **THEN** it SHALL include a case id, query, expected document identifiers or labels, and expected evidence keywords or breadcrumbs
|
|
|
|
#### Scenario: Golden case distinguishes scenario type
|
|
- **WHEN** a retrieval golden case is defined
|
|
- **THEN** it SHALL indicate whether it covers Chat-style knowledge lookup, AIOps alert diagnosis retrieval, or another explicit retrieval scenario type
|
|
|
|
### Requirement: Retrieval evaluation SHALL run offline against fixtures
|
|
The evaluator SHALL run without requiring live MySQL, Redis, Milvus, LLM, or Spring Boot services.
|
|
|
|
#### Scenario: Fixture evaluation
|
|
- **WHEN** the evaluator is run with a golden case file and retrieval fixture directory
|
|
- **THEN** it SHALL evaluate each case against its matching fixture file
|
|
- **AND** it SHALL not call external services
|
|
|
|
#### Scenario: Missing fixture is reported
|
|
- **WHEN** a golden case has no matching retrieval fixture
|
|
- **THEN** the evaluator SHALL report the case as failed or not run with a clear reason
|
|
|
|
### Requirement: Retrieval evaluation SHALL classify hit quality
|
|
The evaluator SHALL classify each case into a deterministic hit level.
|
|
|
|
#### Scenario: Strong hit classification
|
|
- **WHEN** retrieved candidates include an expected document and satisfy expected breadcrumb or evidence keyword coverage
|
|
- **THEN** the evaluator SHALL classify the case as `strong`
|
|
|
|
#### Scenario: Medium hit classification
|
|
- **WHEN** retrieved candidates include an expected document but do not satisfy expected breadcrumb or evidence keyword coverage
|
|
- **THEN** the evaluator SHALL classify the case as `medium`
|
|
|
|
#### Scenario: Miss classification
|
|
- **WHEN** retrieved candidates do not include expected documents or expected evidence
|
|
- **THEN** the evaluator SHALL classify the case as `miss`
|
|
|
|
### Requirement: Retrieval evaluation SHALL report ranking signals
|
|
The evaluator SHALL report ranking and aggregate retrieval signals suitable for future regression checks.
|
|
|
|
#### Scenario: Per-case ranking output
|
|
- **WHEN** a case is evaluated
|
|
- **THEN** the report SHALL include hit level, first expected document rank when available, top candidate labels, and failed checks
|
|
|
|
#### Scenario: Aggregate metrics output
|
|
- **WHEN** multiple cases are evaluated
|
|
- **THEN** the report SHALL include case count, strong hit count, medium hit count, miss count, recall at configured K, and average first hit rank when available
|
|
|
|
### Requirement: Retrieval evaluation SHALL preserve baseline reports
|
|
The system SHALL preserve generated baseline reports in JSON and Markdown formats.
|
|
|
|
#### Scenario: Baseline report generation
|
|
- **WHEN** the baseline evaluator is run for the fixed golden case set
|
|
- **THEN** it SHALL write a JSON report and a Markdown report under the retrieval evaluation documentation area
|
|
|
|
#### Scenario: Baseline regeneration is documented
|
|
- **WHEN** a developer changes golden cases, fixtures, or evaluator logic
|
|
- **THEN** the repository SHALL explain how to regenerate the retrieval baseline reports
|