60 lines
3.1 KiB
Markdown
60 lines
3.1 KiB
Markdown
## ADDED Requirements
|
|
|
|
### Requirement: Evaluation harness SHALL define fixed diagnosis cases
|
|
The system SHALL provide a small fixed set of MVP diagnosis evaluation cases with explicit expected trace and answer criteria.
|
|
|
|
#### Scenario: Case definition includes expected evidence
|
|
- **WHEN** an evaluation case is defined
|
|
- **THEN** it SHALL include a case id, user question, expected root-cause keywords, required evidence tools, and allowed verifier verdicts
|
|
|
|
#### Scenario: Case definition can express forbidden behavior
|
|
- **WHEN** a case has known unsafe behavior
|
|
- **THEN** the case definition SHALL be able to declare forbidden answer keywords or forbidden verdicts
|
|
|
|
### Requirement: Evaluation harness SHALL validate diagnosis traces
|
|
The system SHALL validate a diagnosis trace against the corresponding case definition using deterministic rules.
|
|
|
|
#### Scenario: Evidence coverage validation
|
|
- **WHEN** a trace is evaluated
|
|
- **THEN** the evaluator SHALL verify that required evidence tools appear in `toolInvocations` or verifier trace summaries
|
|
|
|
#### Scenario: Verifier evaluation validation
|
|
- **WHEN** a trace is evaluated
|
|
- **THEN** the evaluator SHALL verify that `selfEvaluation.verifier_evaluation.verdict` exists
|
|
- **AND** the verdict SHALL be one of the case's allowed verdicts
|
|
|
|
#### Scenario: Answer keyword validation
|
|
- **WHEN** a trace is evaluated
|
|
- **THEN** the evaluator SHALL verify that the final answer includes at least the configured minimum root-cause keyword coverage
|
|
|
|
#### Scenario: Degraded output validation
|
|
- **WHEN** a trace verdict is `REJECT`
|
|
- **THEN** the evaluator SHALL verify that the final answer follows the degraded-output contract rather than leaking an unverified executor answer
|
|
|
|
### Requirement: Evaluation harness SHALL report quality and cost signals
|
|
The system SHALL produce a report that summarizes pass/fail results and key trace metrics.
|
|
|
|
#### Scenario: JSON report output
|
|
- **WHEN** an evaluation run completes
|
|
- **THEN** the evaluator SHALL output a JSON report with per-case result, failed checks, verdict, evidence coverage, tool-call count, and duration
|
|
|
|
#### Scenario: Markdown report output
|
|
- **WHEN** an evaluation run completes
|
|
- **THEN** the evaluator SHALL output a Markdown report suitable for review in the repository
|
|
|
|
#### Scenario: Aggregate metrics
|
|
- **WHEN** multiple cases are evaluated
|
|
- **THEN** the report SHALL include aggregate pass rate, verdict distribution, average tool-call count, and average duration when available
|
|
|
|
### Requirement: Evaluation harness SHALL support offline fixture mode
|
|
The first evaluator version SHALL be runnable without live MySQL, Redis, Milvus, or LLM services by evaluating saved trace fixtures.
|
|
|
|
#### Scenario: Fixture trace evaluation
|
|
- **WHEN** the evaluator is run against a directory of trace fixture files
|
|
- **THEN** it SHALL evaluate each trace file against its matching case definition
|
|
- **AND** it SHALL not require a running application service
|
|
|
|
#### Scenario: Missing fixture is reported clearly
|
|
- **WHEN** a case has no matching trace fixture
|
|
- **THEN** the evaluator SHALL mark the case as not run or failed with a clear reason
|