SuperBizAgent-java/openspec/changes/archive/2026-07-05-expand-diagnosis-eval-fixtures/specs/diagnosis-eval-harness/spec.md

ADDED Requirements

Requirement: Evaluation harness SHALL provide complete fixture coverage for fixed cases

The system SHALL include an offline trace fixture for every fixed MVP diagnosis evaluation case.

Scenario: Every case resolves to a fixture file

• WHEN the evaluator loads the fixed case definition file
• THEN every case's traceFixture value SHALL resolve to an existing JSON fixture file

Scenario: Fixture files are loadable as diagnosis traces

• WHEN each referenced fixture is loaded
• THEN it SHALL deserialize into the trace response shape used by the evaluator

Requirement: Evaluation harness SHALL preserve a reproducible baseline report

The system SHALL preserve a generated baseline report for the full fixed fixture set.

Scenario: Baseline report includes all fixed cases

• WHEN the baseline report is generated from the fixed case file and fixture directory
• THEN the report SHALL include one result for every fixed case

Scenario: Baseline report is reviewable

• WHEN the baseline report is written
• THEN it SHALL be available in JSON and Markdown formats under the eval documentation area

Scenario: Baseline regeneration is documented

• WHEN a developer changes fixtures or evaluator rules
• THEN the eval documentation SHALL explain how to regenerate the baseline report