2.1 KiB
2.1 KiB
ADDED Requirements
Requirement: Evaluation harness SHALL compare reports against a baseline
The system SHALL compare a current diagnosis evaluation report against a saved baseline report using deterministic rules.
Scenario: Aggregate regression detection
- WHEN the current report has a lower pass rate than the baseline report
- THEN the diff SHALL record a regression with the old value, new value, and delta
Scenario: Cost signal detection
- WHEN average tool-call count or average duration changes between reports
- THEN the diff SHALL record the baseline value, current value, and delta
Scenario: Verdict distribution comparison
- WHEN verdict counts differ between reports
- THEN the diff SHALL record the verdict distribution changes
Requirement: Evaluation harness SHALL compare case-level report results
The system SHALL compare case results by case id and report actionable per-case changes.
Scenario: Case pass/fail regression
- WHEN a case changes from passing in the baseline to failing in the current report
- THEN the diff SHALL record a regression for that case
Scenario: Evidence coverage regression
- WHEN a required evidence tool changes from covered to uncovered for a case
- THEN the diff SHALL record a regression naming the case and tool
Scenario: Missing case detection
- WHEN a baseline case is absent from the current report
- THEN the diff SHALL record a regression for the missing case
Scenario: New case detection
- WHEN a current report contains a case absent from the baseline
- THEN the diff SHALL record the case as a non-regression change
Requirement: Evaluation harness SHALL report baseline diff results
The system SHALL expose baseline diff output in structured JSON and reviewable Markdown formats.
Scenario: JSON diff output
- WHEN a baseline diff is written as JSON
- THEN it SHALL include aggregate summary fields and detailed diff items
Scenario: Markdown diff output
- WHEN a baseline diff is written as Markdown
- THEN it SHALL include a readable summary and a table of diff items