Add diagnosis eval baseline diff

2026-07-05 00:59:53 +08:00
parent 4c7c53b024
commit 69deb15330
22 changed files with 1069 additions and 0 deletions
@@ -0,0 +1,46 @@
+## ADDED Requirements
+
+### Requirement: Evaluation harness SHALL compare reports against a baseline
+The system SHALL compare a current diagnosis evaluation report against a saved baseline report using deterministic rules.
+
+#### Scenario: Aggregate regression detection
+- **WHEN** the current report has a lower pass rate than the baseline report
+- **THEN** the diff SHALL record a regression with the old value, new value, and delta
+
+#### Scenario: Cost signal detection
+- **WHEN** average tool-call count or average duration changes between reports
+- **THEN** the diff SHALL record the baseline value, current value, and delta
+
+#### Scenario: Verdict distribution comparison
+- **WHEN** verdict counts differ between reports
+- **THEN** the diff SHALL record the verdict distribution changes
+
+### Requirement: Evaluation harness SHALL compare case-level report results
+The system SHALL compare case results by case id and report actionable per-case changes.
+
+#### Scenario: Case pass/fail regression
+- **WHEN** a case changes from passing in the baseline to failing in the current report
+- **THEN** the diff SHALL record a regression for that case
+
+#### Scenario: Evidence coverage regression
+- **WHEN** a required evidence tool changes from covered to uncovered for a case
+- **THEN** the diff SHALL record a regression naming the case and tool
+
+#### Scenario: Missing case detection
+- **WHEN** a baseline case is absent from the current report
+- **THEN** the diff SHALL record a regression for the missing case
+
+#### Scenario: New case detection
+- **WHEN** a current report contains a case absent from the baseline
+- **THEN** the diff SHALL record the case as a non-regression change
+
+### Requirement: Evaluation harness SHALL report baseline diff results
+The system SHALL expose baseline diff output in structured JSON and reviewable Markdown formats.
+
+#### Scenario: JSON diff output
+- **WHEN** a baseline diff is written as JSON
+- **THEN** it SHALL include aggregate summary fields and detailed diff items
+
+#### Scenario: Markdown diff output
+- **WHEN** a baseline diff is written as Markdown
+- **THEN** it SHALL include a readable summary and a table of diff items