Add diagnosis eval baseline diff

2026-07-05 00:59:53 +08:00
parent 4c7c53b024
commit 69deb15330
22 changed files with 1069 additions and 0 deletions
@@ -0,0 +1,28 @@
+## Why
+
+The evaluation baseline is now complete, but developers still need a repeatable way to decide whether a new Agent run regressed against that baseline. A deterministic baseline diff turns saved reports into an actionable regression signal instead of a static artifact.
+
+## What Changes
+
+- Add a baseline diff model that compares two `DiagnosisEvalReport` objects.
+- Detect aggregate changes such as pass-rate drops, verdict distribution shifts, tool-call cost changes, and duration changes.
+- Detect per-case changes such as pass/fail regression, verdict changes, keyword coverage changes, evidence coverage loss, and missing/new cases.
+- Add JSON and Markdown diff output suitable for review.
+- Document how to interpret the diff in the eval docs.
+
+## Capabilities
+
+### New Capabilities
+
+- None.
+
+### Modified Capabilities
+
+- `diagnosis-eval-harness`: Extend the existing evaluation harness so a current report can be compared against the saved baseline report.
+
+## Impact
+
+- Affects eval-only Java code under `src/main/java/com/superbiz/agent/eval`.
+- Adds focused tests under `src/test/java/com/superbiz/agent/eval`.
+- Updates `mvp/eval` documentation and may add sample diff output.
+- No production Agent runtime, API, database schema, or external dependency changes are expected.