SuperBizAgent-java/openspec/changes/archive/2026-07-05-diagnosis-eval-baseline-diff/proposal.md

Why

The evaluation baseline is now complete, but developers still need a repeatable way to decide whether a new Agent run regressed against that baseline. A deterministic baseline diff turns saved reports into an actionable regression signal instead of a static artifact.

What Changes

• Add a baseline diff model that compares two DiagnosisEvalReport objects.
• Detect aggregate changes such as pass-rate drops, verdict distribution shifts, tool-call cost changes, and duration changes.
• Detect per-case changes such as pass/fail regression, verdict changes, keyword coverage changes, evidence coverage loss, and missing/new cases.
• Add JSON and Markdown diff output suitable for review.
• Document how to interpret the diff in the eval docs.

Capabilities

New Capabilities

• None.

Modified Capabilities

• diagnosis-eval-harness: Extend the existing evaluation harness so a current report can be compared against the saved baseline report.

Impact

• Affects eval-only Java code under src/main/java/com/superbiz/agent/eval.
• Adds focused tests under src/test/java/com/superbiz/agent/eval.
• Updates mvp/eval documentation and may add sample diff output.
• No production Agent runtime, API, database schema, or external dependency changes are expected.