29 lines
1.3 KiB
Markdown
29 lines
1.3 KiB
Markdown
## Why
|
|
|
|
The evaluation baseline is now complete, but developers still need a repeatable way to decide whether a new Agent run regressed against that baseline. A deterministic baseline diff turns saved reports into an actionable regression signal instead of a static artifact.
|
|
|
|
## What Changes
|
|
|
|
- Add a baseline diff model that compares two `DiagnosisEvalReport` objects.
|
|
- Detect aggregate changes such as pass-rate drops, verdict distribution shifts, tool-call cost changes, and duration changes.
|
|
- Detect per-case changes such as pass/fail regression, verdict changes, keyword coverage changes, evidence coverage loss, and missing/new cases.
|
|
- Add JSON and Markdown diff output suitable for review.
|
|
- Document how to interpret the diff in the eval docs.
|
|
|
|
## Capabilities
|
|
|
|
### New Capabilities
|
|
|
|
- None.
|
|
|
|
### Modified Capabilities
|
|
|
|
- `diagnosis-eval-harness`: Extend the existing evaluation harness so a current report can be compared against the saved baseline report.
|
|
|
|
## Impact
|
|
|
|
- Affects eval-only Java code under `src/main/java/com/superbiz/agent/eval`.
|
|
- Adds focused tests under `src/test/java/com/superbiz/agent/eval`.
|
|
- Updates `mvp/eval` documentation and may add sample diff output.
|
|
- No production Agent runtime, API, database schema, or external dependency changes are expected.
|