31 lines
759 B
Markdown
31 lines
759 B
Markdown
# Brief: diagnosis-eval-baseline-diff
|
|
|
|
## Background
|
|
|
|
The eval harness now has a complete saved baseline. This change adds the comparison layer that turns the baseline into an actionable regression signal.
|
|
|
|
## Goals
|
|
|
|
1. Compare baseline and current `DiagnosisEvalReport` objects.
|
|
2. Detect aggregate and per-case regressions.
|
|
3. Output JSON and Markdown diff reports.
|
|
4. Document how to read the diff in interview and engineering terms.
|
|
|
|
## Scope
|
|
|
|
- Diff data structures
|
|
- Deterministic report comparison
|
|
- JSON / Markdown diff output
|
|
- Focused tests and eval docs
|
|
|
|
## Non-Goals
|
|
|
|
- No live Agent execution
|
|
- No LLM-as-judge
|
|
- No evaluator scoring rule changes
|
|
- No production API changes
|
|
|
|
## Related OpenSpec
|
|
|
|
`openspec/changes/diagnosis-eval-baseline-diff/`
|