Add diagnosis eval baseline diff

2026-07-05 00:59:53 +08:00
parent 4c7c53b024
commit 69deb15330
22 changed files with 1069 additions and 0 deletions
@@ -0,0 +1,28 @@
+# Diagnosis Eval Baseline Diff Decisions
+
+## Clarify
+
+- Entry summary: add report diffing on top of the completed diagnosis eval baseline.
+- Slug: `diagnosis-eval-baseline-diff`
+- Devflow scale: standard-light
+
+## Context
+
+- `diagnosis-eval-harness` created deterministic fixture evaluation.
+- `expand-diagnosis-eval-fixtures` created a complete saved baseline.
+- This change compares new reports against that baseline.
+
+## Key Decisions
+
+- Decision: Diff report DTOs instead of raw traces.
+  - Reason: the report is the stable contract for regression review.
+
+- Decision: Use deterministic code rules instead of LLM-as-judge.
+  - Reason: baseline regression checks should be repeatable and explainable.
+
+- Decision: Output both JSON and Markdown.
+  - Reason: JSON supports automation; Markdown is useful in reviews and interviews.
+
+## Open Questions
+
+- Whether a future change should expose this through a CLI or Maven goal.