SuperBizAgent-java/devflow/projects/2026-07-05-diagnosis-eval-baseline-diff/decisions.md

Diagnosis Eval Baseline Diff Decisions

Clarify

• Entry summary: add report diffing on top of the completed diagnosis eval baseline.
• Slug: diagnosis-eval-baseline-diff
• Devflow scale: standard-light

Context

• diagnosis-eval-harness created deterministic fixture evaluation.
• expand-diagnosis-eval-fixtures created a complete saved baseline.
• This change compares new reports against that baseline.

Key Decisions

• Decision: Diff report DTOs instead of raw traces.

  • Reason: the report is the stable contract for regression review.

• Decision: Use deterministic code rules instead of LLM-as-judge.

  • Reason: baseline regression checks should be repeatable and explainable.

• Decision: Output both JSON and Markdown.

  • Reason: JSON supports automation; Markdown is useful in reviews and interviews.

Open Questions

• Whether a future change should expose this through a CLI or Maven goal.