SuperBizAgent-java/devflow/projects/2026-07-04-diagnosis-eval-harness/brief.md

Brief: diagnosis-eval-harness

Background

The MVP has a runnable demo and hardened evidence trace semantics, but it still lacks a fixed regression baseline for Agent diagnosis quality. P1-B creates a small evaluation harness that can validate diagnosis traces against fixed cases and produce repeatable reports.

Goals

1. Define fixed diagnosis cases for the MVP demo domain.
2. Validate trace evidence, verifier verdicts, answer keywords, and degraded-output behavior.
3. Produce JSON and Markdown reports for interview and regression use.
4. Keep the first version offline by supporting trace fixtures.

Scope

• Evaluation case definitions
• Trace fixture shape
• Rule-based evaluator
• JSON / Markdown report output
• Focused offline tests and docs

Non-Goals

• No LLM-as-judge
• No live end-to-end runtime requirement
• No production API
• No chat or verifier runtime change

Related OpenSpec

openspec/changes/diagnosis-eval-harness/