SuperBizAgent-java/devflow/projects/2026-07-04-diagnosis-eval-harness/decisions.md

Diagnosis Eval Harness Decisions

Clarify

• Entry summary: build P1-B fixed case evaluation after evidence trace hardening.
• Slug: diagnosis-eval-harness
• Devflow scale: standard-light

Context

• P1-A evidence-trace-hardening created stable evidence semantics for supported, no-evidence, deduped, and failed tool calls.
• The MVP demo trace API already provides an aggregate trace shape suitable for evaluation.
• The first evaluator should avoid depending on external infrastructure so it can run in regular development.

Key Decisions

• Decision: Start with rule-based trace validation instead of LLM-as-judge.

  • Reason: The first regression signal should be deterministic and tied to trace contracts.

• Decision: Support offline fixture traces first.

  • Reason: This makes the harness usable without MySQL, Redis, Milvus, or a real LLM.

• Decision: Output both JSON and Markdown.

  • Reason: JSON supports automation; Markdown is easier to discuss in interviews.

Open Questions

• Whether live trace API polling belongs in this change or a follow-up after fixture mode lands.