SuperBizAgent-java/openspec/changes/archive/2026-07-04-diagnosis-eval-harness/tasks.md

1. Case Definitions

• 1.1 Add evaluation case definition format for fixed MVP diagnosis scenarios.
• 1.2 Add the first 5 case definitions: payment timeout, MySQL pool exhausted, Redis timeout, slow response, and JVM memory risk.
• 1.3 Document the meaning of expected keywords, required evidence tools, allowed verdicts, and forbidden behavior.

2. Trace Fixtures

• 2.1 Add fixture trace schema or DTOs that match DiagnosisTraceResponse enough for offline evaluation.
• 2.2 Add at least one representative trace fixture for a passing case.
• 2.3 Add at least one fixture covering low-confidence or degraded behavior.

3. Evaluator

• 3.1 Implement trace validation rules for evidence coverage, verifier verdict, answer keyword coverage, and degraded-output contract.
• 3.2 Implement aggregate metrics: pass rate, verdict distribution, average tool-call count, and average duration.
• 3.3 Implement JSON report output.
• 3.4 Implement Markdown report output.

4. Tests And Documentation

• 4.1 Add focused offline tests for the evaluator.
• 4.2 Add run instructions under mvp/demo or mvp/notes.
• 4.3 Run targeted tests for the evaluator.
• 4.4 Run compile verification.