1.2 KiB
1.2 KiB
1. Case Definitions
- 1.1 Add evaluation case definition format for fixed MVP diagnosis scenarios.
- 1.2 Add the first 5 case definitions: payment timeout, MySQL pool exhausted, Redis timeout, slow response, and JVM memory risk.
- 1.3 Document the meaning of expected keywords, required evidence tools, allowed verdicts, and forbidden behavior.
2. Trace Fixtures
- 2.1 Add fixture trace schema or DTOs that match
DiagnosisTraceResponseenough for offline evaluation. - 2.2 Add at least one representative trace fixture for a passing case.
- 2.3 Add at least one fixture covering low-confidence or degraded behavior.
3. Evaluator
- 3.1 Implement trace validation rules for evidence coverage, verifier verdict, answer keyword coverage, and degraded-output contract.
- 3.2 Implement aggregate metrics: pass rate, verdict distribution, average tool-call count, and average duration.
- 3.3 Implement JSON report output.
- 3.4 Implement Markdown report output.
4. Tests And Documentation
- 4.1 Add focused offline tests for the evaluator.
- 4.2 Add run instructions under
mvp/demoormvp/notes. - 4.3 Run targeted tests for the evaluator.
- 4.4 Run compile verification.