26 lines
1.2 KiB
Markdown
26 lines
1.2 KiB
Markdown
## 1. Case Definitions
|
|
|
|
- [x] 1.1 Add evaluation case definition format for fixed MVP diagnosis scenarios.
|
|
- [x] 1.2 Add the first 5 case definitions: payment timeout, MySQL pool exhausted, Redis timeout, slow response, and JVM memory risk.
|
|
- [x] 1.3 Document the meaning of expected keywords, required evidence tools, allowed verdicts, and forbidden behavior.
|
|
|
|
## 2. Trace Fixtures
|
|
|
|
- [x] 2.1 Add fixture trace schema or DTOs that match `DiagnosisTraceResponse` enough for offline evaluation.
|
|
- [x] 2.2 Add at least one representative trace fixture for a passing case.
|
|
- [x] 2.3 Add at least one fixture covering low-confidence or degraded behavior.
|
|
|
|
## 3. Evaluator
|
|
|
|
- [x] 3.1 Implement trace validation rules for evidence coverage, verifier verdict, answer keyword coverage, and degraded-output contract.
|
|
- [x] 3.2 Implement aggregate metrics: pass rate, verdict distribution, average tool-call count, and average duration.
|
|
- [x] 3.3 Implement JSON report output.
|
|
- [x] 3.4 Implement Markdown report output.
|
|
|
|
## 4. Tests And Documentation
|
|
|
|
- [x] 4.1 Add focused offline tests for the evaluator.
|
|
- [x] 4.2 Add run instructions under `mvp/demo` or `mvp/notes`.
|
|
- [x] 4.3 Run targeted tests for the evaluator.
|
|
- [x] 4.4 Run compile verification.
|