Add diagnosis eval harness

2026-07-04 23:51:43 +08:00
parent dc6cd32a67
commit ca5c61fabf
24 changed files with 1251 additions and 2 deletions
@@ -0,0 +1,25 @@
+## 1. Case Definitions
+
+- [x] 1.1 Add evaluation case definition format for fixed MVP diagnosis scenarios.
+- [x] 1.2 Add the first 5 case definitions: payment timeout, MySQL pool exhausted, Redis timeout, slow response, and JVM memory risk.
+- [x] 1.3 Document the meaning of expected keywords, required evidence tools, allowed verdicts, and forbidden behavior.
+
+## 2. Trace Fixtures
+
+- [x] 2.1 Add fixture trace schema or DTOs that match `DiagnosisTraceResponse` enough for offline evaluation.
+- [x] 2.2 Add at least one representative trace fixture for a passing case.
+- [x] 2.3 Add at least one fixture covering low-confidence or degraded behavior.
+
+## 3. Evaluator
+
+- [x] 3.1 Implement trace validation rules for evidence coverage, verifier verdict, answer keyword coverage, and degraded-output contract.
+- [x] 3.2 Implement aggregate metrics: pass rate, verdict distribution, average tool-call count, and average duration.
+- [x] 3.3 Implement JSON report output.
+- [x] 3.4 Implement Markdown report output.
+
+## 4. Tests And Documentation
+
+- [x] 4.1 Add focused offline tests for the evaluator.
+- [x] 4.2 Add run instructions under `mvp/demo` or `mvp/notes`.
+- [x] 4.3 Run targeted tests for the evaluator.
+- [x] 4.4 Run compile verification.