1.3 KiB
1.3 KiB
Diagnosis Eval Harness
This folder contains the first fixed-case evaluation set for the MVP diagnosis Agent.
Scope
- Case definitions:
cases/diagnosis-cases.json - Offline trace fixtures:
fixtures/*.json - Field definitions:
schema.md - Baseline reports:
reports/baseline-report.jsonandreports/baseline-report.md - Evaluator implementation:
DiagnosisTraceEvaluator - Report writer:
DiagnosisEvalReportWriter
Current Mode
The first version evaluates saved trace fixtures. It does not start the application and does not require MySQL, Redis, Milvus, or a real LLM.
Verification
Run the focused evaluator test:
mvn -q "-Dtest=DiagnosisTraceEvaluatorTest" test
The committed baseline report represents the current fixed fixture set:
5 fixed cases
5 passing fixture evaluations
2 PASS verdicts
3 LOW_CONFID verdicts
When fixtures or evaluator rules change, regenerate the report from the same case file and fixture directory, then update both JSON and Markdown outputs together.
Interview Story
The harness gives the MVP a repeatable baseline:
fixed diagnosis case
-> saved or runtime trace
-> rule-based trace validation
-> JSON / Markdown report
-> regression signal for prompts, tools, retrieval, and verifier behavior