Expand diagnosis eval fixtures

2026-07-05 00:27:57 +08:00
parent ca5c61fabf
commit 4c7c53b024
21 changed files with 622 additions and 11 deletions
@@ -0,0 +1,37 @@
+# Acceptance: expand-diagnosis-eval-fixtures
+
+## Classification
+
+standard-light
+
+## Task Status
+
+| Task | Status | Notes |
+| --- | --- | --- |
+| Issue and OpenSpec setup | Done | Created slug-based issue and initial OpenSpec artifacts. |
+| Implementation | Done | Added remaining fixtures, full baseline reports, and documentation updates. |
+| Verification | Done | Evaluator tests, compile verification, and OpenSpec validation passed. |
+
+## Current State
+
+- Fixture coverage is complete for the five fixed diagnosis cases.
+- Baseline reports are saved under `mvp/eval/reports`.
+- No production runtime behavior has been changed.
+
+## Verification
+
+### Script Verification
+
+- Command: `mvn -q "-Dtest=DiagnosisTraceEvaluatorTest" test`
+- Result: passed
+- Notes: Covers full fixture coverage, baseline report matching, reject degraded-output validation, and report writing.
+
+### Static Verification
+
+- Command: `mvn -q -DskipTests compile`
+- Result: passed
+
+### OpenSpec Verification
+
+- Command: `openspec validate expand-diagnosis-eval-fixtures --strict`
+- Result: passed
@@ -0,0 +1,31 @@
+# Brief: expand-diagnosis-eval-fixtures
+
+## Background
+
+The diagnosis eval harness is implemented and archived, but the fixed baseline is incomplete because three of the five diagnosis cases still reference missing fixtures.
+
+## Goals
+
+1. Add representative trace fixtures for all remaining fixed diagnosis cases.
+2. Save a reproducible baseline report in JSON and Markdown.
+3. Document how to regenerate and interpret the baseline.
+4. Keep evaluation offline and deterministic.
+
+## Scope
+
+- Redis timeout fixture
+- Slow response fixture
+- JVM memory risk fixture
+- Baseline reports under `mvp/eval/reports`
+- Focused tests for full fixture coverage and report generation
+
+## Non-Goals
+
+- No new diagnosis cases
+- No production Agent runtime changes
+- No LLM-as-judge
+- No live infrastructure requirement
+
+## Related OpenSpec
+
+`openspec/changes/expand-diagnosis-eval-fixtures/`
@@ -0,0 +1,28 @@
+# Expand Diagnosis Eval Fixtures Decisions
+
+## Clarify
+
+- Entry summary: complete the fixed diagnosis eval baseline after the harness is in place.
+- Slug: `expand-diagnosis-eval-fixtures`
+- Devflow scale: standard-light
+
+## Context
+
+- `diagnosis-eval-harness` created the evaluator, case file, fixture mode, and report writer.
+- The first baseline still has missing fixtures by design.
+- This follow-up turns that partial baseline into a full fixed-case baseline.
+
+## Key Decisions
+
+- Decision: Keep this change data-focused.
+  - Reason: the evaluator rules already landed; this change should not blur fixture expansion with harness behavior changes.
+
+- Decision: Save baseline reports in the repository.
+  - Reason: interview review and future diffs are easier when the expected baseline is visible.
+
+- Decision: Use deterministic fixture traces instead of live trace generation.
+  - Reason: this baseline should run without infrastructure or external model calls.
+
+## Open Questions
+
+- Whether a future change should add a CLI or Maven goal for report regeneration.
@@ -0,0 +1,11 @@
+# Evidence: expand-diagnosis-eval-fixtures
+
+## Evidence Log
+
+- 2026-07-04: Created slug-based issue `expand-diagnosis-eval-fixtures.md`.
+- 2026-07-04: Created OpenSpec change `expand-diagnosis-eval-fixtures`.
+- 2026-07-04: Added Redis timeout, slow response, and JVM memory risk fixtures.
+- 2026-07-04: Added baseline JSON and Markdown reports under `mvp/eval/reports`.
+- 2026-07-04: Verification passed with `mvn -q "-Dtest=DiagnosisTraceEvaluatorTest" test`.
+- 2026-07-04: Verification passed with `mvn -q -DskipTests compile`.
+- 2026-07-04: Verification passed with `openspec validate expand-diagnosis-eval-fixtures --strict`.