feat(eval): add executor audit closure checks

2026-07-08 10:21:39 +08:00
parent 6015bcbf6f
commit a08672b31e
20 changed files with 1142 additions and 188 deletions
@@ -0,0 +1,37 @@
+## Why
+
+Executor Structured Output V2 已经完成 Executor、Gatekeeper、Verifier、Composer 四段主链路改造，但现有离线评测仍主要检查最终答案关键词、证据工具覆盖和 Verifier verdict。阶段 5 需要把新增的审计字段纳入固定回归门禁，证明 `Planner -> Executor -> Gatekeeper -> Verifier -> Composer -> final answer` 链路不是只在单次 demo 中可用。
+
+## What Changes
+
+- Extend the diagnosis eval harness so V2 audit fields are validated as first-class regression checks.
+- Add fixture coverage for Gatekeeper failure, claim verification filtering, Composer fallback, and raw JSON leakage prevention.
+- Update baseline reports to reflect the expanded fixed fixture set.
+- Update eval documentation so another agent can understand the background, stages, data fields, and acceptance commands.
+- No production protocol change is intended in this phase; production chain behavior should remain unchanged unless tests reveal a bug.
+
+## Capabilities
+
+### New Capabilities
+
+- None.
+
+### Modified Capabilities
+
+- `diagnosis-eval-harness`: add V2 audit-closure requirements for Gatekeeper, Verifier claim checks, Composer output, and final-answer consistency.
+
+## Impact
+
+- Affected code:
+  - `src/main/java/com/superbiz/agent/eval/*`
+  - `src/test/java/com/superbiz/agent/eval/*`
+- Affected data:
+  - `mvp/eval/cases/diagnosis-cases.json`
+  - `mvp/eval/fixtures/*.json`
+  - `mvp/eval/reports/baseline-report.*`
+  - `mvp/eval/schema.md`
+  - `mvp/eval/README.md`
+- Verification:
+  - focused evaluator tests
+  - relevant Executor/Gatekeeper/Verifier/Composer integration tests
+  - OpenSpec validation and archive