SuperBizAgent-java/openspec/changes/archive/2026-07-08-executor-eval-audit-closure/proposal.md

Why

Executor Structured Output V2 已经完成 Executor、Gatekeeper、Verifier、Composer 四段主链路改造，但现有离线评测仍主要检查最终答案关键词、证据工具覆盖和 Verifier verdict。阶段 5 需要把新增的审计字段纳入固定回归门禁，证明 Planner -> Executor -> Gatekeeper -> Verifier -> Composer -> final answer 链路不是只在单次 demo 中可用。

What Changes

• Extend the diagnosis eval harness so V2 audit fields are validated as first-class regression checks.
• Add fixture coverage for Gatekeeper failure, claim verification filtering, Composer fallback, and raw JSON leakage prevention.
• Update baseline reports to reflect the expanded fixed fixture set.
• Update eval documentation so another agent can understand the background, stages, data fields, and acceptance commands.
• No production protocol change is intended in this phase; production chain behavior should remain unchanged unless tests reveal a bug.

Capabilities

New Capabilities

• None.

Modified Capabilities

• diagnosis-eval-harness: add V2 audit-closure requirements for Gatekeeper, Verifier claim checks, Composer output, and final-answer consistency.

Impact

• Affected code:
  • src/main/java/com/superbiz/agent/eval/*
  • src/test/java/com/superbiz/agent/eval/*
• Affected data:
  • mvp/eval/cases/diagnosis-cases.json
  • mvp/eval/fixtures/*.json
  • mvp/eval/reports/baseline-report.*
  • mvp/eval/schema.md
  • mvp/eval/README.md
• Verification:
  • focused evaluator tests
  • relevant Executor/Gatekeeper/Verifier/Composer integration tests
  • OpenSpec validation and archive