38 lines
1.6 KiB
Markdown
38 lines
1.6 KiB
Markdown
## Why
|
|
|
|
Executor Structured Output V2 已经完成 Executor、Gatekeeper、Verifier、Composer 四段主链路改造，但现有离线评测仍主要检查最终答案关键词、证据工具覆盖和 Verifier verdict。阶段 5 需要把新增的审计字段纳入固定回归门禁，证明 `Planner -> Executor -> Gatekeeper -> Verifier -> Composer -> final answer` 链路不是只在单次 demo 中可用。
|
|
|
|
## What Changes
|
|
|
|
- Extend the diagnosis eval harness so V2 audit fields are validated as first-class regression checks.
|
|
- Add fixture coverage for Gatekeeper failure, claim verification filtering, Composer fallback, and raw JSON leakage prevention.
|
|
- Update baseline reports to reflect the expanded fixed fixture set.
|
|
- Update eval documentation so another agent can understand the background, stages, data fields, and acceptance commands.
|
|
- No production protocol change is intended in this phase; production chain behavior should remain unchanged unless tests reveal a bug.
|
|
|
|
## Capabilities
|
|
|
|
### New Capabilities
|
|
|
|
- None.
|
|
|
|
### Modified Capabilities
|
|
|
|
- `diagnosis-eval-harness`: add V2 audit-closure requirements for Gatekeeper, Verifier claim checks, Composer output, and final-answer consistency.
|
|
|
|
## Impact
|
|
|
|
- Affected code:
|
|
- `src/main/java/com/superbiz/agent/eval/*`
|
|
- `src/test/java/com/superbiz/agent/eval/*`
|
|
- Affected data:
|
|
- `mvp/eval/cases/diagnosis-cases.json`
|
|
- `mvp/eval/fixtures/*.json`
|
|
- `mvp/eval/reports/baseline-report.*`
|
|
- `mvp/eval/schema.md`
|
|
- `mvp/eval/README.md`
|
|
- Verification:
|
|
- focused evaluator tests
|
|
- relevant Executor/Gatekeeper/Verifier/Composer integration tests
|
|
- OpenSpec validation and archive
|