40 lines
2.1 KiB
Markdown
40 lines
2.1 KiB
Markdown
## 1. Prompt Contract
|
|
|
|
- [x] 1.1 Update `chat-verifier-prompt.md` so `executor_structured_output.claims` is the primary verification target.
|
|
- [x] 1.2 Remove the valid-structured-output path that scans `executor_final_answer` for extra confirmed facts.
|
|
- [x] 1.3 Add `claim_checks` and optional `hypothesis_checks` to the output contract.
|
|
- [x] 1.4 Keep `facts_checked` as compatibility output.
|
|
- [x] 1.5 State that missing/malformed structured output cannot produce PASS through natural-language fallback.
|
|
|
|
## 2. Parser And Compatibility Mapping
|
|
|
|
- [x] 2.1 Extend `VerifierDecision` to store `claim_checks`.
|
|
- [x] 2.2 Parse `claim_checks` from verifier output.
|
|
- [x] 2.3 Derive compatibility `facts_checked` from `claim_checks` when present.
|
|
- [x] 2.4 Preserve old `facts_checked` parsing when `claim_checks` is absent.
|
|
- [x] 2.5 Persist `claim_checks` in verifier evaluation.
|
|
|
|
## 3. Effective Verdict Guardrails
|
|
|
|
- [x] 3.1 Add code-side guard so `gatekeeper_result.status=fail` cannot result in effective PASS.
|
|
- [x] 3.2 Downgrade `evidence.invocation_ref` failures to REJECT.
|
|
- [x] 3.3 Downgrade other Gatekeeper failures to at least LOW_CONFID.
|
|
- [x] 3.4 Ensure missing/malformed Executor structured output cannot produce effective PASS.
|
|
|
|
## 4. Compatibility Consumers
|
|
|
|
- [x] 4.1 Ensure low-confidence rendering still uses compatibility `facts_checked`.
|
|
- [x] 4.2 Ensure `buildRetryContext(...)` still receives evidence gaps from compatibility `facts_checked`.
|
|
- [x] 4.3 Update verifier logging summary to account for `claim_checks`.
|
|
- [x] 4.4 Preserve existing traceability and Gatekeeper audit fields.
|
|
|
|
## 5. Tests And Verification
|
|
|
|
- [x] 5.1 Add or update tests for parsing `claim_checks`.
|
|
- [x] 5.2 Add tests for `claim_checks` to `facts_checked` compatibility mapping.
|
|
- [x] 5.3 Add tests for each verification mapping class: `direct_observation`, `reasonable_inference`, `overstated`, `unsupported`, `external_unknown`, `contradicted`.
|
|
- [x] 5.4 Add tests proving Gatekeeper fail cannot remain PASS.
|
|
- [x] 5.5 Add tests proving malformed/missing structured output cannot remain PASS.
|
|
- [x] 5.6 Run targeted tests.
|
|
- [x] 5.7 Validate this OpenSpec change.
|