feat(agent): add verifier claim checks
This commit is contained in:
aruo
@@ -0,0 +1,58 @@
|
||||
# Acceptance
|
||||
|
||||
## Implementation Result
|
||||
|
||||
Implemented stage three of Executor Structured Output V2:
|
||||
|
||||
- Verifier prompt now validates claim derivability rather than scanning final natural-language output.
|
||||
- Verifier output supports `claim_checks`.
|
||||
- `ChatService` derives compatibility `facts_checked` from `claim_checks`.
|
||||
- `ChatService` persists both `claim_checks` and `facts_checked`.
|
||||
- Effective verdict guardrails prevent Gatekeeper failures and malformed structured output from remaining `PASS`.
|
||||
- Verifier logging summarizes `claim_checks`.
|
||||
|
||||
## Static Verification
|
||||
|
||||
- Reviewed `git diff --stat` and changed files are scoped to stage three implementation, tests, OpenSpec/devflow, and the issue handoff document.
|
||||
- `cmd /c openspec validate executor-verifier-claim-checks` passed.
|
||||
- After OpenSpec archive, `cmd /c openspec validate --specs` passed.
|
||||
|
||||
## Script Verification
|
||||
|
||||
Passed:
|
||||
|
||||
```powershell
|
||||
mvn "-Dtest=ExecutorGatekeeperServiceTest,VerifierInputHookTest,ChatServiceSequentialAgentTest" test
|
||||
```
|
||||
|
||||
Coverage:
|
||||
|
||||
- Verifier payload and Gatekeeper hook behavior.
|
||||
- Gatekeeper schema/invocation validation.
|
||||
- `claim_checks` parsing and persistence.
|
||||
- `claim_checks` to `facts_checked` compatibility mapping.
|
||||
- all claim verification mapping classes.
|
||||
- Gatekeeper failure downgrade from model `PASS`.
|
||||
- malformed Executor output downgrade from model `PASS`.
|
||||
|
||||
The targeted Maven test command was re-run after OpenSpec archive and passed.
|
||||
|
||||
## Browser / Manual Verification
|
||||
|
||||
Not run. This stage changes backend prompt, parser, audit, and tests only.
|
||||
|
||||
## OpenSpec Archive Status
|
||||
|
||||
Archived:
|
||||
|
||||
```text
|
||||
openspec/changes/archive/2026-07-07-executor-verifier-claim-checks
|
||||
```
|
||||
|
||||
Archive follow-up: the generated canonical `chat-verifier-agent` spec was reviewed and amended to preserve pre-existing verifier input and Gatekeeper schema scenarios while adding the new claim-check scenarios.
|
||||
|
||||
## Remaining Risks
|
||||
|
||||
- Composer is not implemented in this stage; final PASS rendering still uses the temporary V2 renderer until stage four.
|
||||
- Full eval fixture expansion is deferred to stage five.
|
||||
- Existing Maven warnings about duplicate test dependency and Lombok builder defaults remain outside this stage.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Executor Verifier Claim Checks
|
||||
|
||||
## Background
|
||||
|
||||
Stage one moved Chat Executor to `executor_evidence_v2`, and stage two added deterministic Gatekeeper checks before Verifier. After those stages, Verifier still primarily used the legacy `facts_checked` contract and could still treat `executor_final_answer` as a fact source.
|
||||
|
||||
That left two risks:
|
||||
|
||||
- Verifier could still extract extra confirmed facts from natural-language Executor output.
|
||||
- Downstream audit and retry consumers could not distinguish V2 claim-level verification from legacy fact checks.
|
||||
|
||||
## Goal
|
||||
|
||||
Make Verifier V2 claim-oriented:
|
||||
|
||||
- verify `executor_structured_output.claims` as the primary target;
|
||||
- emit `claim_checks` as the authoritative V2 result;
|
||||
- keep `facts_checked` only as a compatibility projection;
|
||||
- enforce code-side guardrails so Gatekeeper failures or malformed structured output cannot remain effective `PASS`.
|
||||
|
||||
## Scope
|
||||
|
||||
- Updated `chat-verifier-prompt.md` to frame verification as claim derivability.
|
||||
- Extended `ChatService` to parse, normalize, map, and persist `claim_checks`.
|
||||
- Added effective verdict guardrails for Gatekeeper failure and malformed/missing Executor structured output.
|
||||
- Updated verifier logging summaries to count `claim_checks`.
|
||||
- Updated sequential workflow tests to cover claim mapping and downgrade behavior.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No Composer integration in this phase.
|
||||
- No final-answer material filtering beyond existing templates and temporary V2 renderer.
|
||||
- No Executor retry behavior change.
|
||||
- No Gatekeeper rule expansion.
|
||||
- No database schema migration.
|
||||
|
||||
## OpenSpec
|
||||
|
||||
- Active change before archive: `openspec/changes/executor-verifier-claim-checks`
|
||||
- Capability: `chat-verifier-agent`
|
||||
- Scale: standard
|
||||
@@ -0,0 +1,57 @@
|
||||
# Decisions
|
||||
|
||||
## Scope Decision
|
||||
|
||||
Stage three is limited to Verifier V2 claim checks. Composer is explicitly deferred to stage four.
|
||||
|
||||
Reason: Composer requires stable verifier output and allowed-material filtering; mixing it into this stage would make rollback and acceptance unclear.
|
||||
|
||||
## Contract Decision
|
||||
|
||||
`claim_checks` is the authoritative V2 verifier output.
|
||||
|
||||
`facts_checked` remains as a compatibility projection generated from `claim_checks` when present.
|
||||
|
||||
Reason: existing low-confidence rendering, retry context, trace output, and evaluation code still depend on `facts_checked`.
|
||||
|
||||
## Mapping Decision
|
||||
|
||||
Claim verification maps to legacy facts as follows:
|
||||
|
||||
| claim verification | legacy facts_checked verification |
|
||||
|---|---|
|
||||
| `direct_observation` | `direct_evidence` |
|
||||
| `reasonable_inference` | `indirect_support` |
|
||||
| `overstated` | `indirect_support` |
|
||||
| `unsupported` | `no_evidence` |
|
||||
| `external_unknown` | `no_evidence` |
|
||||
| `contradicted` | `contradicted` |
|
||||
|
||||
## Guardrail Decision
|
||||
|
||||
Effective verdict is enforced in code:
|
||||
|
||||
- `gatekeeper_result.status=fail` cannot remain `PASS`.
|
||||
- `evidence.invocation_ref` failure downgrades to `REJECT`.
|
||||
- Other Gatekeeper failures downgrade at least to `LOW_CONFID`.
|
||||
- missing/malformed Executor structured output cannot remain `PASS`.
|
||||
|
||||
Reason: prompt compliance is not deterministic enough for safety-critical evidence attribution.
|
||||
|
||||
## Apply Fix Record
|
||||
|
||||
Initial targeted Maven verification failed because older tests expected PASS to return Executor natural-language output or V1 `user_facing_answer`.
|
||||
|
||||
Classification: test drift from the committed OpenSpec, not a design blocker.
|
||||
|
||||
Resolution: update tests to use valid Executor V2 output for PASS paths and assert downgrade behavior for malformed or Gatekeeper-failed outputs.
|
||||
|
||||
## Interface Impact
|
||||
|
||||
L2 internal contract extension:
|
||||
|
||||
- Verifier output gains `claim_checks`.
|
||||
- Existing `facts_checked` remains available.
|
||||
- Persistence JSON gains `claim_checks` under existing `self_evaluation`.
|
||||
|
||||
No external API or database schema changes.
|
||||
@@ -0,0 +1,35 @@
|
||||
# Evidence
|
||||
|
||||
## Relevant History
|
||||
|
||||
- `executor-v2-output-contract`: Executor emits `executor_evidence_v2` and no longer emits final-expression fields.
|
||||
- `executor-gatekeeper-hook`: Gatekeeper validates schema and invocation references before Verifier and persists `gatekeeper_result`.
|
||||
- `chat-verifier-agent`: Existing Verifier used `facts_checked`, low-confidence rendering, retry context, and verifier audit.
|
||||
|
||||
## Code Evidence
|
||||
|
||||
- `src/main/resources/prompts/chat-verifier-prompt.md`: Verifier prompt now makes `executor_structured_output.claims` primary and treats `executor_final_answer` as debug/fallback only.
|
||||
- `src/main/java/com/superbiz/agent/service/ChatService.java`: parses `claim_checks`, maps them to compatibility `facts_checked`, persists both, and applies effective verdict guardrails.
|
||||
- `src/main/java/com/superbiz/agent/hook/AgentLoggingHook.java`: verifier thought summaries now include `claim_checks`.
|
||||
- `src/test/java/com/superbiz/agent/service/ChatServiceSequentialAgentTest.java`: covers claim-check mapping, Gatekeeper downgrade, malformed output downgrade, and V2 PASS paths.
|
||||
|
||||
## Evidence-Driven Conclusions
|
||||
|
||||
- `facts_checked` cannot be removed yet because existing low-confidence templates, retry context, trace tooling, and eval paths still consume it.
|
||||
- Prompt-only prevention is insufficient for Gatekeeper failures; `ChatService` must enforce effective verdict downgrades in code.
|
||||
- No database schema migration is needed because `claim_checks` is persisted inside existing `diagnosis_session.self_evaluation`.
|
||||
- Composer remains stage four and must not be mixed into this stage.
|
||||
|
||||
## Verification Evidence
|
||||
|
||||
Script verification passed:
|
||||
|
||||
```powershell
|
||||
mvn "-Dtest=ExecutorGatekeeperServiceTest,VerifierInputHookTest,ChatServiceSequentialAgentTest" test
|
||||
cmd /c openspec validate executor-verifier-claim-checks
|
||||
```
|
||||
|
||||
Known existing warnings:
|
||||
|
||||
- Maven reports duplicate `spring-boot-starter-test` dependency in `pom.xml`.
|
||||
- Existing Lombok `@Builder` default warnings remain.
|
||||
Reference in New Issue
Block a user