feat(agent): add verifier claim checks

2026-07-08 02:33:02 +08:00
parent c5e496e715
commit 1b31e78be5
19 changed files with 1247 additions and 115 deletions
@@ -8,6 +8,7 @@
 | 2026-07-07 | executor-evidence-output-contract | Chat质量门禁/证据归因 | Executor structured output, evidence bindings, Verifier structured claims, LOW_CONFID, hallucination | openspec/changes/archive/2026-07-07-executor-evidence-output-contract | archived |
 | 2026-07-07 | executor-v2-output-contract | Chat质量门禁/证据归因 | executor_evidence_v2, user_facing_answer removal, diagnosis_summary removal, structured renderer | openspec/changes/archive/2026-07-07-executor-v2-output-contract | archived |
 | 2026-07-07 | executor-gatekeeper-hook | Chat质量门禁/证据归因 | Gatekeeper, verifier payload, source_invocation_ids, tool_name match, self_evaluation | openspec/changes/archive/2026-07-07-executor-gatekeeper-hook | archived |
+| 2026-07-07 | executor-verifier-claim-checks | Chat质量门禁/证据归因 | Verifier claim_checks, facts_checked compatibility, effective verdict guardrail, malformed output downgrade | openspec/changes/archive/2026-07-07-executor-verifier-claim-checks | archived |
 | 2026-07-06 | rag-eval-pipeline-closure | RAG/评测/回归闭环 | lookupResult fixture, LookupKnowledgeTool snapshot, evidenceBlocks, contextPack, retrievalTrace, rerankTrace, baseline diff, fallback case | devflow/projects/2026-07-06-rag-eval-pipeline-closure | archived |
 | 2026-07-06 | modular-rag-pipeline | RAG/Agent工具/证据链 | modular RAG, lookup_knowledge, evidenceBlocks, contextPack, rerank, retrievalTrace, L0 hint, unfiltered retry | openspec/changes/archive/2026-07-06-modular-rag-pipeline | archived |
 | 2026-07-05 | mvp-demo-interview-runbook | MVP Demo/Interview | Plan C, payment timeout, runbook, trace checklist, demo script | openspec/changes/archive/2026-07-05-mvp-demo-interview-runbook | archived |
@@ -0,0 +1,58 @@
+# Acceptance
+
+## Implementation Result
+
+Implemented stage three of Executor Structured Output V2:
+
+- Verifier prompt now validates claim derivability rather than scanning final natural-language output.
+- Verifier output supports `claim_checks`.
+- `ChatService` derives compatibility `facts_checked` from `claim_checks`.
+- `ChatService` persists both `claim_checks` and `facts_checked`.
+- Effective verdict guardrails prevent Gatekeeper failures and malformed structured output from remaining `PASS`.
+- Verifier logging summarizes `claim_checks`.
+
+## Static Verification
+
+- Reviewed `git diff --stat` and changed files are scoped to stage three implementation, tests, OpenSpec/devflow, and the issue handoff document.
+- `cmd /c openspec validate executor-verifier-claim-checks` passed.
+- After OpenSpec archive, `cmd /c openspec validate --specs` passed.
+
+## Script Verification
+
+Passed:
+
+```powershell
+mvn "-Dtest=ExecutorGatekeeperServiceTest,VerifierInputHookTest,ChatServiceSequentialAgentTest" test
+```
+
+Coverage:
+
+- Verifier payload and Gatekeeper hook behavior.
+- Gatekeeper schema/invocation validation.
+- `claim_checks` parsing and persistence.
+- `claim_checks` to `facts_checked` compatibility mapping.
+- all claim verification mapping classes.
+- Gatekeeper failure downgrade from model `PASS`.
+- malformed Executor output downgrade from model `PASS`.
+
+The targeted Maven test command was re-run after OpenSpec archive and passed.
+
+## Browser / Manual Verification
+
+Not run. This stage changes backend prompt, parser, audit, and tests only.
+
+## OpenSpec Archive Status
+
+Archived:
+
+```text
+openspec/changes/archive/2026-07-07-executor-verifier-claim-checks
+```
+
+Archive follow-up: the generated canonical `chat-verifier-agent` spec was reviewed and amended to preserve pre-existing verifier input and Gatekeeper schema scenarios while adding the new claim-check scenarios.
+
+## Remaining Risks
+
+- Composer is not implemented in this stage; final PASS rendering still uses the temporary V2 renderer until stage four.
+- Full eval fixture expansion is deferred to stage five.
+- Existing Maven warnings about duplicate test dependency and Lombok builder defaults remain outside this stage.
@@ -0,0 +1,41 @@
+# Executor Verifier Claim Checks
+
+## Background
+
+Stage one moved Chat Executor to `executor_evidence_v2`, and stage two added deterministic Gatekeeper checks before Verifier. After those stages, Verifier still primarily used the legacy `facts_checked` contract and could still treat `executor_final_answer` as a fact source.
+
+That left two risks:
+
+- Verifier could still extract extra confirmed facts from natural-language Executor output.
+- Downstream audit and retry consumers could not distinguish V2 claim-level verification from legacy fact checks.
+
+## Goal
+
+Make Verifier V2 claim-oriented:
+
+- verify `executor_structured_output.claims` as the primary target;
+- emit `claim_checks` as the authoritative V2 result;
+- keep `facts_checked` only as a compatibility projection;
+- enforce code-side guardrails so Gatekeeper failures or malformed structured output cannot remain effective `PASS`.
+
+## Scope
+
+- Updated `chat-verifier-prompt.md` to frame verification as claim derivability.
+- Extended `ChatService` to parse, normalize, map, and persist `claim_checks`.
+- Added effective verdict guardrails for Gatekeeper failure and malformed/missing Executor structured output.
+- Updated verifier logging summaries to count `claim_checks`.
+- Updated sequential workflow tests to cover claim mapping and downgrade behavior.
+
+## Non-Goals
+
+- No Composer integration in this phase.
+- No final-answer material filtering beyond existing templates and temporary V2 renderer.
+- No Executor retry behavior change.
+- No Gatekeeper rule expansion.
+- No database schema migration.
+
+## OpenSpec
+
+- Active change before archive: `openspec/changes/executor-verifier-claim-checks`
+- Capability: `chat-verifier-agent`
+- Scale: standard
@@ -0,0 +1,57 @@
+# Decisions
+
+## Scope Decision
+
+Stage three is limited to Verifier V2 claim checks. Composer is explicitly deferred to stage four.
+
+Reason: Composer requires stable verifier output and allowed-material filtering; mixing it into this stage would make rollback and acceptance unclear.
+
+## Contract Decision
+
+`claim_checks` is the authoritative V2 verifier output.
+
+`facts_checked` remains as a compatibility projection generated from `claim_checks` when present.
+
+Reason: existing low-confidence rendering, retry context, trace output, and evaluation code still depend on `facts_checked`.
+
+## Mapping Decision
+
+Claim verification maps to legacy facts as follows:
+
+| claim verification | legacy facts_checked verification |
+|---|---|
+| `direct_observation` | `direct_evidence` |
+| `reasonable_inference` | `indirect_support` |
+| `overstated` | `indirect_support` |
+| `unsupported` | `no_evidence` |
+| `external_unknown` | `no_evidence` |
+| `contradicted` | `contradicted` |
+
+## Guardrail Decision
+
+Effective verdict is enforced in code:
+
+- `gatekeeper_result.status=fail` cannot remain `PASS`.
+- `evidence.invocation_ref` failure downgrades to `REJECT`.
+- Other Gatekeeper failures downgrade at least to `LOW_CONFID`.
+- missing/malformed Executor structured output cannot remain `PASS`.
+
+Reason: prompt compliance is not deterministic enough for safety-critical evidence attribution.
+
+## Apply Fix Record
+
+Initial targeted Maven verification failed because older tests expected PASS to return Executor natural-language output or V1 `user_facing_answer`.
+
+Classification: test drift from the committed OpenSpec, not a design blocker.
+
+Resolution: update tests to use valid Executor V2 output for PASS paths and assert downgrade behavior for malformed or Gatekeeper-failed outputs.
+
+## Interface Impact
+
+L2 internal contract extension:
+
+- Verifier output gains `claim_checks`.
+- Existing `facts_checked` remains available.
+- Persistence JSON gains `claim_checks` under existing `self_evaluation`.
+
+No external API or database schema changes.
@@ -0,0 +1,35 @@
+# Evidence
+
+## Relevant History
+
+- `executor-v2-output-contract`: Executor emits `executor_evidence_v2` and no longer emits final-expression fields.
+- `executor-gatekeeper-hook`: Gatekeeper validates schema and invocation references before Verifier and persists `gatekeeper_result`.
+- `chat-verifier-agent`: Existing Verifier used `facts_checked`, low-confidence rendering, retry context, and verifier audit.
+
+## Code Evidence
+
+- `src/main/resources/prompts/chat-verifier-prompt.md`: Verifier prompt now makes `executor_structured_output.claims` primary and treats `executor_final_answer` as debug/fallback only.
+- `src/main/java/com/superbiz/agent/service/ChatService.java`: parses `claim_checks`, maps them to compatibility `facts_checked`, persists both, and applies effective verdict guardrails.
+- `src/main/java/com/superbiz/agent/hook/AgentLoggingHook.java`: verifier thought summaries now include `claim_checks`.
+- `src/test/java/com/superbiz/agent/service/ChatServiceSequentialAgentTest.java`: covers claim-check mapping, Gatekeeper downgrade, malformed output downgrade, and V2 PASS paths.
+
+## Evidence-Driven Conclusions
+
+- `facts_checked` cannot be removed yet because existing low-confidence templates, retry context, trace tooling, and eval paths still consume it.
+- Prompt-only prevention is insufficient for Gatekeeper failures; `ChatService` must enforce effective verdict downgrades in code.
+- No database schema migration is needed because `claim_checks` is persisted inside existing `diagnosis_session.self_evaluation`.
+- Composer remains stage four and must not be mixed into this stage.
+
+## Verification Evidence
+
+Script verification passed:
+
+```powershell
+mvn "-Dtest=ExecutorGatekeeperServiceTest,VerifierInputHookTest,ChatServiceSequentialAgentTest" test
+cmd /c openspec validate executor-verifier-claim-checks
+```
+
+Known existing warnings:
+
+- Maven reports duplicate `spring-boot-starter-test` dependency in `pom.xml`.
+- Existing Lombok `@Builder` default warnings remain.