SuperBizAgent-java/openspec/changes/archive/2026-07-07-executor-verifier-claim-checks/specs/chat-verifier-agent/spec.md

MODIFIED Requirements

Requirement: Verifier SHALL fact-check Executor answers

The system SHALL have a Verifier Agent that reads structured Executor claims and the tool call history, then produces a structured verdict based on claim derivability.

Scenario: PASS verdict when all claims have evidence

• WHEN all critical claims in executor_structured_output.claims have direct observation or reasonable inference support in tool call results
• AND at least one critical claim has direct observation
• AND no critical claim is contradicted, unsupported, external unknown, or overstated
• AND gatekeeper_result.status is not fail
• THEN the Verifier MAY output verdict="PASS" with groundedness_score ≥ 0.5

Scenario: LOW_CONFID verdict with partial evidence

• WHEN no critical claim contradicts the tool results
• AND some critical claims are unsupported, external_unknown, or overstated
• THEN the Verifier SHALL output verdict="LOW_CONFID"

Scenario: LOW_CONFID verdict with only inference support

• WHEN no critical claim contradicts the tool results
• AND all critical claims are only reasonable_inference
• THEN the Verifier SHALL output verdict="LOW_CONFID"

Scenario: REJECT verdict when claims contradict evidence

• WHEN any critical claim in executor_structured_output.claims contradicts tool call results
• OR the claim fabricates a key entity, error code, or conclusion that does not exist in the tool evidence
• THEN the Verifier SHALL output verdict="REJECT"

Scenario: Structured Executor claims are verified first

• WHEN executor_structured_output.claims is present and valid
• THEN Verifier SHALL verify each structured claim against tool_trace_summary through claim_checks
• AND each claim's evidence bindings SHALL reference existing trace or invocation identifiers when those identifiers are available
• AND a claim with fabricated or missing evidence references SHALL NOT be classified as direct_observation
• AND Verifier SHALL NOT add extra confirmed facts from executor_final_answer that are absent from executor_structured_output.claims

Scenario: Malformed structured output cannot pass through natural language fallback

• WHEN Executor does not return parseable structured output
• THEN Verifier SHALL NOT produce an effective PASS by extracting facts from executor_final_answer
• AND the effective verdict SHALL be LOW_CONFID

Requirement: Verifier SHALL output structured JSON

The Verifier SHALL output a JSON object with verdict, groundedness_score, claim_checks array, compatibility facts_checked array, and rationale.

Scenario: claim-level verifier output is accepted

• WHEN the Verifier checks Executor V2 structured output
• THEN the output SHALL contain verdict, groundedness_score, critical_fact_count, claim_checks, facts_checked, and rationale
• AND claim_checks SHALL be the primary V2 verification result
• AND facts_checked SHALL remain available for compatibility

Requirement: facts_checked SHALL use a fixed classification set

The system SHALL continue to expose legacy facts_checked using its fixed verification classification set.

Scenario: claim checks are mapped to legacy facts

• WHEN Verifier output contains claim_checks
• THEN ChatService SHALL derive compatibility facts_checked
• AND direct_observation SHALL map to direct_evidence
• AND reasonable_inference and overstated SHALL map to indirect_support
• AND unsupported and external_unknown SHALL map to no_evidence
• AND contradicted SHALL map to contradicted

Requirement: Verifier SHALL be observable

The Verifier's verdict SHALL be persisted for observability.

Scenario: claim checks written to self_evaluation

• WHEN the Verifier evaluation is persisted
• THEN diagnosis_session.self_evaluation.verifier_evaluation SHALL include claim_checks
• AND it SHALL continue to include compatibility facts_checked
• AND existing fields such as verdict, groundedness_score, rationale, executor_output_parse_status, tool_trace_summary, and gatekeeper_result SHALL be preserved

Requirement: Verifier SHALL consume explicit verification inputs

The Verifier SHALL receive explicit verification inputs rather than inferring them only from raw conversation history.

Scenario: structured claims are the primary verification target

• WHEN executor_output_parse_status.status is valid
• AND executor_structured_output.claims is available
• THEN Verifier SHALL verify each claim through claim_checks
• AND Verifier SHALL NOT add extra confirmed facts from executor_final_answer that are absent from executor_structured_output.claims

Scenario: malformed structured output cannot pass through natural language fallback

• WHEN executor_output_parse_status.status is missing or malformed
• THEN Verifier SHALL NOT produce an effective PASS by extracting facts from executor_final_answer
• AND the effective verdict SHALL be LOW_CONFID

Requirement: Executor Gatekeeper SHALL validate deterministic structured-output failures

The system SHALL run deterministic Gatekeeper checks after Executor output parsing and before Verifier model execution.

Scenario: gatekeeper fail prevents PASS

• WHEN gatekeeper_result.status is fail
• AND the Verifier model returns verdict = "PASS"
• THEN ChatService SHALL downgrade the effective verdict
• AND the effective verdict SHALL NOT be PASS

Scenario: invocation reference failure downgrades to reject

• WHEN gatekeeper_result.failed_rules contains evidence.invocation_ref
• AND the Verifier model returns verdict = "PASS"
• THEN ChatService SHALL set the effective verdict to REJECT

ADDED Requirements

Requirement: Verifier claim checks SHALL use a fixed derivability classification set

The Verifier SHALL classify each structured claim using a fixed derivability classification set.

Scenario: claim check verification values are constrained

• WHEN Verifier emits claim_checks
• THEN each item SHALL use one of direct_observation, reasonable_inference, overstated, unsupported, external_unknown, or contradicted

Scenario: claim check evidence references remain auditable

• WHEN Verifier emits claim_checks
• THEN each claim check SHALL include claim_id, verification, detail, and evidence_refs
• AND every evidence ref SHALL preserve available trace_ref, tool_name, and source_invocation_ids