SuperBizAgent-java/openspec/specs/chat-verifier-agent/spec.md

chat-verifier-agent Specification

Purpose

TBD - created by archiving change chat-verifier-agent. Update Purpose after archive.

Requirements

Requirement: Verifier SHALL fact-check Executor answers

The system SHALL have a Verifier Agent that reads only Gatekeeper-projected structured Executor claims and verified claim-local evidence, then produces a structured verdict based on claim derivability.

Scenario: PASS verdict when all claims have evidence

• WHEN all critical claims in verified_executor_output.claims have direct observation or reasonable inference support in verified_evidence
• AND at least one critical claim has direct observation
• AND no critical claim is contradicted, unsupported, external unknown, or overstated
• AND verdict ceiling is PASS
• THEN the Verifier MAY output model verdict="PASS" with groundedness_score ≥ 0.5

Scenario: LOW_CONFID verdict with partial evidence

• WHEN no critical claim contradicts verified evidence
• AND some critical claims are unsupported, external_unknown, or overstated
• THEN the Verifier SHALL output model verdict="LOW_CONFID"

Scenario: LOW_CONFID verdict with only inference support

• WHEN no critical claim contradicts verified evidence
• AND all critical claims are only reasonable_inference
• THEN the Verifier SHALL output model verdict="LOW_CONFID"

Scenario: REJECT verdict when claims contradict evidence

• WHEN any critical claim in verified_executor_output.claims contradicts verified evidence
• OR the claim fabricates a key entity, error code, or conclusion that does not exist in verified evidence
• THEN the Verifier SHALL output model verdict="REJECT"

Scenario: Verified structured claims are the only verification target

• WHEN verified_executor_output.claims is present
• THEN Verifier SHALL verify each structured claim against matching verified_evidence through claim_checks
• AND each claim's evidence references SHALL match existing claim/invocation/tool/path identifiers when available
• AND a claim without matching verified evidence SHALL NOT be classified as direct_observation
• AND Verifier SHALL NOT receive or add confirmed facts from raw Executor text

Scenario: Executor output is invalid

• WHEN Executor does not return a legal structured contract
• THEN the Graph SHALL route directly to pre-verification Fallback
• AND Verifier SHALL NOT execute or fabricate a diagnostic verdict

Requirement: Verifier SHALL output structured JSON

The Verifier SHALL output a JSON object with verdict, groundedness_score, claim_checks array, compatibility facts_checked array, and rationale.

Scenario: claim-level verifier output is accepted

• WHEN the Verifier checks Executor V2 structured output
• THEN the output SHALL contain verdict, groundedness_score, critical_fact_count, claim_checks, facts_checked, and rationale
• AND claim_checks SHALL be the primary V2 verification result
• AND facts_checked SHALL remain available for compatibility

Scenario: Output format validation

• WHEN the Verifier completes its analysis
• THEN the output SHALL contain "verdict", "groundedness_score", "claim_checks", "facts_checked", and "rationale" fields
• AND groundedness_score SHALL be a float between 0.0 and 1.0
• AND verdict SHALL be one of "PASS", "LOW_CONFID", or "REJECT"

Scenario: strict schema output

• WHEN the Verifier returns its result
• THEN it SHALL output exactly one JSON object
• AND it SHALL NOT output Markdown, code fences, or explanatory text outside the JSON object
• AND the JSON object SHALL include critical_fact_count
• AND each claim_checks item SHALL include claim_id, verification, detail, and evidence_refs
• AND each facts_checked item SHALL include fact, is_critical, verification, and detail

Requirement: facts_checked SHALL use a fixed classification set

The system SHALL continue to expose compatibility facts_checked using its fixed verification classification set.

Scenario: claim checks are mapped to legacy facts

• WHEN Verifier output contains claim_checks
• THEN the shared Verifier protocol parser SHALL derive compatibility facts_checked when the model did not provide them
• AND direct_observation SHALL map to direct_evidence
• AND reasonable_inference and overstated SHALL map to indirect_support
• AND unsupported and external_unknown SHALL map to no_evidence
• AND contradicted SHALL map to contradicted

Requirement: groundedness_score SHALL be derived from fact classifications

The groundedness score SHALL be computed from critical fact classifications instead of being freely chosen by the model.

Scenario: contradicted fact forces reject

• WHEN any critical fact is labeled contradicted
• THEN the Verifier SHALL output verdict="REJECT"
• AND groundedness_score SHALL be 0.0

Scenario: score derived from supported facts

• WHEN no critical fact is contradicted
• THEN groundedness_score SHALL be computed from the mapped values of critical facts
• AND the implementation SHALL use the fixed mapping direct_evidence=1.0, indirect_support=0.6, no_evidence=0.0
• AND the result SHALL be clamped into [0.0, 1.0]

Requirement: ChatService SHALL route based on Verifier verdict

The system SHALL use Diagnosis StateGraph conditional edges, rather than a ChatService outer loop, to route explicit Verifier execution status and effective verdict.

Scenario: PASS routes to Composer

• WHEN Verifier completes with effective verdict="PASS"
• THEN the Graph SHALL invoke Composer with filtered Verifier-allowed material
• AND the final user-facing answer SHALL NOT pass through raw Executor output
• AND the final user-facing answer SHALL NOT read Executor user_facing_answer

Scenario: LOW_CONFID does not qualify for evidence retry

• WHEN Verifier completes LOW_CONFID but ceiling is LOW_CONFID, no valid critical evidence gap exists, or evidence retry count is already one
• THEN the Graph SHALL route to Composer without another Planner cycle
• AND the final answer SHALL distinguish confirmed information, possible directions, and evidence gaps

Scenario: LOW_CONFID qualifies for evidence retry

• WHEN Verifier completes LOW_CONFID with ceiling PASS, at least one critical valid evidence gap, and evidence retry count zero
• THEN the Graph SHALL invoke one EVIDENCE_GAP_ONLY Planner cycle
• AND it SHALL NOT use groundedness threshold or a ChatService feature flag to decide the retry

Scenario: REJECT does not enter retry round

• WHEN Verifier completes with effective verdict="REJECT"
• THEN the Graph SHALL NOT start an evidence supplementation round
• AND it SHALL route to Composer-safe output

Scenario: REJECT produces bounded output

• WHEN effective verdict is REJECT
• THEN the system SHALL output a degraded result indicating current evidence cannot support a reliable conclusion
• AND it SHALL NOT pass through raw Executor answer
• AND it SHALL NOT include an unsupported root-cause conclusion

Scenario: Verifier execution fails

• WHEN Verifier exhausts technical retry or returns a non-retryable failure
• THEN the Graph SHALL route to pre-verification Fallback
• AND no execution status string SHALL be used as model or effective verdict

Requirement: Verifier SHALL be observable

The Verifier execution, effective verdict, and downstream final-answer composition SHALL be persisted in the current Diagnosis Run self-evaluation container.

Scenario: claim checks written to self_evaluation

• WHEN a completed Verifier evaluation is persisted
• THEN diagnosis_run.self_evaluation.verifier_evaluation SHALL include claim_checks
• AND it SHALL continue to include compatibility facts_checked
• AND it SHALL include verifier_status, model_verdict, effective_verdict, verdict, groundedness_score, rationale, verified output/evidence, and Gatekeeper audit

Scenario: composer output written to self_evaluation

• WHEN final answer composition completes
• THEN diagnosis_run.self_evaluation.verifier_evaluation SHALL include compact composer_output when available
• AND handled Composer fallback SHALL remain observable through orchestration trace and status/reason fields
• AND existing claim/fact and Gatekeeper fields SHALL be preserved

Scenario: verdict written to self_evaluation

• WHEN Verifier completes
• THEN Graph result mapping SHALL write effective verdict under diagnosis_run.self_evaluation.verifier_evaluation.verdict
• AND existing rule_evaluation and aiops_rule_evaluation channels SHALL be preserved

Scenario: pre-verification fallback is persisted

• WHEN Graph reaches Fallback before Verifier completes
• THEN verifier evaluation SHALL include available status, Gatekeeper audit, failure reason, and Prompt audit
• AND it SHALL NOT fabricate model_verdict or effective_verdict

Scenario: gatekeeper result written to self_evaluation

• WHEN Graph result mapping persists available Gatekeeper state
• THEN diagnosis_run.self_evaluation.verifier_evaluation SHALL include gatekeeper_result
• AND the result SHALL retain status, severity, checked bindings, rules, failed rules, warnings, and errors when provided by Gatekeeper

Scenario: prompt audit written to verifier evaluation

• WHEN a complex Chat Graph result is persisted
• THEN the system SHALL include a prompt_audit object under diagnosis_run.self_evaluation.verifier_evaluation
• AND prompt_audit.version SHALL identify the Chat Prompt audit catalog version
• AND prompt_audit.prompts SHALL include Planner, Executor, Verifier, and Composer Prompt names and versions
• AND full Prompt text SHALL NOT be persisted

Scenario: prompt audit available on fallback paths

• WHEN Planner, Executor, Gatekeeper, Verifier, or Composer reaches a handled Fallback
• THEN the persisted verifier evaluation SHALL still include prompt_audit

Scenario: evaluation payload is inspected

• WHEN Graph verifier evaluation is persisted
• THEN it SHALL NOT contain raw Executor text or complete tool_trace_summary
• AND compatibility executor_structured_output SHALL contain at most the verified projection

Requirement: self_evaluation SHALL be a container object

The diagnosis_session.self_evaluation field SHALL store multiple evaluation channels in one JSON object.

Scenario: rule evaluation stored separately

• WHEN the rule-based evidence scoring completes
• THEN the EvaluationService SHALL write the result under rule_evaluation
• AND existing verifier_evaluation data SHALL be preserved

Scenario: verifier evaluation stored separately

• WHEN the Verifier completes
• THEN the ChatService SHALL write the result under verifier_evaluation
• AND existing rule_evaluation data SHALL be preserved

Scenario: no whole-object overwrite after initialization

• WHEN either evaluation channel updates self_evaluation
• THEN the implementation SHALL use read-modify-write semantics
• AND it SHALL NOT replace the whole JSON object except when initializing from null

Requirement: Verifier SHALL consume explicit verification inputs

The Verifier SHALL receive a Graph-built verified-only payload rather than inferring business inputs from conversation history, ThreadLocal state, raw Executor text, or complete tool history.

Scenario: explicit input blocks available to Verifier

• WHEN the Verifier Graph Node starts
• THEN the payload SHALL provide diagnosis_context, verified_executor_output, verified_evidence, gatekeeper_audit, and verdict_ceiling
• AND permitted structured retry_context SHALL be provided only after evidence retry preparation

Scenario: Verifier remains isolated from intermediate and raw material

• WHEN the Verifier input is serialized
• THEN it SHALL exclude Planner reasoning, Executor intermediate reasoning, raw Executor text, complete tool trace summary, Prompt text, and unrelated parent Graph State

Scenario: only passed bindings are available

• WHEN Gatekeeper returns mixed passed and failed checked bindings
• THEN verified_executor_output and verified_evidence SHALL contain only claims/material matching passed bindings
• AND the Verifier SHALL NOT receive failed or unreferenced tool material

Scenario: verified evidence preserves precise references

• WHEN the system prepares Verifier input
• THEN each verified evidence item SHALL preserve claim id, source invocation id, tool name, raw path, and matched text
• AND the item SHALL be traceable to current-run Gatekeeper validation

Scenario: gatekeeper audit and ceiling are available

• WHEN the system prepares Verifier input
• THEN the payload SHALL include raw Gatekeeper audit separately from normalized verdict ceiling
• AND a LOW_CONFID ceiling SHALL prevent effective PASS

Scenario: technical retry occurs

• WHEN the first Verifier attempt returns invalid output or a retryable invocation failure
• THEN the second attempt SHALL receive byte-identical serialized input
• AND Executor, Gatekeeper, and tools SHALL NOT rerun

Requirement: Verifier facts SHALL be auditable

Verifier claims and facts SHALL be linkable to the verified binding projection used during verification.

Scenario: claim checks contain evidence refs

• WHEN the Verifier emits claim_checks
• THEN each check SHALL include an evidence_refs array
• AND any non-empty evidence ref SHALL correspond to existing verified evidence by claim id, source invocation id, tool name, or raw path
• AND it SHALL NOT reference a failed or unverified binding

Scenario: verifier evaluation persists traceability snapshot

• WHEN Graph result mapping persists verifier evaluation
• THEN it SHALL include traceability_version
• AND it SHALL include the bounded verified_evidence snapshot used by the Verifier
• AND it SHALL NOT persist a complete tool trace summary as Verifier input

Requirement: Verifier inputs SHALL tolerate hardened no-evidence semantics

The verifier integration SHALL continue to work when evidence summaries distinguish failed calls, no-hit calls, and deduped retrievals more explicitly.

Scenario: Failed evidence remains a verifier-visible gap

• WHEN an evidence-bearing tool invocation fails
• THEN the verifier-facing trace summary SHALL preserve that failure as a gap
• AND the verifier flow SHALL continue without crashing

Scenario: Deduped retrievals do not count as fresh support

• WHEN the verifier-facing trace summary contains deduped lookup_knowledge entries
• THEN those entries SHALL be treated as no-new-evidence
• AND they SHALL NOT be interpreted as fresh direct support for the answer

Requirement: Verifier evidence summaries SHALL preserve concrete supporting facts

The verifier-facing tool_trace_summary SHALL preserve compact concrete facts from persisted evidence-tool outputs so direct evidence is not misclassified as missing merely because raw output was truncated.

Scenario: Log evidence contains a concrete matching message

• WHEN a persisted query_logs invocation output contains a concrete log message matching a critical fact
• THEN the generated tool_trace_summary SHALL include that message or a bounded excerpt of it in output_summary
• AND Verifier SHALL be able to reference the invocation id as direct evidence

Scenario: Metrics evidence contains concrete alert fields

• WHEN a persisted query_metrics invocation output contains alert names, services, or metric values
• THEN the generated tool_trace_summary SHALL include the relevant alert names, services, and bounded metric values
• AND it SHALL NOT imply unsupported alerts that are absent from the tool output

Scenario: Summary remains bounded

• WHEN a tool output is large
• THEN the generated tool_trace_summary SHALL remain bounded
• AND it SHALL preserve concrete facts before generic boilerplate or low-value formatting

Requirement: Verifier low-confidence output SHALL not present unsupported claims as confirmed

When Verifier returns LOW_CONFID, user-facing output SHALL clearly separate confirmed facts from evidence gaps and SHALL NOT leave unsupported Executor claims formatted as confirmed findings.

Scenario: LOW_CONFID with critical evidence gaps

• WHEN Verifier labels critical facts as no_evidence
• THEN the final user-facing response SHALL identify those gaps from verifier output
• AND unsupported raw Executor claims SHALL NOT be presented as confirmed conclusions

Requirement: Executor SHALL output an evidence-attribution contract

The Chat Executor SHALL produce a machine-checkable final output that separates confirmed claims from hypotheses, recommendations, and missing information.

Scenario: Executor V2 final output contains only structured diagnostic fields

• WHEN Executor completes a Chat diagnosis step under the V2 contract
• THEN its final output SHALL contain answer_version, claims, hypotheses, recommended_actions, and missing_info
• AND answer_version SHALL equal executor_evidence_v2
• AND the output SHOULD be parseable as one JSON object without Markdown fences
• AND the output SHALL NOT contain diagnosis_summary
• AND the output SHALL NOT contain user_facing_answer

Scenario: Confirmed claims carry evidence bindings

• WHEN Executor emits an item under claims
• THEN the item SHALL include claim_id, claim_type, claim_text, support_level, and evidence_bindings
• AND support_level SHALL be one of direct or indirect
• AND evidence_bindings SHALL contain at least one evidence binding

Scenario: Evidence bindings support multiple tool types

• WHEN Executor binds evidence to a claim
• THEN each binding SHALL include source_type, tool_name, source_invocation_ids, and evidence_excerpt
• AND the binding MAY include source_id
• AND the binding SHALL be able to reference lookup_knowledge, query_logs, query_metrics, or other evidence-bearing tool traces
• AND the binding SHALL NOT rely only on a RAG-specific chunk_id

Scenario: Unsupported conclusions are not confirmed claims

• WHEN a possible root cause, detail, or remediation lacks current-session tool evidence
• THEN Executor SHALL place it under hypotheses, recommended_actions, or missing_info
• AND Executor SHALL NOT present it as a confirmed claim

Scenario: Runbook and skill guidance do not become incident facts

• WHEN Executor uses runbook, skill, or historical-case guidance
• THEN the guidance MAY influence recommended_actions
• AND the guidance SHALL NOT be emitted as a current incident fact unless current-session tool evidence supports it

Requirement: User-facing Chat answers SHALL remain readable Chinese

The system SHALL preserve a readable Chinese answer for normal Chat users even when Executor emits a machine-checkable contract.

Scenario: V2 machine contract is not exposed as normal user answer

• WHEN Executor emits executor_evidence_v2
• AND Verifier returns PASS
• THEN normal user output SHALL be rendered as readable Chinese from the structured contract or a safe fallback template
• AND normal user output SHALL NOT be the raw Executor JSON object

Scenario: Machine contract remains available for trace inspection

• WHEN the Chat trace or verifier evaluation is inspected
• THEN the structured Executor contract MAY be shown for debugging or audit
• AND normal user output SHALL use the existing verifier-routed display path rather than exposing raw JSON by default

Requirement: Structured Executor output SHALL degrade safely

The StateGraph runtime SHALL tolerate malformed or absent structured Executor output without crashing the Chat flow or invoking Verifier with untrusted material.

Scenario: Malformed Executor JSON is classified

• WHEN Executor returns malformed JSON or text outside the expected contract
• THEN Executor Node SHALL set INVALID_OUTPUT
• AND the Graph SHALL route directly to deterministic pre-verification Fallback
• AND Gatekeeper, Verifier, and model Composer SHALL NOT execute

Scenario: Structured parse failure remains observable

• WHEN Executor output parsing fails
• THEN orchestration events and verifier evaluation status/failure fields SHALL make the parse failure visible
• AND the failure SHALL NOT be treated as a successful evidence-attribution contract or diagnostic verdict

Requirement: Executor Gatekeeper SHALL validate deterministic structured-output failures

The system SHALL run deterministic Gatekeeper checks as an explicit Graph Node after legal Executor output parsing and before Verifier model execution.

Scenario: schema rule rejects removed fields

• WHEN Executor structured output contains diagnosis_summary or user_facing_answer
• THEN gatekeeper_result.status SHALL be fail
• AND gatekeeper_result.failed_rules SHALL contain schema.executor_v2

Scenario: schema rule rejects missing evidence bindings

• WHEN a confirmed claim has no evidence_bindings
• THEN gatekeeper_result.status SHALL be fail
• AND gatekeeper_result.failed_rules SHALL contain schema.executor_v2

Scenario: invocation rule rejects fabricated invocation ids

• WHEN a claim evidence binding references an invocation id absent from current-run tool_invocation rows
• THEN gatekeeper_result.status SHALL be fail
• AND gatekeeper_result.failed_rules SHALL contain evidence.invocation_ref

Scenario: invocation rule rejects tool name mismatch

• WHEN a claim evidence binding references an existing current-run invocation id
• AND binding tool_name does not match persisted invocation tool_name
• THEN gatekeeper_result.status SHALL be fail
• AND gatekeeper_result.failed_rules SHALL contain evidence.invocation_ref

Scenario: valid structured output passes initial gatekeeper rules

• WHEN Executor emits legal executor_evidence_v2
• AND each claim has evidence bindings pointing to current-run invocations with matching tool names and paths
• THEN gatekeeper_result.status SHALL be pass
• AND gatekeeper_result.failed_rules SHALL be empty

Scenario: gatekeeper reject bypasses Verifier

• WHEN normalized Gatekeeper status is REJECT
• THEN the Graph SHALL route directly to pre-verification Fallback
• AND Verifier SHALL NOT execute

Scenario: gatekeeper low confidence is bounded

• WHEN normalized Gatekeeper status is LOW_CONFID with at least one passed binding
• THEN verified input SHALL contain only passed bindings
• AND effective verdict SHALL NOT exceed LOW_CONFID

Requirement: Verifier claim checks SHALL use a fixed derivability classification set

The Verifier SHALL classify each structured claim using a fixed derivability classification set.

Scenario: claim check verification values are constrained

• WHEN Verifier emits claim_checks
• THEN each item SHALL use one of direct_observation, reasonable_inference, overstated, unsupported, external_unknown, or contradicted

Scenario: claim check evidence references remain auditable

• WHEN Verifier emits claim_checks
• THEN each claim check SHALL include claim_id, verification, detail, and evidence_refs
• AND every evidence ref SHALL preserve available trace_ref, tool_name, and source_invocation_ids

Requirement: User-facing verifier outputs SHALL follow Composer-safe protocols

The system SHALL use Composer or fixed safe templates for PASS, LOW_CONFID, and REJECT user-facing responses.

Scenario: PASS uses Composer-safe output

• WHEN the final verdict is PASS
• THEN the user-facing response SHALL be generated from Verifier-allowed material through Composer or a safe fixed template
• AND it SHALL NOT use Executor user_facing_answer
• AND it SHALL NOT expose raw Executor JSON

Scenario: LOW_CONFID uses Composer-safe uncertainty output

• WHEN the final verdict is LOW_CONFID
• THEN the user-facing response SHALL include only confirmed facts, possible directions, evidence gaps, and next-step suggestions derived from Verifier-allowed material
• AND optional evidence gaps, if present, SHALL come only from verifier-identified gaps
• AND unsupported raw Executor claims SHALL NOT be presented as confirmed conclusions

Scenario: REJECT uses degraded template

• WHEN the final verdict is REJECT
• THEN the user-facing response SHALL use a degraded template or Composer-safe degraded output
• AND it SHALL include only confirmed facts, evidence gaps, and next-step suggestions
• AND it SHALL NOT include unverified raw answer content
• AND it SHALL NOT include a root-cause conclusion

Requirement: Executor evidence bindings SHALL support precise evidence references

Executor V2 evidence bindings SHALL support precise evidence references that locate evidence inside a persisted tool invocation.

Scenario: Precise evidence binding contains invocation path and excerpt

• WHEN Executor binds evidence to a claim
• THEN the binding SHOULD include singular source_invocation_id
• AND the binding SHOULD include raw_path
• AND the binding SHALL include tool_name and evidence_excerpt
• AND the raw_path SHALL be interpreted relative to the referenced tool invocation's retrieval_details.evidence_refs

Scenario: Legacy plural invocation ids remain compatibility only

• WHEN Executor emits legacy source_invocation_ids
• THEN the system MAY read them for compatibility
• AND they SHALL NOT be sufficient for a precise Gatekeeper pass without raw_path

Requirement: Gatekeeper SHALL validate evidence reference fidelity

Gatekeeper SHALL validate that Executor evidence bindings point to real current-session evidence references before Verifier uses them as primary evidence.

Scenario: Valid precise binding passes

• WHEN a binding's source_invocation_id exists in the current session
• AND the binding's tool_name matches the persisted invocation
• AND the binding's raw_path exists in retrieval_details.evidence_refs
• AND the binding's evidence_excerpt is supported by the matching evidence ref text
• THEN Gatekeeper SHALL return status=pass
• AND Gatekeeper SHALL return severity=none

Scenario: Missing raw path is low confidence

• WHEN a binding references an existing invocation
• AND the binding omits raw_path
• THEN Gatekeeper SHALL return status=fail
• AND Gatekeeper SHALL return severity=low_confid
• AND the effective verifier result SHALL NOT be PASS

Scenario: Old invocation without evidence refs is low confidence

• WHEN a binding references an existing invocation
• AND the invocation does not contain retrieval_details.evidence_refs
• THEN Gatekeeper SHALL return status=fail
• AND Gatekeeper SHALL return severity=low_confid
• AND the effective verifier result SHALL NOT be PASS

Scenario: Unknown raw path is rejected

• WHEN a binding references an existing invocation
• AND the binding's raw_path is absent from that invocation's retrieval_details.evidence_refs
• THEN Gatekeeper SHALL return status=fail
• AND Gatekeeper SHALL return severity=reject
• AND failed_rules SHALL include evidence.raw_path

Scenario: Mismatched excerpt is rejected

• WHEN a binding references an existing invocation and raw path
• AND the binding's evidence_excerpt is not supported by the matching system-side evidence ref text
• THEN Gatekeeper SHALL return status=fail
• AND Gatekeeper SHALL return severity=reject
• AND failed_rules SHALL include evidence.excerpt_mismatch

Requirement: Verifier SHALL use verified claim-local evidence for derivability

Verifier SHALL judge structured claims only against Gatekeeper-verified claim-local evidence excerpts and their precise current-run references.

Scenario: Verified excerpt supports direct observation

• WHEN verdict ceiling is PASS
• AND a claim's verified evidence matched text directly contains the claim's concrete facts
• THEN Verifier MAY classify that claim as direct_observation

Scenario: Verified evidence is complete Verifier context

• WHEN verified claims and evidence are available
• THEN Verifier SHALL use them as its evidence context
• AND it SHALL NOT require or request a complete tool trace summary
• AND it SHALL NOT read raw Executor or unreferenced tool material

Requirement: Gatekeeper severity SHALL constrain effective verdict

Runtime effective verdict calculation SHALL treat normalized Gatekeeper ceiling as a hard upper bound independent from Verifier model output.

Scenario: Reject severity bypasses Verifier

• WHEN gatekeeper_result.severity=reject
• THEN the Graph SHALL route to pre-verification Fallback without invoking Verifier
• AND it SHALL NOT fabricate an effective diagnostic verdict

Scenario: Low confidence severity prevents PASS

• WHEN gatekeeper_result.severity=low_confid
• AND the Verifier model returns verdict=PASS
• THEN deterministic effective-verdict calculation SHALL downgrade the result
• AND effective verdict SHALL be LOW_CONFID

Scenario: Gatekeeper audit includes severity

• WHEN Graph verifier evaluation is persisted
• THEN diagnosis_run.self_evaluation.verifier_evaluation.gatekeeper_result SHALL include available status, severity, checked_bindings, failed_rules, warnings, and errors

Requirement: Verifier input hook SHALL only perform narrow compatibility backfill

The verifier input hook SHALL avoid converting broad tool summaries into precise evidence references.

Scenario: Unique invocation candidate may be backfilled

• WHEN an evidence binding omits source_invocation_id
• AND exactly one current-session invocation exists for the binding's tool_name
• THEN the hook MAY backfill source_invocation_id
• AND it SHALL add a Gatekeeper warning describing the auto-backfill

Scenario: Raw path is never backfilled

• WHEN an evidence binding omits raw_path
• THEN the hook SHALL NOT synthesize raw_path
• AND Gatekeeper SHALL treat the binding as not precise enough to pass

Requirement: Executor prompt SHALL constrain narrow-scope over-expansion

The Executor prompt SHALL instruct Executor to keep narrow confirmation questions focused on observation-level claims.

Scenario: Narrow scope produces minimal observation claims

• WHEN the user asks to confirm one specific service, alert, or symptom
• THEN Executor SHOULD output the minimum necessary claims, normally one and at most two
• AND those claims SHALL be observation or negative_observation unless current-session evidence proves more
• AND Executor SHALL NOT emit unrelated root-cause, remediation, or excluded-topic claims as confirmed facts

Requirement: Gatekeeper SHALL expose rule catalog metadata in audit output

Gatekeeper SHALL include the rule catalog version and enabled rule metadata in its validation result.

Scenario: Gatekeeper pass includes rule metadata

• WHEN Gatekeeper returns status=pass
• THEN the result SHALL include rule_set_version
• AND it SHALL include a rule metadata summary

Scenario: Gatekeeper fail includes rule metadata

• WHEN Gatekeeper returns status=fail
• THEN the result SHALL include rule_set_version
• AND it SHALL include a rule metadata summary

Scenario: Gatekeeper fallback pass includes rule metadata

• WHEN the Verifier input hook returns a fallback Gatekeeper pass because no Gatekeeper service is available
• THEN the result SHOULD still include the default rule set version and an empty or default rule metadata summary

Requirement: Gatekeeper rule catalog SHALL remain deterministic

The Gatekeeper rule catalog SHALL configure metadata and simple parameters only; validation behavior SHALL remain deterministic Java code.

Scenario: Rule metadata is lightweight

• WHEN rule metadata is loaded
• THEN each enabled rule SHOULD expose an id, description, enabled flag, and default severity or relevant parameter
• AND rule metadata SHALL NOT execute dynamic scripts