SuperBizAgent-java/openspec/specs/chat-verifier-agent/spec.md

chat-verifier-agent Specification

Purpose

TBD - created by archiving change chat-verifier-agent. Update Purpose after archive.

Requirements

Requirement: Verifier SHALL fact-check Executor answers

The system SHALL have a Verifier Agent that reads structured Executor claims and the tool call history, then produces a structured verdict based on claim derivability.

Scenario: PASS verdict when all claims have evidence

• WHEN all critical claims in executor_structured_output.claims have direct observation or reasonable inference support in tool call results
• AND at least one critical claim has direct observation
• AND no critical claim is contradicted, unsupported, external unknown, or overstated
• AND gatekeeper_result.status is not fail
• THEN the Verifier MAY output verdict="PASS" with groundedness_score ≥ 0.5

Scenario: LOW_CONFID verdict with partial evidence

• WHEN no critical claim contradicts the tool results
• AND some critical claims are unsupported, external_unknown, or overstated
• THEN the Verifier SHALL output verdict="LOW_CONFID"

Scenario: LOW_CONFID verdict with only inference support

• WHEN no critical claim contradicts the tool results
• AND all critical claims are only reasonable_inference
• THEN the Verifier SHALL output verdict="LOW_CONFID"

Scenario: REJECT verdict when claims contradict evidence

• WHEN any critical claim in executor_structured_output.claims contradicts tool call results
• OR the claim fabricates a key entity, error code, or conclusion that does not exist in the tool evidence
• THEN the Verifier SHALL output verdict="REJECT"

Scenario: Structured Executor claims are verified first

• WHEN executor_structured_output.claims is present and valid
• THEN Verifier SHALL verify each structured claim against tool_trace_summary through claim_checks
• AND each claim's evidence bindings SHALL reference existing trace or invocation identifiers when those identifiers are available
• AND a claim with fabricated or missing evidence references SHALL NOT be classified as direct_observation
• AND Verifier SHALL NOT add extra confirmed facts from executor_final_answer that are absent from executor_structured_output.claims

Scenario: Malformed structured output cannot pass through natural language fallback

• WHEN Executor does not return parseable structured output
• THEN Verifier SHALL NOT produce an effective PASS by extracting facts from executor_final_answer
• AND the effective verdict SHALL be LOW_CONFID

Requirement: Verifier SHALL output structured JSON

The Verifier SHALL output a JSON object with verdict, groundedness_score, claim_checks array, compatibility facts_checked array, and rationale.

Scenario: claim-level verifier output is accepted

• WHEN the Verifier checks Executor V2 structured output
• THEN the output SHALL contain verdict, groundedness_score, critical_fact_count, claim_checks, facts_checked, and rationale
• AND claim_checks SHALL be the primary V2 verification result
• AND facts_checked SHALL remain available for compatibility

Scenario: Output format validation

• WHEN the Verifier completes its analysis
• THEN the output SHALL contain "verdict", "groundedness_score", "claim_checks", "facts_checked", and "rationale" fields
• AND groundedness_score SHALL be a float between 0.0 and 1.0
• AND verdict SHALL be one of "PASS", "LOW_CONFID", or "REJECT"

Scenario: strict schema output

• WHEN the Verifier returns its result
• THEN it SHALL output exactly one JSON object
• AND it SHALL NOT output Markdown, code fences, or explanatory text outside the JSON object
• AND the JSON object SHALL include critical_fact_count
• AND each claim_checks item SHALL include claim_id, verification, detail, and evidence_refs
• AND each facts_checked item SHALL include fact, is_critical, verification, and detail

Requirement: facts_checked SHALL use a fixed classification set

The system SHALL continue to expose legacy facts_checked using its fixed verification classification set.

Scenario: claim checks are mapped to legacy facts

• WHEN Verifier output contains claim_checks
• THEN ChatService SHALL derive compatibility facts_checked
• AND direct_observation SHALL map to direct_evidence
• AND reasonable_inference and overstated SHALL map to indirect_support
• AND unsupported and external_unknown SHALL map to no_evidence
• AND contradicted SHALL map to contradicted

Requirement: groundedness_score SHALL be derived from fact classifications

The groundedness score SHALL be computed from critical fact classifications instead of being freely chosen by the model.

Scenario: contradicted fact forces reject

• WHEN any critical fact is labeled contradicted
• THEN the Verifier SHALL output verdict="REJECT"
• AND groundedness_score SHALL be 0.0

Scenario: score derived from supported facts

• WHEN no critical fact is contradicted
• THEN groundedness_score SHALL be computed from the mapped values of critical facts
• AND the implementation SHALL use the fixed mapping direct_evidence=1.0, indirect_support=0.6, no_evidence=0.0
• AND the result SHALL be clamped into [0.0, 1.0]

Requirement: ChatService SHALL route based on Verifier verdict

The system SHALL use ChatService for explicit single-round Planner -> Executor -> Verifier -> Composer orchestration and SHALL use ChatService to control whether an additional round is allowed.

Scenario: PASS -> Composer output

• WHEN Verifier outputs verdict="PASS"
• THEN ChatService SHALL filter Verifier-allowed material and invoke Composer or a safe fixed template
• AND the final user-facing answer SHALL NOT pass through raw Executor output
• AND the final user-facing answer SHALL NOT read Executor user_facing_answer

Scenario: LOW_CONFID score>=0.5 -> Composer output with uncertainty

• WHEN Verifier outputs verdict="LOW_CONFID" with groundedness_score >= 0.5
• THEN ChatService SHALL filter Verifier-allowed material and invoke Composer or a safe fixed template
• AND the final user-facing answer SHALL distinguish confirmed information, possible directions, and evidence gaps
• AND unsupported raw Executor claims SHALL NOT be presented as confirmed conclusions

Scenario: LOW_CONFID score<0.5 -> trigger one additional round

• WHEN Verifier outputs verdict="LOW_CONFID" with groundedness_score < 0.5 and this is the first callback
• THEN the ChatService SHALL invoke one additional Planner -> Executor -> Verifier round to supplement evidence
• AND after the second Verifier run, verdict="LOW_CONFID" SHALL be routed to Composer or a safe fixed template
• AND after the second Verifier run, verdict="REJECT" SHALL still produce a degraded Composer-safe output

Scenario: REJECT does not enter retry round

• WHEN Verifier outputs verdict="REJECT"
• THEN the system SHALL NOT start a retry round for evidence supplementation
• AND it SHALL produce a degraded output directly through Composer-safe rendering

Scenario: REJECT -> degraded output

• WHEN Verifier outputs verdict="REJECT"
• THEN the system SHALL output a degraded result indicating the answer cannot be reliably generated
• AND it SHALL NOT pass through the raw Executor answer
• AND it SHALL NOT include a root-cause conclusion

Requirement: Verifier SHALL be observable

The Verifier's verdict and downstream final-answer composition SHALL be persisted for observability.

Scenario: claim checks written to self_evaluation

• WHEN the Verifier evaluation is persisted
• THEN diagnosis_session.self_evaluation.verifier_evaluation SHALL include claim_checks
• AND it SHALL continue to include compatibility facts_checked
• AND existing fields such as verdict, groundedness_score, rationale, executor_output_parse_status, tool_trace_summary, and gatekeeper_result SHALL be preserved

Scenario: composer output written to self_evaluation

• WHEN final answer composition completes
• THEN diagnosis_session.self_evaluation.verifier_evaluation SHALL include composer_output
• AND composer_output SHALL indicate whether parsed Composer output or fallback rendering was used
• AND existing verifier fields such as claim_checks, facts_checked, gatekeeper_result, and tool_trace_summary SHALL be preserved

Scenario: verdict written to self_evaluation

• WHEN the Verifier produces a verdict
• THEN the ChatService SHALL write the verdict data under diagnosis_session.self_evaluation.verifier_evaluation
• AND existing rule_evaluation data SHALL be preserved

Scenario: gatekeeper result written to self_evaluation

• WHEN the Verifier evaluation is persisted
• THEN diagnosis_session.self_evaluation.verifier_evaluation SHALL include gatekeeper_result
• AND existing verifier fields such as verdict, facts_checked, executor_output_parse_status, and tool_trace_summary SHALL be preserved

Scenario: prompt audit written to verifier evaluation

• WHEN the Chat verifier evaluation is persisted
• THEN the system SHALL include a prompt_audit object under diagnosis_session.self_evaluation.verifier_evaluation
• AND prompt_audit.version SHALL identify the Chat prompt audit catalog version
• AND prompt_audit.prompts SHALL include the planner, executor, verifier, and composer prompt names and versions
• AND full prompt text SHALL NOT be persisted in prompt_audit

Scenario: prompt audit available on fallback paths

• WHEN Chat verifier parsing fails, Composer parsing fails, or Chat produces a degraded answer
• THEN the persisted verifier evaluation SHALL still include prompt_audit

Requirement: self_evaluation SHALL be a container object

The diagnosis_session.self_evaluation field SHALL store multiple evaluation channels in one JSON object.

Scenario: rule evaluation stored separately

• WHEN the rule-based evidence scoring completes
• THEN the EvaluationService SHALL write the result under rule_evaluation
• AND existing verifier_evaluation data SHALL be preserved

Scenario: verifier evaluation stored separately

• WHEN the Verifier completes
• THEN the ChatService SHALL write the result under verifier_evaluation
• AND existing rule_evaluation data SHALL be preserved

Scenario: no whole-object overwrite after initialization

• WHEN either evaluation channel updates self_evaluation
• THEN the implementation SHALL use read-modify-write semantics
• AND it SHALL NOT replace the whole JSON object except when initializing from null

Requirement: Verifier SHALL consume explicit verification inputs

The Verifier SHALL receive explicit verification inputs rather than inferring them only from raw conversation history.

Scenario: explicit input blocks available to Verifier

• WHEN the Verifier starts
• THEN the system SHALL provide original_query, executor_final_answer, and tool_trace_summary as explicit inputs
• AND retry_context SHALL be provided on the second round only
• AND when Executor returns a valid evidence-attribution contract, the system SHALL provide executor_structured_output
• AND when Executor output parsing fails, the system SHALL provide an executor_output_parse_status that indicates the failure
• AND message filtering MAY be used only to remove intermediate reasoning or unrelated noise

Scenario: Verifier remains isolated from intermediate reasoning

• WHEN executor_structured_output is added to the verifier input
• THEN the input SHALL still exclude Planner reasoning and Executor intermediate reasoning
• AND the input SHALL be limited to the original query, final Executor output, parsed Executor evidence contract, tool trace summary, gatekeeper result, and retry context

Scenario: tool trace summary derived from tool facts

• WHEN the system prepares verifier inputs
• THEN tool_trace_summary SHALL be generated from tool invocation facts
• AND each summary item SHALL include tool name, success state, input summary, output summary, and evidence level
• AND raw conversation history SHALL NOT be the only source of verifier evidence context

Scenario: tool trace summary preserves invocation references

• WHEN the system prepares verifier inputs
• THEN each summary item SHALL include a stable trace_ref
• AND each summary item SHALL preserve source_invocation_ids for the tool invocation rows that contributed to the summary
• AND each summary item SHOULD include query samples, retrieval layers, relevance levels, and source document labels when available

Scenario: only evidence-bearing tools included

• WHEN the system generates tool_trace_summary
• THEN it SHALL include only evidence-bearing tool invocations
• AND non-evidence helper tools such as time or formatting tools SHALL be excluded by default

Scenario: failed evidence calls preserved as evidence gaps

• WHEN an evidence-bearing tool invocation fails or returns no usable evidence
• THEN the summary SHALL still include that invocation
• AND it SHALL mark the entry as unsuccessful with an evidence level representing no evidence

Scenario: repeated tool calls may be compacted

• WHEN repeated tool invocations concern the same tool, topic domain, and round
• THEN the system MAY compact them into a merged summary entry
• AND the merged entry SHALL preserve the first effective hit and the count of repeated, failed, or no-hit calls

Scenario: raw outputs not passed through in full

• WHEN a tool invocation returns large raw content
• THEN tool_trace_summary SHALL keep only a minimal evidence summary
• AND the raw output SHALL NOT be passed through in full to the Verifier

Scenario: MessagesModelHook used only for noise reduction

• WHEN a MessagesModelHook is used for the Verifier
• THEN it MAY remove intermediate reasoning or irrelevant messages
• AND it SHALL NOT be the primary source for assembling verifier business inputs

Scenario: gatekeeper result available to Verifier

• WHEN the system prepares verifier inputs from Executor output
• THEN the payload SHALL include gatekeeper_result
• AND gatekeeper_result.status SHALL be one of pass, warn, or fail
• AND gatekeeper_result SHALL include failed_rules, warnings, and errors

Scenario: structured claims are the primary verification target

• WHEN executor_output_parse_status.status is valid
• AND executor_structured_output.claims is available
• THEN Verifier SHALL verify each claim through claim_checks
• AND Verifier SHALL NOT add extra confirmed facts from executor_final_answer that are absent from executor_structured_output.claims

Scenario: malformed structured output cannot pass through natural language fallback

• WHEN executor_output_parse_status.status is missing or malformed
• THEN Verifier SHALL NOT produce an effective PASS by extracting facts from executor_final_answer
• AND the effective verdict SHALL be LOW_CONFID

Requirement: Verifier facts SHALL be auditable

Verifier facts SHALL be linkable to the evidence summaries used during verification.

Scenario: facts_checked contains evidence refs

• WHEN the Verifier emits facts_checked
• THEN each fact SHALL include evidence_refs
• AND each evidence ref SHALL point to an existing tool_trace_summary.trace_ref
• AND each evidence ref SHALL preserve the relevant source_invocation_ids when available

Scenario: verifier evaluation persists traceability snapshot

• WHEN the ChatService persists verifier_evaluation
• THEN it SHALL include traceability_version
• AND it SHALL include the tool_trace_summary snapshot used by the Verifier

Requirement: Verifier inputs SHALL tolerate hardened no-evidence semantics

The verifier integration SHALL continue to work when evidence summaries distinguish failed calls, no-hit calls, and deduped retrievals more explicitly.

Scenario: Failed evidence remains a verifier-visible gap

• WHEN an evidence-bearing tool invocation fails
• THEN the verifier-facing trace summary SHALL preserve that failure as a gap
• AND the verifier flow SHALL continue without crashing

Scenario: Deduped retrievals do not count as fresh support

• WHEN the verifier-facing trace summary contains deduped lookup_knowledge entries
• THEN those entries SHALL be treated as no-new-evidence
• AND they SHALL NOT be interpreted as fresh direct support for the answer

Requirement: Verifier evidence summaries SHALL preserve concrete supporting facts

The verifier-facing tool_trace_summary SHALL preserve compact concrete facts from persisted evidence-tool outputs so direct evidence is not misclassified as missing merely because raw output was truncated.

Scenario: Log evidence contains a concrete matching message

• WHEN a persisted query_logs invocation output contains a concrete log message matching a critical fact
• THEN the generated tool_trace_summary SHALL include that message or a bounded excerpt of it in output_summary
• AND Verifier SHALL be able to reference the invocation id as direct evidence

Scenario: Metrics evidence contains concrete alert fields

• WHEN a persisted query_metrics invocation output contains alert names, services, or metric values
• THEN the generated tool_trace_summary SHALL include the relevant alert names, services, and bounded metric values
• AND it SHALL NOT imply unsupported alerts that are absent from the tool output

Scenario: Summary remains bounded

• WHEN a tool output is large
• THEN the generated tool_trace_summary SHALL remain bounded
• AND it SHALL preserve concrete facts before generic boilerplate or low-value formatting

Requirement: Verifier low-confidence output SHALL not present unsupported claims as confirmed

When Verifier returns LOW_CONFID, user-facing output SHALL clearly separate confirmed facts from evidence gaps and SHALL NOT leave unsupported Executor claims formatted as confirmed findings.

Scenario: LOW_CONFID with critical evidence gaps

• WHEN Verifier labels critical facts as no_evidence
• THEN the final user-facing response SHALL identify those gaps from verifier output
• AND unsupported raw Executor claims SHALL NOT be presented as confirmed conclusions

Requirement: Executor SHALL output an evidence-attribution contract

The Chat Executor SHALL produce a machine-checkable final output that separates confirmed claims from hypotheses, recommendations, and missing information.

Scenario: Executor V2 final output contains only structured diagnostic fields

• WHEN Executor completes a Chat diagnosis step under the V2 contract
• THEN its final output SHALL contain answer_version, claims, hypotheses, recommended_actions, and missing_info
• AND answer_version SHALL equal executor_evidence_v2
• AND the output SHOULD be parseable as one JSON object without Markdown fences
• AND the output SHALL NOT contain diagnosis_summary
• AND the output SHALL NOT contain user_facing_answer

Scenario: Confirmed claims carry evidence bindings

• WHEN Executor emits an item under claims
• THEN the item SHALL include claim_id, claim_type, claim_text, support_level, and evidence_bindings
• AND support_level SHALL be one of direct or indirect
• AND evidence_bindings SHALL contain at least one evidence binding

Scenario: Evidence bindings support multiple tool types

• WHEN Executor binds evidence to a claim
• THEN each binding SHALL include source_type, tool_name, source_invocation_ids, and evidence_excerpt
• AND the binding MAY include source_id
• AND the binding SHALL be able to reference lookup_knowledge, query_logs, query_metrics, or other evidence-bearing tool traces
• AND the binding SHALL NOT rely only on a RAG-specific chunk_id

Scenario: Unsupported conclusions are not confirmed claims

• WHEN a possible root cause, detail, or remediation lacks current-session tool evidence
• THEN Executor SHALL place it under hypotheses, recommended_actions, or missing_info
• AND Executor SHALL NOT present it as a confirmed claim

Scenario: Runbook and skill guidance do not become incident facts

• WHEN Executor uses runbook, skill, or historical-case guidance
• THEN the guidance MAY influence recommended_actions
• AND the guidance SHALL NOT be emitted as a current incident fact unless current-session tool evidence supports it

Requirement: User-facing Chat answers SHALL remain readable Chinese

The system SHALL preserve a readable Chinese answer for normal Chat users even when Executor emits a machine-checkable contract.

Scenario: V2 machine contract is not exposed as normal user answer

• WHEN Executor emits executor_evidence_v2
• AND Verifier returns PASS
• THEN normal user output SHALL be rendered as readable Chinese from the structured contract or a safe fallback template
• AND normal user output SHALL NOT be the raw Executor JSON object

Scenario: Machine contract remains available for trace inspection

• WHEN the Chat trace or verifier evaluation is inspected
• THEN the structured Executor contract MAY be shown for debugging or audit
• AND normal user output SHALL use the existing verifier-routed display path rather than exposing raw JSON by default

Requirement: Structured Executor output SHALL degrade safely

The system SHALL tolerate malformed or absent structured Executor output without crashing the Chat flow.

Scenario: Malformed Executor JSON is captured

• WHEN Executor returns malformed JSON or text outside the expected contract
• THEN Chat runtime SHALL preserve the raw executor_final_answer
• AND it SHALL mark executor_output_parse_status as failed
• AND Verifier SHALL use natural-language fallback behavior

Scenario: Structured parse failure remains observable

• WHEN Executor output parsing fails
• THEN the verifier evaluation or trace snapshot SHALL make the parse failure visible
• AND the failure SHALL NOT be silently treated as a successful evidence-attribution contract

Requirement: Executor Gatekeeper SHALL validate deterministic structured-output failures

The system SHALL run deterministic Gatekeeper checks after Executor output parsing and before Verifier model execution.

Scenario: schema rule rejects removed fields

• WHEN Executor structured output contains diagnosis_summary or user_facing_answer
• THEN gatekeeper_result.status SHALL be fail
• AND gatekeeper_result.failed_rules SHALL contain schema.executor_v2

Scenario: schema rule rejects missing evidence bindings

• WHEN a confirmed claim has no evidence_bindings
• THEN gatekeeper_result.status SHALL be fail
• AND gatekeeper_result.failed_rules SHALL contain schema.executor_v2

Scenario: invocation rule rejects fabricated invocation ids

• WHEN a claim evidence binding references a source_invocation_ids value that is not present in current-session tool_invocation rows
• THEN gatekeeper_result.status SHALL be fail
• AND gatekeeper_result.failed_rules SHALL contain evidence.invocation_ref

Scenario: invocation rule rejects tool name mismatch

• WHEN a claim evidence binding references an existing invocation id
• AND the binding tool_name does not match the invocation's persisted tool_name
• THEN gatekeeper_result.status SHALL be fail
• AND gatekeeper_result.failed_rules SHALL contain evidence.invocation_ref

Scenario: valid structured output passes initial gatekeeper rules

• WHEN Executor emits executor_evidence_v2
• AND each claim has evidence bindings pointing to current-session invocations with matching tool names
• THEN gatekeeper_result.status SHALL be pass
• AND gatekeeper_result.failed_rules SHALL be empty

Scenario: gatekeeper fail prevents PASS

• WHEN gatekeeper_result.status is fail
• AND the Verifier model returns verdict = "PASS"
• THEN ChatService SHALL downgrade the effective verdict
• AND the effective verdict SHALL NOT be PASS

Scenario: invocation reference failure downgrades to reject

• WHEN gatekeeper_result.failed_rules contains evidence.invocation_ref
• AND the Verifier model returns verdict = "PASS"
• THEN ChatService SHALL set the effective verdict to REJECT

Requirement: Verifier claim checks SHALL use a fixed derivability classification set

The Verifier SHALL classify each structured claim using a fixed derivability classification set.

Scenario: claim check verification values are constrained

• WHEN Verifier emits claim_checks
• THEN each item SHALL use one of direct_observation, reasonable_inference, overstated, unsupported, external_unknown, or contradicted

Scenario: claim check evidence references remain auditable

• WHEN Verifier emits claim_checks
• THEN each claim check SHALL include claim_id, verification, detail, and evidence_refs
• AND every evidence ref SHALL preserve available trace_ref, tool_name, and source_invocation_ids

Requirement: User-facing verifier outputs SHALL follow Composer-safe protocols

The system SHALL use Composer or fixed safe templates for PASS, LOW_CONFID, and REJECT user-facing responses.

Scenario: PASS uses Composer-safe output

• WHEN the final verdict is PASS
• THEN the user-facing response SHALL be generated from Verifier-allowed material through Composer or a safe fixed template
• AND it SHALL NOT use Executor user_facing_answer
• AND it SHALL NOT expose raw Executor JSON

Scenario: LOW_CONFID uses Composer-safe uncertainty output

• WHEN the final verdict is LOW_CONFID
• THEN the user-facing response SHALL include only confirmed facts, possible directions, evidence gaps, and next-step suggestions derived from Verifier-allowed material
• AND optional evidence gaps, if present, SHALL come only from verifier-identified gaps
• AND unsupported raw Executor claims SHALL NOT be presented as confirmed conclusions

Scenario: REJECT uses degraded template

• WHEN the final verdict is REJECT
• THEN the user-facing response SHALL use a degraded template or Composer-safe degraded output
• AND it SHALL include only confirmed facts, evidence gaps, and next-step suggestions
• AND it SHALL NOT include unverified raw answer content
• AND it SHALL NOT include a root-cause conclusion

Requirement: Executor evidence bindings SHALL support precise evidence references

Executor V2 evidence bindings SHALL support precise evidence references that locate evidence inside a persisted tool invocation.

Scenario: Precise evidence binding contains invocation path and excerpt

• WHEN Executor binds evidence to a claim
• THEN the binding SHOULD include singular source_invocation_id
• AND the binding SHOULD include raw_path
• AND the binding SHALL include tool_name and evidence_excerpt
• AND the raw_path SHALL be interpreted relative to the referenced tool invocation's retrieval_details.evidence_refs

Scenario: Legacy plural invocation ids remain compatibility only

• WHEN Executor emits legacy source_invocation_ids
• THEN the system MAY read them for compatibility
• AND they SHALL NOT be sufficient for a precise Gatekeeper pass without raw_path

Requirement: Gatekeeper SHALL validate evidence reference fidelity

Gatekeeper SHALL validate that Executor evidence bindings point to real current-session evidence references before Verifier uses them as primary evidence.

Scenario: Valid precise binding passes

• WHEN a binding's source_invocation_id exists in the current session
• AND the binding's tool_name matches the persisted invocation
• AND the binding's raw_path exists in retrieval_details.evidence_refs
• AND the binding's evidence_excerpt is supported by the matching evidence ref text
• THEN Gatekeeper SHALL return status=pass
• AND Gatekeeper SHALL return severity=none

Scenario: Missing raw path is low confidence

• WHEN a binding references an existing invocation
• AND the binding omits raw_path
• THEN Gatekeeper SHALL return status=fail
• AND Gatekeeper SHALL return severity=low_confid
• AND the effective verifier result SHALL NOT be PASS

Scenario: Old invocation without evidence refs is low confidence

• WHEN a binding references an existing invocation
• AND the invocation does not contain retrieval_details.evidence_refs
• THEN Gatekeeper SHALL return status=fail
• AND Gatekeeper SHALL return severity=low_confid
• AND the effective verifier result SHALL NOT be PASS

Scenario: Unknown raw path is rejected

• WHEN a binding references an existing invocation
• AND the binding's raw_path is absent from that invocation's retrieval_details.evidence_refs
• THEN Gatekeeper SHALL return status=fail
• AND Gatekeeper SHALL return severity=reject
• AND failed_rules SHALL include evidence.raw_path

Scenario: Mismatched excerpt is rejected

• WHEN a binding references an existing invocation and raw path
• AND the binding's evidence_excerpt is not supported by the matching system-side evidence ref text
• THEN Gatekeeper SHALL return status=fail
• AND Gatekeeper SHALL return severity=reject
• AND failed_rules SHALL include evidence.excerpt_mismatch

Requirement: Verifier SHALL use verified claim-local evidence for derivability

Verifier SHALL judge structured claims primarily against Gatekeeper-verified claim-local evidence excerpts.

Scenario: Verified excerpt supports direct observation

• WHEN gatekeeper_result.severity=none
• AND a claim's verified evidence excerpts directly contain the claim's concrete facts
• THEN Verifier MAY classify that claim as direct_observation

Scenario: Tool trace summary is navigation context

• WHEN executor_structured_output.claims[].evidence_bindings are available
• THEN Verifier SHALL use tool_trace_summary as navigation and audit context
• AND it SHALL NOT require tool_trace_summary.output_summary to contain every fact already present in verified claim-local evidence

Requirement: Gatekeeper severity SHALL constrain effective verdict

Runtime effective verdict calculation SHALL treat Gatekeeper severity as a hard upper bound.

Scenario: Reject severity prevents PASS

• WHEN gatekeeper_result.severity=reject
• AND the Verifier model returns verdict=PASS
• THEN ChatService SHALL downgrade the effective verdict
• AND the effective verdict SHALL be REJECT

Scenario: Low confidence severity prevents PASS

• WHEN gatekeeper_result.severity=low_confid
• AND the Verifier model returns verdict=PASS
• THEN ChatService SHALL downgrade the effective verdict
• AND the effective verdict SHALL be LOW_CONFID

Scenario: Gatekeeper audit includes severity

• WHEN verifier evaluation is persisted
• THEN diagnosis_session.self_evaluation.verifier_evaluation.gatekeeper_result SHALL include status, severity, checked_bindings, failed_rules, warnings, and errors

Requirement: Verifier input hook SHALL only perform narrow compatibility backfill

The verifier input hook SHALL avoid converting broad tool summaries into precise evidence references.

Scenario: Unique invocation candidate may be backfilled

• WHEN an evidence binding omits source_invocation_id
• AND exactly one current-session invocation exists for the binding's tool_name
• THEN the hook MAY backfill source_invocation_id
• AND it SHALL add a Gatekeeper warning describing the auto-backfill

Scenario: Raw path is never backfilled

• WHEN an evidence binding omits raw_path
• THEN the hook SHALL NOT synthesize raw_path
• AND Gatekeeper SHALL treat the binding as not precise enough to pass

Requirement: Executor prompt SHALL constrain narrow-scope over-expansion

The Executor prompt SHALL instruct Executor to keep narrow confirmation questions focused on observation-level claims.

Scenario: Narrow scope produces minimal observation claims

• WHEN the user asks to confirm one specific service, alert, or symptom
• THEN Executor SHOULD output the minimum necessary claims, normally one and at most two
• AND those claims SHALL be observation or negative_observation unless current-session evidence proves more
• AND Executor SHALL NOT emit unrelated root-cause, remediation, or excluded-topic claims as confirmed facts

Requirement: Gatekeeper SHALL expose rule catalog metadata in audit output

Gatekeeper SHALL include the rule catalog version and enabled rule metadata in its validation result.

Scenario: Gatekeeper pass includes rule metadata

• WHEN Gatekeeper returns status=pass
• THEN the result SHALL include rule_set_version
• AND it SHALL include a rule metadata summary

Scenario: Gatekeeper fail includes rule metadata

• WHEN Gatekeeper returns status=fail
• THEN the result SHALL include rule_set_version
• AND it SHALL include a rule metadata summary

Scenario: Gatekeeper fallback pass includes rule metadata

• WHEN the Verifier input hook returns a fallback Gatekeeper pass because no Gatekeeper service is available
• THEN the result SHOULD still include the default rule set version and an empty or default rule metadata summary

Requirement: Gatekeeper rule catalog SHALL remain deterministic

The Gatekeeper rule catalog SHALL configure metadata and simple parameters only; validation behavior SHALL remain deterministic Java code.

Scenario: Rule metadata is lightweight

• WHEN rule metadata is loaded
• THEN each enabled rule SHOULD expose an id, description, enabled flag, and default severity or relevant parameter
• AND rule metadata SHALL NOT execute dynamic scripts