534 lines
32 KiB
Markdown
534 lines
32 KiB
Markdown
# chat-verifier-agent Specification
|
|
|
|
## Purpose
|
|
TBD - created by archiving change chat-verifier-agent. Update Purpose after archive.
|
|
## Requirements
|
|
### Requirement: Verifier SHALL fact-check Executor answers
|
|
The system SHALL have a Verifier Agent that reads structured Executor claims and the tool call history, then produces a structured verdict based on claim derivability.
|
|
|
|
#### Scenario: PASS verdict when all claims have evidence
|
|
- **WHEN** all critical claims in `executor_structured_output.claims` have direct observation or reasonable inference support in tool call results
|
|
- **AND** at least one critical claim has direct observation
|
|
- **AND** no critical claim is contradicted, unsupported, external unknown, or overstated
|
|
- **AND** `gatekeeper_result.status` is not `fail`
|
|
- **THEN** the Verifier MAY output verdict="PASS" with groundedness_score ≥ 0.5
|
|
|
|
#### Scenario: LOW_CONFID verdict with partial evidence
|
|
- **WHEN** no critical claim contradicts the tool results
|
|
- **AND** some critical claims are `unsupported`, `external_unknown`, or `overstated`
|
|
- **THEN** the Verifier SHALL output verdict="LOW_CONFID"
|
|
|
|
#### Scenario: LOW_CONFID verdict with only inference support
|
|
- **WHEN** no critical claim contradicts the tool results
|
|
- **AND** all critical claims are only `reasonable_inference`
|
|
- **THEN** the Verifier SHALL output verdict="LOW_CONFID"
|
|
|
|
#### Scenario: REJECT verdict when claims contradict evidence
|
|
- **WHEN** any critical claim in `executor_structured_output.claims` contradicts tool call results
|
|
- **OR** the claim fabricates a key entity, error code, or conclusion that does not exist in the tool evidence
|
|
- **THEN** the Verifier SHALL output verdict="REJECT"
|
|
|
|
#### Scenario: Structured Executor claims are verified first
|
|
- **WHEN** `executor_structured_output.claims` is present and valid
|
|
- **THEN** Verifier SHALL verify each structured claim against `tool_trace_summary` through `claim_checks`
|
|
- **AND** each claim's evidence bindings SHALL reference existing trace or invocation identifiers when those identifiers are available
|
|
- **AND** a claim with fabricated or missing evidence references SHALL NOT be classified as `direct_observation`
|
|
- **AND** Verifier SHALL NOT add extra confirmed facts from `executor_final_answer` that are absent from `executor_structured_output.claims`
|
|
|
|
#### Scenario: Malformed structured output cannot pass through natural language fallback
|
|
- **WHEN** Executor does not return parseable structured output
|
|
- **THEN** Verifier SHALL NOT produce an effective `PASS` by extracting facts from `executor_final_answer`
|
|
- **AND** the effective verdict SHALL be `LOW_CONFID`
|
|
|
|
### Requirement: Verifier SHALL output structured JSON
|
|
The Verifier SHALL output a JSON object with verdict, groundedness_score, claim_checks array, compatibility facts_checked array, and rationale.
|
|
|
|
#### Scenario: claim-level verifier output is accepted
|
|
- **WHEN** the Verifier checks Executor V2 structured output
|
|
- **THEN** the output SHALL contain `verdict`, `groundedness_score`, `critical_fact_count`, `claim_checks`, `facts_checked`, and `rationale`
|
|
- **AND** `claim_checks` SHALL be the primary V2 verification result
|
|
- **AND** `facts_checked` SHALL remain available for compatibility
|
|
|
|
#### Scenario: Output format validation
|
|
- **WHEN** the Verifier completes its analysis
|
|
- **THEN** the output SHALL contain "verdict", "groundedness_score", "claim_checks", "facts_checked", and "rationale" fields
|
|
- **AND** groundedness_score SHALL be a float between 0.0 and 1.0
|
|
- **AND** verdict SHALL be one of "PASS", "LOW_CONFID", or "REJECT"
|
|
|
|
#### Scenario: strict schema output
|
|
- **WHEN** the Verifier returns its result
|
|
- **THEN** it SHALL output exactly one JSON object
|
|
- **AND** it SHALL NOT output Markdown, code fences, or explanatory text outside the JSON object
|
|
- **AND** the JSON object SHALL include `critical_fact_count`
|
|
- **AND** each `claim_checks` item SHALL include `claim_id`, `verification`, `detail`, and `evidence_refs`
|
|
- **AND** each `facts_checked` item SHALL include `fact`, `is_critical`, `verification`, and `detail`
|
|
|
|
### Requirement: facts_checked SHALL use a fixed classification set
|
|
The system SHALL continue to expose legacy `facts_checked` using its fixed verification classification set.
|
|
|
|
#### Scenario: claim checks are mapped to legacy facts
|
|
- **WHEN** Verifier output contains `claim_checks`
|
|
- **THEN** ChatService SHALL derive compatibility `facts_checked`
|
|
- **AND** `direct_observation` SHALL map to `direct_evidence`
|
|
- **AND** `reasonable_inference` and `overstated` SHALL map to `indirect_support`
|
|
- **AND** `unsupported` and `external_unknown` SHALL map to `no_evidence`
|
|
- **AND** `contradicted` SHALL map to `contradicted`
|
|
|
|
### Requirement: groundedness_score SHALL be derived from fact classifications
|
|
The groundedness score SHALL be computed from critical fact classifications instead of being freely chosen by the model.
|
|
|
|
#### Scenario: contradicted fact forces reject
|
|
- **WHEN** any critical fact is labeled `contradicted`
|
|
- **THEN** the Verifier SHALL output verdict="REJECT"
|
|
- **AND** groundedness_score SHALL be `0.0`
|
|
|
|
#### Scenario: score derived from supported facts
|
|
- **WHEN** no critical fact is contradicted
|
|
- **THEN** groundedness_score SHALL be computed from the mapped values of critical facts
|
|
- **AND** the implementation SHALL use the fixed mapping `direct_evidence=1.0`, `indirect_support=0.6`, `no_evidence=0.0`
|
|
- **AND** the result SHALL be clamped into `[0.0, 1.0]`
|
|
|
|
### Requirement: ChatService SHALL route based on Verifier verdict
|
|
The system SHALL use ChatService for explicit single-round `Planner -> Executor -> Verifier -> Composer` orchestration and SHALL use ChatService to control whether an additional round is allowed.
|
|
|
|
#### Scenario: PASS -> Composer output
|
|
- **WHEN** Verifier outputs verdict="PASS"
|
|
- **THEN** ChatService SHALL filter Verifier-allowed material and invoke Composer or a safe fixed template
|
|
- **AND** the final user-facing answer SHALL NOT pass through raw Executor output
|
|
- **AND** the final user-facing answer SHALL NOT read Executor `user_facing_answer`
|
|
|
|
#### Scenario: LOW_CONFID score>=0.5 -> Composer output with uncertainty
|
|
- **WHEN** Verifier outputs verdict="LOW_CONFID" with groundedness_score >= 0.5
|
|
- **THEN** ChatService SHALL filter Verifier-allowed material and invoke Composer or a safe fixed template
|
|
- **AND** the final user-facing answer SHALL distinguish confirmed information, possible directions, and evidence gaps
|
|
- **AND** unsupported raw Executor claims SHALL NOT be presented as confirmed conclusions
|
|
|
|
#### Scenario: LOW_CONFID score<0.5 -> trigger one additional round
|
|
- **WHEN** Verifier outputs verdict="LOW_CONFID" with groundedness_score < 0.5 and this is the first callback
|
|
- **THEN** the ChatService SHALL invoke one additional `Planner -> Executor -> Verifier` round to supplement evidence
|
|
- **AND** after the second Verifier run, verdict="LOW_CONFID" SHALL be routed to Composer or a safe fixed template
|
|
- **AND** after the second Verifier run, verdict="REJECT" SHALL still produce a degraded Composer-safe output
|
|
|
|
#### Scenario: REJECT does not enter retry round
|
|
- **WHEN** Verifier outputs verdict="REJECT"
|
|
- **THEN** the system SHALL NOT start a retry round for evidence supplementation
|
|
- **AND** it SHALL produce a degraded output directly through Composer-safe rendering
|
|
|
|
#### Scenario: REJECT -> degraded output
|
|
- **WHEN** Verifier outputs verdict="REJECT"
|
|
- **THEN** the system SHALL output a degraded result indicating the answer cannot be reliably generated
|
|
- **AND** it SHALL NOT pass through the raw Executor answer
|
|
- **AND** it SHALL NOT include a root-cause conclusion
|
|
|
|
### Requirement: Verifier SHALL be observable
|
|
The Verifier's verdict and downstream final-answer composition SHALL be persisted for observability.
|
|
|
|
#### Scenario: claim checks written to self_evaluation
|
|
- **WHEN** the Verifier evaluation is persisted
|
|
- **THEN** `diagnosis_session.self_evaluation.verifier_evaluation` SHALL include `claim_checks`
|
|
- **AND** it SHALL continue to include compatibility `facts_checked`
|
|
- **AND** existing fields such as `verdict`, `groundedness_score`, `rationale`, `executor_output_parse_status`, `tool_trace_summary`, and `gatekeeper_result` SHALL be preserved
|
|
|
|
#### Scenario: composer output written to self_evaluation
|
|
- **WHEN** final answer composition completes
|
|
- **THEN** `diagnosis_session.self_evaluation.verifier_evaluation` SHALL include `composer_output`
|
|
- **AND** `composer_output` SHALL indicate whether parsed Composer output or fallback rendering was used
|
|
- **AND** existing verifier fields such as `claim_checks`, `facts_checked`, `gatekeeper_result`, and `tool_trace_summary` SHALL be preserved
|
|
|
|
#### Scenario: verdict written to self_evaluation
|
|
- **WHEN** the Verifier produces a verdict
|
|
- **THEN** the ChatService SHALL write the verdict data under `diagnosis_session.self_evaluation.verifier_evaluation`
|
|
- **AND** existing `rule_evaluation` data SHALL be preserved
|
|
|
|
#### Scenario: gatekeeper result written to self_evaluation
|
|
- **WHEN** the Verifier evaluation is persisted
|
|
- **THEN** `diagnosis_session.self_evaluation.verifier_evaluation` SHALL include `gatekeeper_result`
|
|
- **AND** existing verifier fields such as `verdict`, `facts_checked`, `executor_output_parse_status`, and `tool_trace_summary` SHALL be preserved
|
|
|
|
### Requirement: self_evaluation SHALL be a container object
|
|
The `diagnosis_session.self_evaluation` field SHALL store multiple evaluation channels in one JSON object.
|
|
|
|
#### Scenario: rule evaluation stored separately
|
|
- **WHEN** the rule-based evidence scoring completes
|
|
- **THEN** the EvaluationService SHALL write the result under `rule_evaluation`
|
|
- **AND** existing `verifier_evaluation` data SHALL be preserved
|
|
|
|
#### Scenario: verifier evaluation stored separately
|
|
- **WHEN** the Verifier completes
|
|
- **THEN** the ChatService SHALL write the result under `verifier_evaluation`
|
|
- **AND** existing `rule_evaluation` data SHALL be preserved
|
|
|
|
#### Scenario: no whole-object overwrite after initialization
|
|
- **WHEN** either evaluation channel updates `self_evaluation`
|
|
- **THEN** the implementation SHALL use read-modify-write semantics
|
|
- **AND** it SHALL NOT replace the whole JSON object except when initializing from null
|
|
|
|
### Requirement: Verifier SHALL consume explicit verification inputs
|
|
The Verifier SHALL receive explicit verification inputs rather than inferring them only from raw conversation history.
|
|
|
|
#### Scenario: explicit input blocks available to Verifier
|
|
- **WHEN** the Verifier starts
|
|
- **THEN** the system SHALL provide `original_query`, `executor_final_answer`, and `tool_trace_summary` as explicit inputs
|
|
- **AND** `retry_context` SHALL be provided on the second round only
|
|
- **AND** when Executor returns a valid evidence-attribution contract, the system SHALL provide `executor_structured_output`
|
|
- **AND** when Executor output parsing fails, the system SHALL provide an `executor_output_parse_status` that indicates the failure
|
|
- **AND** message filtering MAY be used only to remove intermediate reasoning or unrelated noise
|
|
|
|
#### Scenario: Verifier remains isolated from intermediate reasoning
|
|
- **WHEN** `executor_structured_output` is added to the verifier input
|
|
- **THEN** the input SHALL still exclude Planner reasoning and Executor intermediate reasoning
|
|
- **AND** the input SHALL be limited to the original query, final Executor output, parsed Executor evidence contract, tool trace summary, gatekeeper result, and retry context
|
|
|
|
#### Scenario: tool trace summary derived from tool facts
|
|
- **WHEN** the system prepares verifier inputs
|
|
- **THEN** `tool_trace_summary` SHALL be generated from tool invocation facts
|
|
- **AND** each summary item SHALL include tool name, success state, input summary, output summary, and evidence level
|
|
- **AND** raw conversation history SHALL NOT be the only source of verifier evidence context
|
|
|
|
#### Scenario: tool trace summary preserves invocation references
|
|
- **WHEN** the system prepares verifier inputs
|
|
- **THEN** each summary item SHALL include a stable `trace_ref`
|
|
- **AND** each summary item SHALL preserve `source_invocation_ids` for the tool invocation rows that contributed to the summary
|
|
- **AND** each summary item SHOULD include query samples, retrieval layers, relevance levels, and source document labels when available
|
|
|
|
#### Scenario: only evidence-bearing tools included
|
|
- **WHEN** the system generates `tool_trace_summary`
|
|
- **THEN** it SHALL include only evidence-bearing tool invocations
|
|
- **AND** non-evidence helper tools such as time or formatting tools SHALL be excluded by default
|
|
|
|
#### Scenario: failed evidence calls preserved as evidence gaps
|
|
- **WHEN** an evidence-bearing tool invocation fails or returns no usable evidence
|
|
- **THEN** the summary SHALL still include that invocation
|
|
- **AND** it SHALL mark the entry as unsuccessful with an evidence level representing no evidence
|
|
|
|
#### Scenario: repeated tool calls may be compacted
|
|
- **WHEN** repeated tool invocations concern the same tool, topic domain, and round
|
|
- **THEN** the system MAY compact them into a merged summary entry
|
|
- **AND** the merged entry SHALL preserve the first effective hit and the count of repeated, failed, or no-hit calls
|
|
|
|
#### Scenario: raw outputs not passed through in full
|
|
- **WHEN** a tool invocation returns large raw content
|
|
- **THEN** `tool_trace_summary` SHALL keep only a minimal evidence summary
|
|
- **AND** the raw output SHALL NOT be passed through in full to the Verifier
|
|
|
|
#### Scenario: MessagesModelHook used only for noise reduction
|
|
- **WHEN** a MessagesModelHook is used for the Verifier
|
|
- **THEN** it MAY remove intermediate reasoning or irrelevant messages
|
|
- **AND** it SHALL NOT be the primary source for assembling verifier business inputs
|
|
|
|
#### Scenario: gatekeeper result available to Verifier
|
|
- **WHEN** the system prepares verifier inputs from Executor output
|
|
- **THEN** the payload SHALL include `gatekeeper_result`
|
|
- **AND** `gatekeeper_result.status` SHALL be one of `pass`, `warn`, or `fail`
|
|
- **AND** `gatekeeper_result` SHALL include `failed_rules`, `warnings`, and `errors`
|
|
|
|
#### Scenario: structured claims are the primary verification target
|
|
- **WHEN** `executor_output_parse_status.status` is `valid`
|
|
- **AND** `executor_structured_output.claims` is available
|
|
- **THEN** Verifier SHALL verify each claim through `claim_checks`
|
|
- **AND** Verifier SHALL NOT add extra confirmed facts from `executor_final_answer` that are absent from `executor_structured_output.claims`
|
|
|
|
#### Scenario: malformed structured output cannot pass through natural language fallback
|
|
- **WHEN** `executor_output_parse_status.status` is `missing` or `malformed`
|
|
- **THEN** Verifier SHALL NOT produce an effective `PASS` by extracting facts from `executor_final_answer`
|
|
- **AND** the effective verdict SHALL be `LOW_CONFID`
|
|
|
|
### Requirement: Verifier facts SHALL be auditable
|
|
Verifier facts SHALL be linkable to the evidence summaries used during verification.
|
|
|
|
#### Scenario: facts_checked contains evidence refs
|
|
- **WHEN** the Verifier emits `facts_checked`
|
|
- **THEN** each fact SHALL include `evidence_refs`
|
|
- **AND** each evidence ref SHALL point to an existing `tool_trace_summary.trace_ref`
|
|
- **AND** each evidence ref SHALL preserve the relevant `source_invocation_ids` when available
|
|
|
|
#### Scenario: verifier evaluation persists traceability snapshot
|
|
- **WHEN** the ChatService persists `verifier_evaluation`
|
|
- **THEN** it SHALL include `traceability_version`
|
|
- **AND** it SHALL include the `tool_trace_summary` snapshot used by the Verifier
|
|
|
|
### Requirement: Verifier inputs SHALL tolerate hardened no-evidence semantics
|
|
The verifier integration SHALL continue to work when evidence summaries distinguish failed calls, no-hit calls, and deduped retrievals more explicitly.
|
|
|
|
#### Scenario: Failed evidence remains a verifier-visible gap
|
|
- **WHEN** an evidence-bearing tool invocation fails
|
|
- **THEN** the verifier-facing trace summary SHALL preserve that failure as a gap
|
|
- **AND** the verifier flow SHALL continue without crashing
|
|
|
|
#### Scenario: Deduped retrievals do not count as fresh support
|
|
- **WHEN** the verifier-facing trace summary contains deduped `lookup_knowledge` entries
|
|
- **THEN** those entries SHALL be treated as no-new-evidence
|
|
- **AND** they SHALL NOT be interpreted as fresh direct support for the answer
|
|
|
|
### Requirement: Verifier evidence summaries SHALL preserve concrete supporting facts
|
|
The verifier-facing `tool_trace_summary` SHALL preserve compact concrete facts from persisted evidence-tool outputs so direct evidence is not misclassified as missing merely because raw output was truncated.
|
|
|
|
#### Scenario: Log evidence contains a concrete matching message
|
|
- **WHEN** a persisted `query_logs` invocation output contains a concrete log message matching a critical fact
|
|
- **THEN** the generated `tool_trace_summary` SHALL include that message or a bounded excerpt of it in `output_summary`
|
|
- **AND** Verifier SHALL be able to reference the invocation id as direct evidence
|
|
|
|
#### Scenario: Metrics evidence contains concrete alert fields
|
|
- **WHEN** a persisted `query_metrics` invocation output contains alert names, services, or metric values
|
|
- **THEN** the generated `tool_trace_summary` SHALL include the relevant alert names, services, and bounded metric values
|
|
- **AND** it SHALL NOT imply unsupported alerts that are absent from the tool output
|
|
|
|
#### Scenario: Summary remains bounded
|
|
- **WHEN** a tool output is large
|
|
- **THEN** the generated `tool_trace_summary` SHALL remain bounded
|
|
- **AND** it SHALL preserve concrete facts before generic boilerplate or low-value formatting
|
|
|
|
### Requirement: Verifier low-confidence output SHALL not present unsupported claims as confirmed
|
|
When Verifier returns `LOW_CONFID`, user-facing output SHALL clearly separate confirmed facts from evidence gaps and SHALL NOT leave unsupported Executor claims formatted as confirmed findings.
|
|
|
|
#### Scenario: LOW_CONFID with critical evidence gaps
|
|
- **WHEN** Verifier labels critical facts as `no_evidence`
|
|
- **THEN** the final user-facing response SHALL identify those gaps from verifier output
|
|
- **AND** unsupported raw Executor claims SHALL NOT be presented as confirmed conclusions
|
|
|
|
### Requirement: Executor SHALL output an evidence-attribution contract
|
|
The Chat Executor SHALL produce a machine-checkable final output that separates confirmed claims from hypotheses, recommendations, and missing information.
|
|
|
|
#### Scenario: Executor V2 final output contains only structured diagnostic fields
|
|
- **WHEN** Executor completes a Chat diagnosis step under the V2 contract
|
|
- **THEN** its final output SHALL contain `answer_version`, `claims`, `hypotheses`, `recommended_actions`, and `missing_info`
|
|
- **AND** `answer_version` SHALL equal `executor_evidence_v2`
|
|
- **AND** the output SHOULD be parseable as one JSON object without Markdown fences
|
|
- **AND** the output SHALL NOT contain `diagnosis_summary`
|
|
- **AND** the output SHALL NOT contain `user_facing_answer`
|
|
|
|
#### Scenario: Confirmed claims carry evidence bindings
|
|
- **WHEN** Executor emits an item under `claims`
|
|
- **THEN** the item SHALL include `claim_id`, `claim_type`, `claim_text`, `support_level`, and `evidence_bindings`
|
|
- **AND** `support_level` SHALL be one of `direct` or `indirect`
|
|
- **AND** `evidence_bindings` SHALL contain at least one evidence binding
|
|
|
|
#### Scenario: Evidence bindings support multiple tool types
|
|
- **WHEN** Executor binds evidence to a claim
|
|
- **THEN** each binding SHALL include `source_type`, `tool_name`, `source_invocation_ids`, and `evidence_excerpt`
|
|
- **AND** the binding MAY include `source_id`
|
|
- **AND** the binding SHALL be able to reference `lookup_knowledge`, `query_logs`, `query_metrics`, or other evidence-bearing tool traces
|
|
- **AND** the binding SHALL NOT rely only on a RAG-specific `chunk_id`
|
|
|
|
#### Scenario: Unsupported conclusions are not confirmed claims
|
|
- **WHEN** a possible root cause, detail, or remediation lacks current-session tool evidence
|
|
- **THEN** Executor SHALL place it under `hypotheses`, `recommended_actions`, or `missing_info`
|
|
- **AND** Executor SHALL NOT present it as a confirmed claim
|
|
|
|
#### Scenario: Runbook and skill guidance do not become incident facts
|
|
- **WHEN** Executor uses runbook, skill, or historical-case guidance
|
|
- **THEN** the guidance MAY influence `recommended_actions`
|
|
- **AND** the guidance SHALL NOT be emitted as a current incident fact unless current-session tool evidence supports it
|
|
|
|
### Requirement: User-facing Chat answers SHALL remain readable Chinese
|
|
The system SHALL preserve a readable Chinese answer for normal Chat users even when Executor emits a machine-checkable contract.
|
|
|
|
#### Scenario: V2 machine contract is not exposed as normal user answer
|
|
- **WHEN** Executor emits `executor_evidence_v2`
|
|
- **AND** Verifier returns `PASS`
|
|
- **THEN** normal user output SHALL be rendered as readable Chinese from the structured contract or a safe fallback template
|
|
- **AND** normal user output SHALL NOT be the raw Executor JSON object
|
|
|
|
#### Scenario: Machine contract remains available for trace inspection
|
|
- **WHEN** the Chat trace or verifier evaluation is inspected
|
|
- **THEN** the structured Executor contract MAY be shown for debugging or audit
|
|
- **AND** normal user output SHALL use the existing verifier-routed display path rather than exposing raw JSON by default
|
|
|
|
### Requirement: Structured Executor output SHALL degrade safely
|
|
The system SHALL tolerate malformed or absent structured Executor output without crashing the Chat flow.
|
|
|
|
#### Scenario: Malformed Executor JSON is captured
|
|
- **WHEN** Executor returns malformed JSON or text outside the expected contract
|
|
- **THEN** Chat runtime SHALL preserve the raw `executor_final_answer`
|
|
- **AND** it SHALL mark `executor_output_parse_status` as failed
|
|
- **AND** Verifier SHALL use natural-language fallback behavior
|
|
|
|
#### Scenario: Structured parse failure remains observable
|
|
- **WHEN** Executor output parsing fails
|
|
- **THEN** the verifier evaluation or trace snapshot SHALL make the parse failure visible
|
|
- **AND** the failure SHALL NOT be silently treated as a successful evidence-attribution contract
|
|
|
|
### Requirement: Executor Gatekeeper SHALL validate deterministic structured-output failures
|
|
The system SHALL run deterministic Gatekeeper checks after Executor output parsing and before Verifier model execution.
|
|
|
|
#### Scenario: schema rule rejects removed fields
|
|
- **WHEN** Executor structured output contains `diagnosis_summary` or `user_facing_answer`
|
|
- **THEN** `gatekeeper_result.status` SHALL be `fail`
|
|
- **AND** `gatekeeper_result.failed_rules` SHALL contain `schema.executor_v2`
|
|
|
|
#### Scenario: schema rule rejects missing evidence bindings
|
|
- **WHEN** a confirmed claim has no `evidence_bindings`
|
|
- **THEN** `gatekeeper_result.status` SHALL be `fail`
|
|
- **AND** `gatekeeper_result.failed_rules` SHALL contain `schema.executor_v2`
|
|
|
|
#### Scenario: invocation rule rejects fabricated invocation ids
|
|
- **WHEN** a claim evidence binding references a `source_invocation_ids` value that is not present in current-session `tool_invocation` rows
|
|
- **THEN** `gatekeeper_result.status` SHALL be `fail`
|
|
- **AND** `gatekeeper_result.failed_rules` SHALL contain `evidence.invocation_ref`
|
|
|
|
#### Scenario: invocation rule rejects tool name mismatch
|
|
- **WHEN** a claim evidence binding references an existing invocation id
|
|
- **AND** the binding `tool_name` does not match the invocation's persisted `tool_name`
|
|
- **THEN** `gatekeeper_result.status` SHALL be `fail`
|
|
- **AND** `gatekeeper_result.failed_rules` SHALL contain `evidence.invocation_ref`
|
|
|
|
#### Scenario: valid structured output passes initial gatekeeper rules
|
|
- **WHEN** Executor emits `executor_evidence_v2`
|
|
- **AND** each claim has evidence bindings pointing to current-session invocations with matching tool names
|
|
- **THEN** `gatekeeper_result.status` SHALL be `pass`
|
|
- **AND** `gatekeeper_result.failed_rules` SHALL be empty
|
|
|
|
#### Scenario: gatekeeper fail prevents PASS
|
|
- **WHEN** `gatekeeper_result.status` is `fail`
|
|
- **AND** the Verifier model returns `verdict = "PASS"`
|
|
- **THEN** ChatService SHALL downgrade the effective verdict
|
|
- **AND** the effective verdict SHALL NOT be `PASS`
|
|
|
|
#### Scenario: invocation reference failure downgrades to reject
|
|
- **WHEN** `gatekeeper_result.failed_rules` contains `evidence.invocation_ref`
|
|
- **AND** the Verifier model returns `verdict = "PASS"`
|
|
- **THEN** ChatService SHALL set the effective verdict to `REJECT`
|
|
|
|
### Requirement: Verifier claim checks SHALL use a fixed derivability classification set
|
|
The Verifier SHALL classify each structured claim using a fixed derivability classification set.
|
|
|
|
#### Scenario: claim check verification values are constrained
|
|
- **WHEN** Verifier emits `claim_checks`
|
|
- **THEN** each item SHALL use one of `direct_observation`, `reasonable_inference`, `overstated`, `unsupported`, `external_unknown`, or `contradicted`
|
|
|
|
#### Scenario: claim check evidence references remain auditable
|
|
- **WHEN** Verifier emits `claim_checks`
|
|
- **THEN** each claim check SHALL include `claim_id`, `verification`, `detail`, and `evidence_refs`
|
|
- **AND** every evidence ref SHALL preserve available `trace_ref`, `tool_name`, and `source_invocation_ids`
|
|
|
|
### Requirement: User-facing verifier outputs SHALL follow Composer-safe protocols
|
|
The system SHALL use Composer or fixed safe templates for PASS, LOW_CONFID, and REJECT user-facing responses.
|
|
|
|
#### Scenario: PASS uses Composer-safe output
|
|
- **WHEN** the final verdict is `PASS`
|
|
- **THEN** the user-facing response SHALL be generated from Verifier-allowed material through Composer or a safe fixed template
|
|
- **AND** it SHALL NOT use Executor `user_facing_answer`
|
|
- **AND** it SHALL NOT expose raw Executor JSON
|
|
|
|
#### Scenario: LOW_CONFID uses Composer-safe uncertainty output
|
|
- **WHEN** the final verdict is `LOW_CONFID`
|
|
- **THEN** the user-facing response SHALL include only confirmed facts, possible directions, evidence gaps, and next-step suggestions derived from Verifier-allowed material
|
|
- **AND** optional evidence gaps, if present, SHALL come only from verifier-identified gaps
|
|
- **AND** unsupported raw Executor claims SHALL NOT be presented as confirmed conclusions
|
|
|
|
#### Scenario: REJECT uses degraded template
|
|
- **WHEN** the final verdict is `REJECT`
|
|
- **THEN** the user-facing response SHALL use a degraded template or Composer-safe degraded output
|
|
- **AND** it SHALL include only confirmed facts, evidence gaps, and next-step suggestions
|
|
- **AND** it SHALL NOT include unverified raw answer content
|
|
- **AND** it SHALL NOT include a root-cause conclusion
|
|
|
|
### Requirement: Executor evidence bindings SHALL support precise evidence references
|
|
Executor V2 evidence bindings SHALL support precise evidence references that locate evidence inside a persisted tool invocation.
|
|
|
|
#### Scenario: Precise evidence binding contains invocation path and excerpt
|
|
- **WHEN** Executor binds evidence to a claim
|
|
- **THEN** the binding SHOULD include singular `source_invocation_id`
|
|
- **AND** the binding SHOULD include `raw_path`
|
|
- **AND** the binding SHALL include `tool_name` and `evidence_excerpt`
|
|
- **AND** the `raw_path` SHALL be interpreted relative to the referenced tool invocation's `retrieval_details.evidence_refs`
|
|
|
|
#### Scenario: Legacy plural invocation ids remain compatibility only
|
|
- **WHEN** Executor emits legacy `source_invocation_ids`
|
|
- **THEN** the system MAY read them for compatibility
|
|
- **AND** they SHALL NOT be sufficient for a precise Gatekeeper pass without `raw_path`
|
|
|
|
### Requirement: Gatekeeper SHALL validate evidence reference fidelity
|
|
Gatekeeper SHALL validate that Executor evidence bindings point to real current-session evidence references before Verifier uses them as primary evidence.
|
|
|
|
#### Scenario: Valid precise binding passes
|
|
- **WHEN** a binding's `source_invocation_id` exists in the current session
|
|
- **AND** the binding's `tool_name` matches the persisted invocation
|
|
- **AND** the binding's `raw_path` exists in `retrieval_details.evidence_refs`
|
|
- **AND** the binding's `evidence_excerpt` is supported by the matching evidence ref text
|
|
- **THEN** Gatekeeper SHALL return `status=pass`
|
|
- **AND** Gatekeeper SHALL return `severity=none`
|
|
|
|
#### Scenario: Missing raw path is low confidence
|
|
- **WHEN** a binding references an existing invocation
|
|
- **AND** the binding omits `raw_path`
|
|
- **THEN** Gatekeeper SHALL return `status=fail`
|
|
- **AND** Gatekeeper SHALL return `severity=low_confid`
|
|
- **AND** the effective verifier result SHALL NOT be `PASS`
|
|
|
|
#### Scenario: Old invocation without evidence refs is low confidence
|
|
- **WHEN** a binding references an existing invocation
|
|
- **AND** the invocation does not contain `retrieval_details.evidence_refs`
|
|
- **THEN** Gatekeeper SHALL return `status=fail`
|
|
- **AND** Gatekeeper SHALL return `severity=low_confid`
|
|
- **AND** the effective verifier result SHALL NOT be `PASS`
|
|
|
|
#### Scenario: Unknown raw path is rejected
|
|
- **WHEN** a binding references an existing invocation
|
|
- **AND** the binding's `raw_path` is absent from that invocation's `retrieval_details.evidence_refs`
|
|
- **THEN** Gatekeeper SHALL return `status=fail`
|
|
- **AND** Gatekeeper SHALL return `severity=reject`
|
|
- **AND** `failed_rules` SHALL include `evidence.raw_path`
|
|
|
|
#### Scenario: Mismatched excerpt is rejected
|
|
- **WHEN** a binding references an existing invocation and raw path
|
|
- **AND** the binding's `evidence_excerpt` is not supported by the matching system-side evidence ref text
|
|
- **THEN** Gatekeeper SHALL return `status=fail`
|
|
- **AND** Gatekeeper SHALL return `severity=reject`
|
|
- **AND** `failed_rules` SHALL include `evidence.excerpt_mismatch`
|
|
|
|
### Requirement: Verifier SHALL use verified claim-local evidence for derivability
|
|
Verifier SHALL judge structured claims primarily against Gatekeeper-verified claim-local evidence excerpts.
|
|
|
|
#### Scenario: Verified excerpt supports direct observation
|
|
- **WHEN** `gatekeeper_result.severity=none`
|
|
- **AND** a claim's verified evidence excerpts directly contain the claim's concrete facts
|
|
- **THEN** Verifier MAY classify that claim as `direct_observation`
|
|
|
|
#### Scenario: Tool trace summary is navigation context
|
|
- **WHEN** `executor_structured_output.claims[].evidence_bindings` are available
|
|
- **THEN** Verifier SHALL use `tool_trace_summary` as navigation and audit context
|
|
- **AND** it SHALL NOT require `tool_trace_summary.output_summary` to contain every fact already present in verified claim-local evidence
|
|
|
|
### Requirement: Gatekeeper severity SHALL constrain effective verdict
|
|
Runtime effective verdict calculation SHALL treat Gatekeeper severity as a hard upper bound.
|
|
|
|
#### Scenario: Reject severity prevents PASS
|
|
- **WHEN** `gatekeeper_result.severity=reject`
|
|
- **AND** the Verifier model returns `verdict=PASS`
|
|
- **THEN** ChatService SHALL downgrade the effective verdict
|
|
- **AND** the effective verdict SHALL be `REJECT`
|
|
|
|
#### Scenario: Low confidence severity prevents PASS
|
|
- **WHEN** `gatekeeper_result.severity=low_confid`
|
|
- **AND** the Verifier model returns `verdict=PASS`
|
|
- **THEN** ChatService SHALL downgrade the effective verdict
|
|
- **AND** the effective verdict SHALL be `LOW_CONFID`
|
|
|
|
#### Scenario: Gatekeeper audit includes severity
|
|
- **WHEN** verifier evaluation is persisted
|
|
- **THEN** `diagnosis_session.self_evaluation.verifier_evaluation.gatekeeper_result` SHALL include `status`, `severity`, `checked_bindings`, `failed_rules`, `warnings`, and `errors`
|
|
|
|
### Requirement: Verifier input hook SHALL only perform narrow compatibility backfill
|
|
The verifier input hook SHALL avoid converting broad tool summaries into precise evidence references.
|
|
|
|
#### Scenario: Unique invocation candidate may be backfilled
|
|
- **WHEN** an evidence binding omits `source_invocation_id`
|
|
- **AND** exactly one current-session invocation exists for the binding's `tool_name`
|
|
- **THEN** the hook MAY backfill `source_invocation_id`
|
|
- **AND** it SHALL add a Gatekeeper warning describing the auto-backfill
|
|
|
|
#### Scenario: Raw path is never backfilled
|
|
- **WHEN** an evidence binding omits `raw_path`
|
|
- **THEN** the hook SHALL NOT synthesize `raw_path`
|
|
- **AND** Gatekeeper SHALL treat the binding as not precise enough to pass
|
|
|
|
### Requirement: Executor prompt SHALL constrain narrow-scope over-expansion
|
|
The Executor prompt SHALL instruct Executor to keep narrow confirmation questions focused on observation-level claims.
|
|
|
|
#### Scenario: Narrow scope produces minimal observation claims
|
|
- **WHEN** the user asks to confirm one specific service, alert, or symptom
|
|
- **THEN** Executor SHOULD output the minimum necessary claims, normally one and at most two
|
|
- **AND** those claims SHALL be `observation` or `negative_observation` unless current-session evidence proves more
|
|
- **AND** Executor SHALL NOT emit unrelated root-cause, remediation, or excluded-topic claims as confirmed facts
|