feat(agent): harden verifier evidence references
This commit is contained in:
aruo
@@ -1,4 +1,4 @@
|
||||
# chat-verifier-agent Specification
|
||||
# chat-verifier-agent Specification
|
||||
|
||||
## Purpose
|
||||
TBD - created by archiving change chat-verifier-agent. Update Purpose after archive.
|
||||
@@ -119,6 +119,7 @@ The system SHALL use ChatService for explicit single-round `Planner -> Executor
|
||||
- **THEN** the system SHALL output a degraded result indicating the answer cannot be reliably generated
|
||||
- **AND** it SHALL NOT pass through the raw Executor answer
|
||||
- **AND** it SHALL NOT include a root-cause conclusion
|
||||
|
||||
### Requirement: Verifier SHALL be observable
|
||||
The Verifier's verdict and downstream final-answer composition SHALL be persisted for observability.
|
||||
|
||||
@@ -421,3 +422,112 @@ The system SHALL use Composer or fixed safe templates for PASS, LOW_CONFID, and
|
||||
- **AND** it SHALL include only confirmed facts, evidence gaps, and next-step suggestions
|
||||
- **AND** it SHALL NOT include unverified raw answer content
|
||||
- **AND** it SHALL NOT include a root-cause conclusion
|
||||
|
||||
### Requirement: Executor evidence bindings SHALL support precise evidence references
|
||||
Executor V2 evidence bindings SHALL support precise evidence references that locate evidence inside a persisted tool invocation.
|
||||
|
||||
#### Scenario: Precise evidence binding contains invocation path and excerpt
|
||||
- **WHEN** Executor binds evidence to a claim
|
||||
- **THEN** the binding SHOULD include singular `source_invocation_id`
|
||||
- **AND** the binding SHOULD include `raw_path`
|
||||
- **AND** the binding SHALL include `tool_name` and `evidence_excerpt`
|
||||
- **AND** the `raw_path` SHALL be interpreted relative to the referenced tool invocation's `retrieval_details.evidence_refs`
|
||||
|
||||
#### Scenario: Legacy plural invocation ids remain compatibility only
|
||||
- **WHEN** Executor emits legacy `source_invocation_ids`
|
||||
- **THEN** the system MAY read them for compatibility
|
||||
- **AND** they SHALL NOT be sufficient for a precise Gatekeeper pass without `raw_path`
|
||||
|
||||
### Requirement: Gatekeeper SHALL validate evidence reference fidelity
|
||||
Gatekeeper SHALL validate that Executor evidence bindings point to real current-session evidence references before Verifier uses them as primary evidence.
|
||||
|
||||
#### Scenario: Valid precise binding passes
|
||||
- **WHEN** a binding's `source_invocation_id` exists in the current session
|
||||
- **AND** the binding's `tool_name` matches the persisted invocation
|
||||
- **AND** the binding's `raw_path` exists in `retrieval_details.evidence_refs`
|
||||
- **AND** the binding's `evidence_excerpt` is supported by the matching evidence ref text
|
||||
- **THEN** Gatekeeper SHALL return `status=pass`
|
||||
- **AND** Gatekeeper SHALL return `severity=none`
|
||||
|
||||
#### Scenario: Missing raw path is low confidence
|
||||
- **WHEN** a binding references an existing invocation
|
||||
- **AND** the binding omits `raw_path`
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=low_confid`
|
||||
- **AND** the effective verifier result SHALL NOT be `PASS`
|
||||
|
||||
#### Scenario: Old invocation without evidence refs is low confidence
|
||||
- **WHEN** a binding references an existing invocation
|
||||
- **AND** the invocation does not contain `retrieval_details.evidence_refs`
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=low_confid`
|
||||
- **AND** the effective verifier result SHALL NOT be `PASS`
|
||||
|
||||
#### Scenario: Unknown raw path is rejected
|
||||
- **WHEN** a binding references an existing invocation
|
||||
- **AND** the binding's `raw_path` is absent from that invocation's `retrieval_details.evidence_refs`
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=reject`
|
||||
- **AND** `failed_rules` SHALL include `evidence.raw_path`
|
||||
|
||||
#### Scenario: Mismatched excerpt is rejected
|
||||
- **WHEN** a binding references an existing invocation and raw path
|
||||
- **AND** the binding's `evidence_excerpt` is not supported by the matching system-side evidence ref text
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=reject`
|
||||
- **AND** `failed_rules` SHALL include `evidence.excerpt_mismatch`
|
||||
|
||||
### Requirement: Verifier SHALL use verified claim-local evidence for derivability
|
||||
Verifier SHALL judge structured claims primarily against Gatekeeper-verified claim-local evidence excerpts.
|
||||
|
||||
#### Scenario: Verified excerpt supports direct observation
|
||||
- **WHEN** `gatekeeper_result.severity=none`
|
||||
- **AND** a claim's verified evidence excerpts directly contain the claim's concrete facts
|
||||
- **THEN** Verifier MAY classify that claim as `direct_observation`
|
||||
|
||||
#### Scenario: Tool trace summary is navigation context
|
||||
- **WHEN** `executor_structured_output.claims[].evidence_bindings` are available
|
||||
- **THEN** Verifier SHALL use `tool_trace_summary` as navigation and audit context
|
||||
- **AND** it SHALL NOT require `tool_trace_summary.output_summary` to contain every fact already present in verified claim-local evidence
|
||||
|
||||
### Requirement: Gatekeeper severity SHALL constrain effective verdict
|
||||
Runtime effective verdict calculation SHALL treat Gatekeeper severity as a hard upper bound.
|
||||
|
||||
#### Scenario: Reject severity prevents PASS
|
||||
- **WHEN** `gatekeeper_result.severity=reject`
|
||||
- **AND** the Verifier model returns `verdict=PASS`
|
||||
- **THEN** ChatService SHALL downgrade the effective verdict
|
||||
- **AND** the effective verdict SHALL be `REJECT`
|
||||
|
||||
#### Scenario: Low confidence severity prevents PASS
|
||||
- **WHEN** `gatekeeper_result.severity=low_confid`
|
||||
- **AND** the Verifier model returns `verdict=PASS`
|
||||
- **THEN** ChatService SHALL downgrade the effective verdict
|
||||
- **AND** the effective verdict SHALL be `LOW_CONFID`
|
||||
|
||||
#### Scenario: Gatekeeper audit includes severity
|
||||
- **WHEN** verifier evaluation is persisted
|
||||
- **THEN** `diagnosis_session.self_evaluation.verifier_evaluation.gatekeeper_result` SHALL include `status`, `severity`, `checked_bindings`, `failed_rules`, `warnings`, and `errors`
|
||||
|
||||
### Requirement: Verifier input hook SHALL only perform narrow compatibility backfill
|
||||
The verifier input hook SHALL avoid converting broad tool summaries into precise evidence references.
|
||||
|
||||
#### Scenario: Unique invocation candidate may be backfilled
|
||||
- **WHEN** an evidence binding omits `source_invocation_id`
|
||||
- **AND** exactly one current-session invocation exists for the binding's `tool_name`
|
||||
- **THEN** the hook MAY backfill `source_invocation_id`
|
||||
- **AND** it SHALL add a Gatekeeper warning describing the auto-backfill
|
||||
|
||||
#### Scenario: Raw path is never backfilled
|
||||
- **WHEN** an evidence binding omits `raw_path`
|
||||
- **THEN** the hook SHALL NOT synthesize `raw_path`
|
||||
- **AND** Gatekeeper SHALL treat the binding as not precise enough to pass
|
||||
|
||||
### Requirement: Executor prompt SHALL constrain narrow-scope over-expansion
|
||||
The Executor prompt SHALL instruct Executor to keep narrow confirmation questions focused on observation-level claims.
|
||||
|
||||
#### Scenario: Narrow scope produces minimal observation claims
|
||||
- **WHEN** the user asks to confirm one specific service, alert, or symptom
|
||||
- **THEN** Executor SHOULD output the minimum necessary claims, normally one and at most two
|
||||
- **AND** those claims SHALL be `observation` or `negative_observation` unless current-session evidence proves more
|
||||
- **AND** Executor SHALL NOT emit unrelated root-cause, remediation, or excluded-topic claims as confirmed facts
|
||||
|
||||
Reference in New Issue
Block a user