feat(agent): harden verifier evidence references
This commit is contained in:
aruo
@@ -1,4 +1,4 @@
|
||||
# chat-verifier-agent Specification
|
||||
# chat-verifier-agent Specification
|
||||
|
||||
## Purpose
|
||||
TBD - created by archiving change chat-verifier-agent. Update Purpose after archive.
|
||||
@@ -119,6 +119,7 @@ The system SHALL use ChatService for explicit single-round `Planner -> Executor
|
||||
- **THEN** the system SHALL output a degraded result indicating the answer cannot be reliably generated
|
||||
- **AND** it SHALL NOT pass through the raw Executor answer
|
||||
- **AND** it SHALL NOT include a root-cause conclusion
|
||||
|
||||
### Requirement: Verifier SHALL be observable
|
||||
The Verifier's verdict and downstream final-answer composition SHALL be persisted for observability.
|
||||
|
||||
@@ -421,3 +422,112 @@ The system SHALL use Composer or fixed safe templates for PASS, LOW_CONFID, and
|
||||
- **AND** it SHALL include only confirmed facts, evidence gaps, and next-step suggestions
|
||||
- **AND** it SHALL NOT include unverified raw answer content
|
||||
- **AND** it SHALL NOT include a root-cause conclusion
|
||||
|
||||
### Requirement: Executor evidence bindings SHALL support precise evidence references
|
||||
Executor V2 evidence bindings SHALL support precise evidence references that locate evidence inside a persisted tool invocation.
|
||||
|
||||
#### Scenario: Precise evidence binding contains invocation path and excerpt
|
||||
- **WHEN** Executor binds evidence to a claim
|
||||
- **THEN** the binding SHOULD include singular `source_invocation_id`
|
||||
- **AND** the binding SHOULD include `raw_path`
|
||||
- **AND** the binding SHALL include `tool_name` and `evidence_excerpt`
|
||||
- **AND** the `raw_path` SHALL be interpreted relative to the referenced tool invocation's `retrieval_details.evidence_refs`
|
||||
|
||||
#### Scenario: Legacy plural invocation ids remain compatibility only
|
||||
- **WHEN** Executor emits legacy `source_invocation_ids`
|
||||
- **THEN** the system MAY read them for compatibility
|
||||
- **AND** they SHALL NOT be sufficient for a precise Gatekeeper pass without `raw_path`
|
||||
|
||||
### Requirement: Gatekeeper SHALL validate evidence reference fidelity
|
||||
Gatekeeper SHALL validate that Executor evidence bindings point to real current-session evidence references before Verifier uses them as primary evidence.
|
||||
|
||||
#### Scenario: Valid precise binding passes
|
||||
- **WHEN** a binding's `source_invocation_id` exists in the current session
|
||||
- **AND** the binding's `tool_name` matches the persisted invocation
|
||||
- **AND** the binding's `raw_path` exists in `retrieval_details.evidence_refs`
|
||||
- **AND** the binding's `evidence_excerpt` is supported by the matching evidence ref text
|
||||
- **THEN** Gatekeeper SHALL return `status=pass`
|
||||
- **AND** Gatekeeper SHALL return `severity=none`
|
||||
|
||||
#### Scenario: Missing raw path is low confidence
|
||||
- **WHEN** a binding references an existing invocation
|
||||
- **AND** the binding omits `raw_path`
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=low_confid`
|
||||
- **AND** the effective verifier result SHALL NOT be `PASS`
|
||||
|
||||
#### Scenario: Old invocation without evidence refs is low confidence
|
||||
- **WHEN** a binding references an existing invocation
|
||||
- **AND** the invocation does not contain `retrieval_details.evidence_refs`
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=low_confid`
|
||||
- **AND** the effective verifier result SHALL NOT be `PASS`
|
||||
|
||||
#### Scenario: Unknown raw path is rejected
|
||||
- **WHEN** a binding references an existing invocation
|
||||
- **AND** the binding's `raw_path` is absent from that invocation's `retrieval_details.evidence_refs`
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=reject`
|
||||
- **AND** `failed_rules` SHALL include `evidence.raw_path`
|
||||
|
||||
#### Scenario: Mismatched excerpt is rejected
|
||||
- **WHEN** a binding references an existing invocation and raw path
|
||||
- **AND** the binding's `evidence_excerpt` is not supported by the matching system-side evidence ref text
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=reject`
|
||||
- **AND** `failed_rules` SHALL include `evidence.excerpt_mismatch`
|
||||
|
||||
### Requirement: Verifier SHALL use verified claim-local evidence for derivability
|
||||
Verifier SHALL judge structured claims primarily against Gatekeeper-verified claim-local evidence excerpts.
|
||||
|
||||
#### Scenario: Verified excerpt supports direct observation
|
||||
- **WHEN** `gatekeeper_result.severity=none`
|
||||
- **AND** a claim's verified evidence excerpts directly contain the claim's concrete facts
|
||||
- **THEN** Verifier MAY classify that claim as `direct_observation`
|
||||
|
||||
#### Scenario: Tool trace summary is navigation context
|
||||
- **WHEN** `executor_structured_output.claims[].evidence_bindings` are available
|
||||
- **THEN** Verifier SHALL use `tool_trace_summary` as navigation and audit context
|
||||
- **AND** it SHALL NOT require `tool_trace_summary.output_summary` to contain every fact already present in verified claim-local evidence
|
||||
|
||||
### Requirement: Gatekeeper severity SHALL constrain effective verdict
|
||||
Runtime effective verdict calculation SHALL treat Gatekeeper severity as a hard upper bound.
|
||||
|
||||
#### Scenario: Reject severity prevents PASS
|
||||
- **WHEN** `gatekeeper_result.severity=reject`
|
||||
- **AND** the Verifier model returns `verdict=PASS`
|
||||
- **THEN** ChatService SHALL downgrade the effective verdict
|
||||
- **AND** the effective verdict SHALL be `REJECT`
|
||||
|
||||
#### Scenario: Low confidence severity prevents PASS
|
||||
- **WHEN** `gatekeeper_result.severity=low_confid`
|
||||
- **AND** the Verifier model returns `verdict=PASS`
|
||||
- **THEN** ChatService SHALL downgrade the effective verdict
|
||||
- **AND** the effective verdict SHALL be `LOW_CONFID`
|
||||
|
||||
#### Scenario: Gatekeeper audit includes severity
|
||||
- **WHEN** verifier evaluation is persisted
|
||||
- **THEN** `diagnosis_session.self_evaluation.verifier_evaluation.gatekeeper_result` SHALL include `status`, `severity`, `checked_bindings`, `failed_rules`, `warnings`, and `errors`
|
||||
|
||||
### Requirement: Verifier input hook SHALL only perform narrow compatibility backfill
|
||||
The verifier input hook SHALL avoid converting broad tool summaries into precise evidence references.
|
||||
|
||||
#### Scenario: Unique invocation candidate may be backfilled
|
||||
- **WHEN** an evidence binding omits `source_invocation_id`
|
||||
- **AND** exactly one current-session invocation exists for the binding's `tool_name`
|
||||
- **THEN** the hook MAY backfill `source_invocation_id`
|
||||
- **AND** it SHALL add a Gatekeeper warning describing the auto-backfill
|
||||
|
||||
#### Scenario: Raw path is never backfilled
|
||||
- **WHEN** an evidence binding omits `raw_path`
|
||||
- **THEN** the hook SHALL NOT synthesize `raw_path`
|
||||
- **AND** Gatekeeper SHALL treat the binding as not precise enough to pass
|
||||
|
||||
### Requirement: Executor prompt SHALL constrain narrow-scope over-expansion
|
||||
The Executor prompt SHALL instruct Executor to keep narrow confirmation questions focused on observation-level claims.
|
||||
|
||||
#### Scenario: Narrow scope produces minimal observation claims
|
||||
- **WHEN** the user asks to confirm one specific service, alert, or symptom
|
||||
- **THEN** Executor SHOULD output the minimum necessary claims, normally one and at most two
|
||||
- **AND** those claims SHALL be `observation` or `negative_observation` unless current-session evidence proves more
|
||||
- **AND** Executor SHALL NOT emit unrelated root-cause, remediation, or excluded-topic claims as confirmed facts
|
||||
|
||||
@@ -108,3 +108,44 @@ The persisted trace SHALL make it possible to audit model step counts separately
|
||||
- **AND** `diagnosis_session.tool_call_count` SHALL count persisted evidence-tool invocation rows
|
||||
- **AND** helper workflow calls that are not evidence rows SHALL be auditable from agent steps or logs without inflating `tool_invocation`
|
||||
|
||||
### Requirement: Evidence tools SHALL persist minimal evidence refs
|
||||
Evidence-bearing tool invocations SHALL persist claim-addressable evidence references in `tool_invocation.retrieval_details.evidence_refs`.
|
||||
|
||||
#### Scenario: Metrics alerts produce evidence refs
|
||||
- **WHEN** a `query_metrics` invocation returns alert entries
|
||||
- **THEN** the persisted retrieval details SHALL include one `evidence_refs` item per usable alert
|
||||
- **AND** each item SHALL include `raw_path` formatted as `$.alerts[i]`
|
||||
- **AND** each item SHALL include bounded `text` containing concrete alert facts such as alert name, state, service, current value, and duration when available
|
||||
|
||||
#### Scenario: Logs produce evidence refs
|
||||
- **WHEN** a `query_logs` invocation returns log entries
|
||||
- **THEN** the persisted retrieval details SHALL include one `evidence_refs` item per usable log
|
||||
- **AND** each item SHALL include `raw_path` formatted as `$.logs[i]`
|
||||
- **AND** each item SHALL include bounded `text` containing concrete log facts such as timestamp, level, service, and message when available
|
||||
|
||||
#### Scenario: Knowledge lookup produces evidence refs
|
||||
- **WHEN** a `lookup_knowledge` invocation returns evidence blocks
|
||||
- **THEN** the persisted retrieval details SHALL include one `evidence_refs` item per usable evidence block
|
||||
- **AND** each item SHALL include `raw_path` formatted as `$.evidence_blocks[i]`
|
||||
- **AND** each item SHALL include bounded `text` containing concrete block content, title, or source when available
|
||||
|
||||
#### Scenario: Evidence ref extraction does not infer diagnosis
|
||||
- **WHEN** the recorder creates `evidence_refs`
|
||||
- **THEN** it SHALL only copy or format concrete tool output fields
|
||||
- **AND** it SHALL NOT infer root cause, remediation, or diagnosis conclusions
|
||||
|
||||
### Requirement: Log mock no-hit semantics SHALL avoid placeholder evidence
|
||||
The log query mock SHALL distinguish positive mock evidence from no-hit results without using placeholder service logs as evidence.
|
||||
|
||||
#### Scenario: HikariCP positive query returns order-service pool evidence
|
||||
- **WHEN** a `query_logs` request targets `order-service` and HikariCP connection-pool exhaustion terms
|
||||
- **THEN** the tool SHALL return order-service HikariCP-related log entries
|
||||
- **AND** the returned evidence SHALL include concrete terms such as `HikariPool`, `active=50/50`, `waiting`, or `request timed out after 30000ms`
|
||||
- **AND** it SHALL NOT return `generic-service` placeholder logs
|
||||
|
||||
#### Scenario: HikariCP no-hit query returns no evidence
|
||||
- **WHEN** a `query_logs` request targets a service without matching HikariCP mock evidence
|
||||
- **THEN** the tool SHALL return an empty `logs` array
|
||||
- **AND** it SHALL mark the output as `evidence_status=no_evidence`
|
||||
- **AND** it SHALL NOT return `generic-service` placeholder logs
|
||||
|
||||
|
||||
Reference in New Issue
Block a user