feat(agent): add executor gatekeeper hook
This commit is contained in:
@@ -129,6 +129,11 @@ The Verifier's verdict SHALL be persisted for observability.
|
||||
- **THEN** the ChatService SHALL write the verdict data under `diagnosis_session.self_evaluation.verifier_evaluation`
|
||||
- **AND** existing `rule_evaluation` data SHALL be preserved
|
||||
|
||||
#### Scenario: gatekeeper result written to self_evaluation
|
||||
- **WHEN** the Verifier evaluation is persisted
|
||||
- **THEN** `diagnosis_session.self_evaluation.verifier_evaluation` SHALL include `gatekeeper_result`
|
||||
- **AND** existing verifier fields such as `verdict`, `facts_checked`, `executor_output_parse_status`, and `tool_trace_summary` SHALL be preserved
|
||||
|
||||
### Requirement: self_evaluation SHALL be a container object
|
||||
The `diagnosis_session.self_evaluation` field SHALL store multiple evaluation channels in one JSON object.
|
||||
|
||||
@@ -200,6 +205,12 @@ The Verifier SHALL receive explicit verification inputs rather than inferring th
|
||||
- **THEN** it MAY remove intermediate reasoning or irrelevant messages
|
||||
- **AND** it SHALL NOT be the primary source for assembling verifier business inputs
|
||||
|
||||
#### Scenario: gatekeeper result available to Verifier
|
||||
- **WHEN** the system prepares verifier inputs from Executor output
|
||||
- **THEN** the payload SHALL include `gatekeeper_result`
|
||||
- **AND** `gatekeeper_result.status` SHALL be one of `pass`, `warn`, or `fail`
|
||||
- **AND** `gatekeeper_result` SHALL include `failed_rules`, `warnings`, and `errors`
|
||||
|
||||
### Requirement: Verifier facts SHALL be auditable
|
||||
Verifier facts SHALL be linkable to the evidence summaries used during verification.
|
||||
|
||||
@@ -315,3 +326,32 @@ The system SHALL tolerate malformed or absent structured Executor output without
|
||||
- **THEN** the verifier evaluation or trace snapshot SHALL make the parse failure visible
|
||||
- **AND** the failure SHALL NOT be silently treated as a successful evidence-attribution contract
|
||||
|
||||
### Requirement: Executor Gatekeeper SHALL validate deterministic structured-output failures
|
||||
The system SHALL run deterministic Gatekeeper checks after Executor output parsing and before Verifier model execution.
|
||||
|
||||
#### Scenario: schema rule rejects removed fields
|
||||
- **WHEN** Executor structured output contains `diagnosis_summary` or `user_facing_answer`
|
||||
- **THEN** `gatekeeper_result.status` SHALL be `fail`
|
||||
- **AND** `gatekeeper_result.failed_rules` SHALL contain `schema.executor_v2`
|
||||
|
||||
#### Scenario: schema rule rejects missing evidence bindings
|
||||
- **WHEN** a confirmed claim has no `evidence_bindings`
|
||||
- **THEN** `gatekeeper_result.status` SHALL be `fail`
|
||||
- **AND** `gatekeeper_result.failed_rules` SHALL contain `schema.executor_v2`
|
||||
|
||||
#### Scenario: invocation rule rejects fabricated invocation ids
|
||||
- **WHEN** a claim evidence binding references a `source_invocation_ids` value that is not present in current-session `tool_invocation` rows
|
||||
- **THEN** `gatekeeper_result.status` SHALL be `fail`
|
||||
- **AND** `gatekeeper_result.failed_rules` SHALL contain `evidence.invocation_ref`
|
||||
|
||||
#### Scenario: invocation rule rejects tool name mismatch
|
||||
- **WHEN** a claim evidence binding references an existing invocation id
|
||||
- **AND** the binding `tool_name` does not match the invocation's persisted `tool_name`
|
||||
- **THEN** `gatekeeper_result.status` SHALL be `fail`
|
||||
- **AND** `gatekeeper_result.failed_rules` SHALL contain `evidence.invocation_ref`
|
||||
|
||||
#### Scenario: valid structured output passes initial gatekeeper rules
|
||||
- **WHEN** Executor emits `executor_evidence_v2`
|
||||
- **AND** each claim has evidence bindings pointing to current-session invocations with matching tool names
|
||||
- **THEN** `gatekeeper_result.status` SHALL be `pass`
|
||||
- **AND** `gatekeeper_result.failed_rules` SHALL be empty
|
||||
|
||||
Reference in New Issue
Block a user