feat(agent): add executor evidence v2 contract
This commit is contained in:
@@ -256,20 +256,24 @@ When Verifier returns `LOW_CONFID`, user-facing output SHALL clearly separate co
|
||||
### Requirement: Executor SHALL output an evidence-attribution contract
|
||||
The Chat Executor SHALL produce a machine-checkable final output that separates confirmed claims from hypotheses, recommendations, and missing information.
|
||||
|
||||
#### Scenario: Executor final output contains required top-level fields
|
||||
- **WHEN** Executor completes a Chat diagnosis step
|
||||
- **THEN** its final output SHALL contain `answer_version`, `diagnosis_summary`, `claims`, `hypotheses`, `recommended_actions`, `missing_info`, and `user_facing_answer`
|
||||
#### Scenario: Executor V2 final output contains only structured diagnostic fields
|
||||
- **WHEN** Executor completes a Chat diagnosis step under the V2 contract
|
||||
- **THEN** its final output SHALL contain `answer_version`, `claims`, `hypotheses`, `recommended_actions`, and `missing_info`
|
||||
- **AND** `answer_version` SHALL equal `executor_evidence_v2`
|
||||
- **AND** the output SHOULD be parseable as one JSON object without Markdown fences
|
||||
- **AND** the output SHALL NOT contain `diagnosis_summary`
|
||||
- **AND** the output SHALL NOT contain `user_facing_answer`
|
||||
|
||||
#### Scenario: Confirmed claims carry evidence bindings
|
||||
- **WHEN** Executor emits an item under `claims`
|
||||
- **THEN** the item SHALL include `claim_id`, `claim_type`, `claim_text`, `support_level`, and `evidence_bindings`
|
||||
- **AND** `support_level` SHALL be one of `direct`, `indirect`, or `none`
|
||||
- **AND** claims with `support_level=direct` or `support_level=indirect` SHALL include at least one evidence binding
|
||||
- **AND** `support_level` SHALL be one of `direct` or `indirect`
|
||||
- **AND** `evidence_bindings` SHALL contain at least one evidence binding
|
||||
|
||||
#### Scenario: Evidence bindings support multiple tool types
|
||||
- **WHEN** Executor binds evidence to a claim
|
||||
- **THEN** each binding SHALL include `source_type`, `source_id`, `tool_name`, `source_invocation_ids`, and `evidence_excerpt`
|
||||
- **THEN** each binding SHALL include `source_type`, `tool_name`, `source_invocation_ids`, and `evidence_excerpt`
|
||||
- **AND** the binding MAY include `source_id`
|
||||
- **AND** the binding SHALL be able to reference `lookup_knowledge`, `query_logs`, `query_metrics`, or other evidence-bearing tool traces
|
||||
- **AND** the binding SHALL NOT rely only on a RAG-specific `chunk_id`
|
||||
|
||||
@@ -286,10 +290,11 @@ The Chat Executor SHALL produce a machine-checkable final output that separates
|
||||
### Requirement: User-facing Chat answers SHALL remain readable Chinese
|
||||
The system SHALL preserve a readable Chinese answer for normal Chat users even when Executor emits a machine-checkable contract.
|
||||
|
||||
#### Scenario: User-facing answer is available
|
||||
- **WHEN** Executor emits structured output
|
||||
- **THEN** `user_facing_answer` SHALL be written in Chinese
|
||||
- **AND** it SHALL be consistent with the confirmed claims, hypotheses, recommended actions, and missing information in the same JSON object
|
||||
#### Scenario: V2 machine contract is not exposed as normal user answer
|
||||
- **WHEN** Executor emits `executor_evidence_v2`
|
||||
- **AND** Verifier returns `PASS`
|
||||
- **THEN** normal user output SHALL be rendered as readable Chinese from the structured contract or a safe fallback template
|
||||
- **AND** normal user output SHALL NOT be the raw Executor JSON object
|
||||
|
||||
#### Scenario: Machine contract remains available for trace inspection
|
||||
- **WHEN** the Chat trace or verifier evaluation is inspected
|
||||
@@ -309,3 +314,4 @@ The system SHALL tolerate malformed or absent structured Executor output without
|
||||
- **WHEN** Executor output parsing fails
|
||||
- **THEN** the verifier evaluation or trace snapshot SHALL make the parse failure visible
|
||||
- **AND** the failure SHALL NOT be silently treated as a successful evidence-attribution contract
|
||||
|
||||
|
||||
Reference in New Issue
Block a user