fix(agent): harden live diagnosis skill observability
This commit is contained in:
@@ -84,3 +84,27 @@ The system SHALL provide offline tests for the hardened evidence contract and de
|
||||
- **WHEN** the test suite runs the focused `ChatService` fallback tests
|
||||
- **THEN** it SHALL verify the missing-output, invalid-JSON, `LOW_CONFID`, and `REJECT` degraded paths without requiring a real LLM or database
|
||||
|
||||
### Requirement: Evidence summaries SHALL preserve strongest concrete support across merged invocations
|
||||
When multiple invocations are merged into one verifier-facing summary entry, the summary SHALL preserve concrete support from the strongest successful invocation.
|
||||
|
||||
#### Scenario: Merged log calls include one direct hit
|
||||
- **WHEN** repeated `query_logs` calls for the same topic are merged
|
||||
- **AND** at least one call contains a concrete direct evidence hit
|
||||
- **THEN** the merged summary SHALL include a bounded direct-hit excerpt
|
||||
- **AND** it SHALL preserve all contributing `source_invocation_ids`
|
||||
|
||||
#### Scenario: Truncated preview still contains direct evidence
|
||||
- **WHEN** a persisted output preview is marked truncated
|
||||
- **AND** the preview contains a concrete direct evidence hit
|
||||
- **THEN** the summary SHALL treat the row as successful evidence
|
||||
- **AND** it SHALL NOT downgrade the row to no-evidence only because the raw output was truncated
|
||||
|
||||
### Requirement: Live trace counts SHALL distinguish model steps from evidence tool invocations
|
||||
The persisted trace SHALL make it possible to audit model step counts separately from evidence tool invocation counts.
|
||||
|
||||
#### Scenario: Diagnosis session exposes aggregate counts
|
||||
- **WHEN** a diagnosis session completes
|
||||
- **THEN** `diagnosis_session.step_count` SHALL count persisted Agent model steps
|
||||
- **AND** `diagnosis_session.tool_call_count` SHALL count persisted evidence-tool invocation rows
|
||||
- **AND** helper workflow calls that are not evidence rows SHALL be auditable from agent steps or logs without inflating `tool_invocation`
|
||||
|
||||
|
||||
Reference in New Issue
Block a user