fix(agent): harden live diagnosis skill observability

This commit is contained in:
aruo
2026-07-07 00:20:34 +08:00
parent 64adb998cf
commit b3315ead52
20 changed files with 792 additions and 26 deletions
@@ -84,3 +84,27 @@ The system SHALL provide offline tests for the hardened evidence contract and de
- **WHEN** the test suite runs the focused `ChatService` fallback tests
- **THEN** it SHALL verify the missing-output, invalid-JSON, `LOW_CONFID`, and `REJECT` degraded paths without requiring a real LLM or database
### Requirement: Evidence summaries SHALL preserve strongest concrete support across merged invocations
When multiple invocations are merged into one verifier-facing summary entry, the summary SHALL preserve concrete support from the strongest successful invocation.
#### Scenario: Merged log calls include one direct hit
- **WHEN** repeated `query_logs` calls for the same topic are merged
- **AND** at least one call contains a concrete direct evidence hit
- **THEN** the merged summary SHALL include a bounded direct-hit excerpt
- **AND** it SHALL preserve all contributing `source_invocation_ids`
#### Scenario: Truncated preview still contains direct evidence
- **WHEN** a persisted output preview is marked truncated
- **AND** the preview contains a concrete direct evidence hit
- **THEN** the summary SHALL treat the row as successful evidence
- **AND** it SHALL NOT downgrade the row to no-evidence only because the raw output was truncated
### Requirement: Live trace counts SHALL distinguish model steps from evidence tool invocations
The persisted trace SHALL make it possible to audit model step counts separately from evidence tool invocation counts.
#### Scenario: Diagnosis session exposes aggregate counts
- **WHEN** a diagnosis session completes
- **THEN** `diagnosis_session.step_count` SHALL count persisted Agent model steps
- **AND** `diagnosis_session.tool_call_count` SHALL count persisted evidence-tool invocation rows
- **AND** helper workflow calls that are not evidence rows SHALL be auditable from agent steps or logs without inflating `tool_invocation`