feat(trace): add diagnosis trace workbench
This commit is contained in:
@@ -1,9 +1,7 @@
|
||||
## Purpose
|
||||
|
||||
Provide a repeatable MVP demo flow that can run a chat diagnosis, expose its persisted execution trace, and submit feedback for the same session id.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Diagnosis trace can be queried by session id
|
||||
The system SHALL expose a read-only HTTP endpoint `GET /api/diagnosis/{sessionId}/trace` that returns the persisted diagnosis trace for the requested session id.
|
||||
|
||||
@@ -64,3 +62,39 @@ The MVP demo SHALL document which trace fields to inspect for evidence, verifier
|
||||
#### Scenario: Checklist maps fields to interview claims
|
||||
- **WHEN** a developer reviews a trace response
|
||||
- **THEN** the checklist SHALL map concrete JSON paths to the claims made in the interview walkthrough
|
||||
|
||||
### Requirement: MVP demo SHALL provide a browser trace workbench
|
||||
The MVP demo SHALL provide a browser-accessible static page for inspecting one
|
||||
diagnosis trace by session id using the existing read-only Trace API.
|
||||
|
||||
#### Scenario: Existing trace renders in the workbench
|
||||
- **WHEN** a reviewer opens the Trace workbench with a session id that exists
|
||||
- **THEN** the page SHALL request `GET /api/diagnosis/{sessionId}/trace`
|
||||
- **AND** it SHALL render session summary, ordered agent steps, ordered tool
|
||||
invocations, verifier evaluation, and final answer when present
|
||||
|
||||
#### Scenario: Trace workbench handles missing or failed traces
|
||||
- **WHEN** the Trace API returns an error or the session id is empty
|
||||
- **THEN** the page SHALL show a clear error or empty state without mutating any
|
||||
diagnosis data
|
||||
|
||||
### Requirement: Trace workbench SHALL expose evidence and skill boundaries
|
||||
The Trace workbench SHALL make tool evidence and skill-loading boundaries visible
|
||||
without requiring raw JSON inspection first.
|
||||
|
||||
#### Scenario: Tool and verifier evidence are inspectable
|
||||
- **WHEN** a trace includes tool invocations and verifier facts
|
||||
- **THEN** the page SHALL show tool invocation counts, success status, tool
|
||||
filters, verifier facts, and evidence references
|
||||
|
||||
#### Scenario: RAG details are inspectable for lookup knowledge calls
|
||||
- **WHEN** a `lookup_knowledge` invocation includes retrieval details
|
||||
- **THEN** the page SHALL show query transform, retrieval trace, context pack,
|
||||
rerank trace, and evidence block data where available
|
||||
|
||||
#### Scenario: Skill boundary checks are visible
|
||||
- **WHEN** a trace includes planner, executor, or verifier steps
|
||||
- **THEN** the page SHALL show best-effort indicators for selected skill,
|
||||
planner `read_skill` text mentions, executor `read_skill` text mentions, and
|
||||
verifier `read_skill` text mentions
|
||||
|
||||
|
||||
Reference in New Issue
Block a user