refactor(trace): enforce run-only diagnosis model
This commit is contained in:
@@ -2,27 +2,28 @@
|
||||
|
||||
Provide a repeatable MVP demo flow that can run a chat diagnosis, expose its persisted execution trace, and submit feedback for the same diagnosis run.
|
||||
## Requirements
|
||||
### Requirement: Diagnosis trace can be queried by session id
|
||||
The system SHALL expose a read-only HTTP endpoint `GET /api/diagnosis/{sessionId}/trace` that returns the persisted diagnosis trace for the requested session id. When `runId` is omitted, the endpoint SHALL return the latest diagnosis run for compatibility. When `runId` is provided, the endpoint SHALL return that exact run after validating it belongs to the path `sessionId`.
|
||||
### Requirement: Diagnosis trace requires an exact run
|
||||
The system SHALL expose a read-only HTTP endpoint `GET /api/diagnosis/{sessionId}/trace?runId=...` that returns only the specified diagnosis run after validating ownership.
|
||||
|
||||
#### Scenario: Existing session latest trace is returned
|
||||
- **WHEN** a caller requests trace data for a session id that has at least one `diagnosis_run`
|
||||
- **THEN** the system returns a success response containing the resolved run id, session summary, run summary, ordered agent steps, ordered tool invocations, self-evaluation data, final answer, and feedback for the latest run
|
||||
#### Scenario: Missing runId is rejected
|
||||
- **WHEN** a caller omits `runId`
|
||||
- **THEN** the request is rejected without inferring a latest run
|
||||
|
||||
#### Scenario: Existing session exact trace is returned
|
||||
- **WHEN** a caller requests trace data with `GET /api/diagnosis/{sessionId}/trace?runId=run-xxx`
|
||||
- **THEN** the system validates that `runId` belongs to `sessionId`
|
||||
- **AND** it returns a success response containing only the trace data for that run
|
||||
- **AND** it returns chat-session metadata, the exact run, ordered steps, ordered tool invocations, self-evaluation, answer, and feedback
|
||||
- **AND** it does not return a compatibility `session` projection
|
||||
|
||||
#### Scenario: Missing session returns not found
|
||||
- **WHEN** a caller requests trace data for a session id that does not exist in `chat_session`, `diagnosis_run`, or historical compatibility data
|
||||
- **WHEN** a caller requests a sessionId/runId pair that does not exist in `diagnosis_run`
|
||||
- **THEN** the system returns a 404 response using the existing session-not-found error contract
|
||||
|
||||
### Requirement: Trace aggregation is read-only
|
||||
The system MUST build trace output from existing persisted diagnosis tables and MUST NOT mutate chat sessions, diagnosis runs, agent steps, tool invocations, feedback, or chat session state while serving the trace request.
|
||||
|
||||
#### Scenario: Trace query does not change persisted state
|
||||
- **WHEN** a caller requests `GET /api/diagnosis/{sessionId}/trace`
|
||||
- **WHEN** a caller requests `GET /api/diagnosis/{sessionId}/trace?runId=run-xxx`
|
||||
- **THEN** the system reads `diagnosis_run`, `agent_step`, and `tool_invocation` records and returns an aggregate without saving any of those records
|
||||
|
||||
#### Scenario: Exact trace query does not change persisted state
|
||||
@@ -150,4 +151,3 @@ The MVP demo SHALL provide stable scenarios that explain how to demonstrate posi
|
||||
- **WHEN** a demo scenario is fixture-backed rather than live-scripted
|
||||
- **THEN** the documentation SHALL say so explicitly
|
||||
- **AND** it SHALL avoid promising deterministic live LLM output for that scenario
|
||||
|
||||
|
||||
@@ -53,19 +53,19 @@ The system SHALL write and read `agent_step` and `tool_invocation` rows using `r
|
||||
- **THEN** its step and tool counts SHALL be calculated from rows matching that `run_id`
|
||||
- **AND** rows from other runs in the same `sessionId` SHALL NOT be counted
|
||||
|
||||
### Requirement: Trace API SHALL support latest-run and exact-run queries
|
||||
The system SHALL allow callers to query a diagnosis trace by `sessionId` alone for compatibility or by `sessionId + runId` for exact run replay.
|
||||
### Requirement: Trace API SHALL require an exact run
|
||||
The system SHALL require `sessionId + runId` for every diagnosis trace query and SHALL NOT infer a latest or historical run.
|
||||
|
||||
#### Scenario: Trace without runId resolves latest run
|
||||
#### Scenario: Trace without runId is rejected
|
||||
- **WHEN** a caller requests `GET /api/diagnosis/{sessionId}/trace` without `runId`
|
||||
- **THEN** the system SHALL resolve the latest run for that session by `diagnosis_run.created_at DESC, id DESC`
|
||||
- **AND** the response SHALL include the resolved `runId`
|
||||
- **THEN** request validation SHALL reject the request
|
||||
- **AND** the service SHALL NOT infer a run from current or historical data
|
||||
|
||||
#### Scenario: Trace with runId returns exact run
|
||||
- **WHEN** a caller requests `GET /api/diagnosis/{sessionId}/trace?runId=run-xxx`
|
||||
- **THEN** the system SHALL validate that `runId` belongs to the path `sessionId`
|
||||
- **AND** it SHALL return only the session summary, run summary, agent steps, tool invocations, self-evaluation, answer, and feedback for that run
|
||||
- **AND** the session summary SHALL come from `chat_session` metadata when available, while the run summary SHALL come from `diagnosis_run`
|
||||
- **AND** it SHALL return only chat-session metadata, run summary, agent steps, tool invocations, self-evaluation, answer, and feedback for that run
|
||||
- **AND** it SHALL NOT expose a compatibility `session` projection
|
||||
|
||||
#### Scenario: Trace rejects run from another session
|
||||
- **WHEN** a caller requests a `runId` that belongs to a different `sessionId`
|
||||
@@ -96,22 +96,11 @@ The system SHALL bind new feedback to a diagnosis run rather than an ambiguous m
|
||||
- **THEN** the system SHALL validate that the run belongs to the session
|
||||
- **AND** it SHALL update feedback on that run
|
||||
- **AND** the response SHALL include the actual bound `runId`
|
||||
- **AND** the response SHALL include `fallbackToLatestRun=false`
|
||||
|
||||
#### Scenario: Feedback without runId falls back observably
|
||||
- **WHEN** a legacy feedback request includes `sessionId` but omits `runId`
|
||||
- **AND** at least one `diagnosis_run` exists for that session
|
||||
- **THEN** the system SHALL bind feedback to the latest run for that session
|
||||
- **AND** the response SHALL include `fallbackToLatestRun=true`
|
||||
- **AND** the response SHALL include the actual bound `runId`
|
||||
|
||||
#### Scenario: Historical feedback without run-backed data remains compatible
|
||||
- **WHEN** a legacy feedback request includes `sessionId` but omits `runId`
|
||||
- **AND** no `diagnosis_run` exists for that session
|
||||
- **AND** a historical `diagnosis_session` row exists for that session
|
||||
- **THEN** the system MAY bind feedback to the historical session row for migration compatibility
|
||||
- **AND** the response SHALL NOT claim latest-run fallback
|
||||
- **AND** the response MAY omit `runId`
|
||||
#### Scenario: Feedback without runId is rejected
|
||||
- **WHEN** a feedback request includes `sessionId` but omits `runId`
|
||||
- **THEN** the system SHALL reject the request
|
||||
- **AND** it SHALL NOT bind feedback to any run
|
||||
|
||||
#### Scenario: Feedback rejects run from another session
|
||||
- **WHEN** a feedback request includes a `runId` that belongs to a different `sessionId`
|
||||
@@ -140,18 +129,13 @@ The system SHALL create and expose a diagnosis run for every valid `/api/ai_ops`
|
||||
- **AND** the metadata payload SHALL expose the created `runId`
|
||||
- **AND** report content SHALL continue to use the existing content message shape
|
||||
|
||||
### Requirement: Migration SHALL preserve historical trace access
|
||||
The system SHALL migrate historical diagnosis data into compatibility runs without deleting the old `diagnosis_session` table.
|
||||
### Requirement: Runtime SHALL use only run-based diagnosis storage
|
||||
The system SHALL use `chat_session` and `diagnosis_run` as the only runtime diagnosis model and SHALL NOT read or write `diagnosis_session`.
|
||||
|
||||
#### Scenario: Historical session gets compatibility run
|
||||
- **WHEN** migration runs on an existing `diagnosis_session` row
|
||||
- **THEN** the system SHALL create a compatible `diagnosis_run` row for that session
|
||||
- **AND** old `agent_step` and `tool_invocation` rows for that session SHALL be backfilled to that `run_id` when possible
|
||||
|
||||
#### Scenario: Old table is retained
|
||||
- **WHEN** the migration completes
|
||||
- **THEN** the `diagnosis_session` table SHALL remain available for historical comparison and rollback
|
||||
- **AND** new execution writes SHALL target `chat_session` and `diagnosis_run`
|
||||
#### Scenario: Runtime components are inspected
|
||||
- **WHEN** Trace, Feedback, Evaluation, AIOps persistence, Gatekeeper, and case creation execute
|
||||
- **THEN** they SHALL resolve data by `runId`
|
||||
- **AND** no executable entity, repository, service fallback, or test SHALL depend on `diagnosis_session`
|
||||
|
||||
### Requirement: Demo and Trace UI SHALL support runId
|
||||
The demo tooling and Trace UI SHALL support minimal run-aware workflows.
|
||||
@@ -214,14 +198,14 @@ The Trace API SHALL parse the current DiagnosisRun orchestration JSON and expose
|
||||
|
||||
- **WHEN** a caller queries a successful new StateGraph Chat run
|
||||
- **THEN** `run.orchestrationTrace` SHALL be a non-empty parsed JSON object
|
||||
- **AND** the response top level and compatibility `session` projection SHALL NOT duplicate the field
|
||||
- **AND** the response SHALL NOT contain a compatibility `session` projection
|
||||
- **AND** no raw orchestration trace field SHALL be added
|
||||
|
||||
#### Scenario: Historical or non-StateGraph run is queried
|
||||
#### Scenario: Non-StateGraph run is queried
|
||||
|
||||
- **WHEN** the selected DiagnosisRun has null orchestration trace
|
||||
- **THEN** `run.orchestrationTrace` MAY be null
|
||||
- **AND** the service SHALL NOT synthesize historical events or read another run's trace
|
||||
- **AND** the service SHALL NOT synthesize events or read another run's trace
|
||||
|
||||
### Requirement: Orchestration trace migration SHALL be additive and nullable
|
||||
|
||||
|
||||
Reference in New Issue
Block a user