feat(trace): isolate aiops runs
This commit is contained in:
zhuyongxin
@@ -2,7 +2,7 @@
|
||||
|
||||
## sm-flow State
|
||||
|
||||
- Checkpoint: Apply / Phase 5 ready
|
||||
- Checkpoint: Apply / Phase 6 ready
|
||||
- Scale: complex
|
||||
- Capability source: sm-flow built-in protocol for context/proposal; grill decisions are recorded from the confirmed user discussion in the issue thread.
|
||||
- Change slug: `session-run-trace-isolation`
|
||||
@@ -179,3 +179,13 @@ Audit conclusions:
|
||||
- Added `CaseLibraryService.createFromRun`, using `diagnosis_run.run_id` as the new automatic `case_library.diagnosis_id`; `createFromSession` remains the legacy session-id path.
|
||||
- Phase 4 evidence is recorded in `phase-4-evidence.md`.
|
||||
- Document review follow-up: clarified that the legacy `DiagnosisSession` feedback path only applies when no `diagnosis_run` exists for the session. It returns no bound `runId` and is not the same as latest-run fallback.
|
||||
|
||||
## Phase 5 Apply Notes
|
||||
|
||||
- Capability source: `openspec-apply-change` + sm-flow apply protocol. `codebase-retrieval` and LSP tools remain unavailable; call-chain confirmation used OpenSpec context, `rg`, targeted file reads, dependency method inspection with `javap`, focused tests, E2E, DB inspection, and logs.
|
||||
- Changed `/api/ai_ops` to allocate a `runId` before execution and emit a JSON `SseMessage` with `type=metadata`, `sessionId`, and `runId` on SSE event name `message`.
|
||||
- Changed `AiOpsService` to create one `diagnosis_run` with `agent_flow=AI_OPS` for each valid execution instead of writing new execution state to `diagnosis_session`.
|
||||
- Changed AIOps execution context propagation to pass `sessionId/runId` through both `RunnableConfig.metadata` and `SessionContextHolder`.
|
||||
- Changed AIOps final report, metrics, status, and `aiops_rule_evaluation` persistence to update the current `diagnosis_run`.
|
||||
- Preserved `persistFinalReport(sessionId, finalReport, request)` as a historical `DiagnosisSession` compatibility path.
|
||||
- Added focused tests for run-scoped final report evaluation, run-scoped metrics, distinct runs under the same AIOps session, and SSE metadata shape.
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
# Phase 5 Evidence: AIOps Run Isolation
|
||||
|
||||
## Scope
|
||||
|
||||
Phase 5 implements AIOps run isolation:
|
||||
|
||||
- `/api/ai_ops` allocates a `runId` before execution.
|
||||
- The SSE stream keeps event name `message` and emits a JSON `SseMessage` with `type=metadata`, `sessionId`, and `runId` before content.
|
||||
- `AiOpsService` creates `diagnosis_run` rows with `agent_flow=AI_OPS`.
|
||||
- AIOps Agent hooks and tool recording receive `sessionId + runId` through `RunnableConfig.metadata` and `SessionContextHolder`.
|
||||
- AIOps status, final report, metrics, and `self_evaluation.aiops_rule_evaluation` write to the current `diagnosis_run`.
|
||||
- Historical `DiagnosisSession` final-report persistence remains available only through the legacy overload.
|
||||
|
||||
## Focused Tests
|
||||
|
||||
Focused tests:
|
||||
|
||||
```text
|
||||
mvn -q "-Dtest=AiOpsServiceTest,ChatControllerTest,AgentLoggingHookTest,ToolInvocationRecorderTest,AiOpsRuleEvaluationServiceTest" test
|
||||
```
|
||||
|
||||
Result: passed.
|
||||
|
||||
Coverage:
|
||||
|
||||
- AIOps final report updates `diagnosis_run.answer` and `diagnosis_run.self_evaluation`.
|
||||
- Rule evaluation reads `tool_invocation` rows by `run_id`.
|
||||
- AIOps run metrics count `agent_step` and `tool_invocation` rows by `run_id`.
|
||||
- The same AIOps `sessionId` can create distinct `runId` values.
|
||||
- SSE metadata uses `type=metadata` and carries `sessionId/runId`.
|
||||
- Existing hook and tool-recorder tests cover `runId` propagation into `agent_step` and `tool_invocation`.
|
||||
|
||||
## E2E Runtime
|
||||
|
||||
Maven startup:
|
||||
|
||||
```text
|
||||
mvn spring-boot:run -Dspring-boot.run.profiles=mvp-demo
|
||||
```
|
||||
|
||||
Captured artifacts:
|
||||
|
||||
- `target/e2e/phase5-mvn-20260710-204732.out.log`
|
||||
- `target/e2e/phase5-mvn-20260710-204732.err.log`
|
||||
- `target/e2e/phase5-aiops-request.json`
|
||||
- `target/e2e/phase5-aiops-sse-response.txt`
|
||||
- `target/e2e/phase5-mvn-20260710-205204.out.log`
|
||||
- `target/e2e/phase5-mvn-20260710-205204.err.log`
|
||||
- `target/e2e/phase5-aiops-request-2053.json`
|
||||
- `target/e2e/phase5-aiops-sse-response-2053.txt`
|
||||
- `target/e2e/phase5-aiops-trace-2053.json`
|
||||
|
||||
The Maven process was stopped after evidence collection.
|
||||
|
||||
First E2E attempt exposed an existing AIOps runtime integration bug:
|
||||
|
||||
```text
|
||||
AI Ops 流程失败: mainAgent (ReactAgent) must be provided for supervisor agent
|
||||
```
|
||||
|
||||
Diagnosis result:
|
||||
|
||||
- feedback loop: fixed `/api/ai_ops` request with `mvp-demo` profile;
|
||||
- root cause: current `SupervisorAgent` dependency validates that `mainAgent(ReactAgent)` is set;
|
||||
- fix: `AiOpsService.buildSupervisorAgent(...)` now sets Planner as `mainAgent` and Executor as sub-agent;
|
||||
- regression coverage: `AiOpsServiceTest.buildSupervisorAgentSetsPlannerAsMainAgent`.
|
||||
|
||||
Successful E2E:
|
||||
|
||||
```text
|
||||
sessionId = e2e-phase5-aiops-codex-20260710-2053
|
||||
runId = run-84b8c02b-4d1e-4b24-ac88-222c88c8db26
|
||||
```
|
||||
|
||||
SSE response:
|
||||
|
||||
```text
|
||||
event:message
|
||||
data:{"type":"metadata","data":null,"sessionId":"e2e-phase5-aiops-codex-20260710-2053","runId":"run-84b8c02b-4d1e-4b24-ac88-222c88c8db26"}
|
||||
|
||||
...
|
||||
|
||||
event:message
|
||||
data:{"type":"done","data":null,"sessionId":null,"runId":null}
|
||||
```
|
||||
|
||||
Exact trace check:
|
||||
|
||||
```text
|
||||
GET /api/diagnosis/e2e-phase5-aiops-codex-20260710-2053/trace?runId=run-84b8c02b-4d1e-4b24-ac88-222c88c8db26
|
||||
```
|
||||
|
||||
Observed:
|
||||
|
||||
```text
|
||||
code=200
|
||||
runId=run-84b8c02b-4d1e-4b24-ac88-222c88c8db26
|
||||
agentFlow=AI_OPS
|
||||
summary.hasAiOpsRuleEvaluation=true
|
||||
summary.persistedStepCount=1
|
||||
summary.persistedToolCallCount=0
|
||||
```
|
||||
|
||||
The E2E model did not call evidence tools. This is captured as rule-evaluation WARN rather than a run-isolation failure.
|
||||
|
||||
## Database Inspection
|
||||
|
||||
Queried through `scripts/query_mysql.py`.
|
||||
|
||||
`diagnosis_run`:
|
||||
|
||||
```text
|
||||
run_id=run-84b8c02b-4d1e-4b24-ac88-222c88c8db26
|
||||
session_id=e2e-phase5-aiops-codex-20260710-2053
|
||||
status=SUCCESS
|
||||
agent_flow=AI_OPS
|
||||
has_answer=1
|
||||
has_aiops_eval=1
|
||||
step_count=1
|
||||
tool_call_count=0
|
||||
```
|
||||
|
||||
`agent_step` grouped by run:
|
||||
|
||||
```text
|
||||
run-84b8c02b-4d1e-4b24-ac88-222c88c8db26 | step_rows=1
|
||||
```
|
||||
|
||||
`tool_invocation` grouped by run:
|
||||
|
||||
```text
|
||||
(empty; the successful E2E did not invoke evidence tools)
|
||||
```
|
||||
|
||||
Log evidence from `phase5-mvn-20260710-205204.out.log`:
|
||||
|
||||
- AIOps request was received with the expected `sessionId` and `runId`.
|
||||
- `AiOpsService` started analysis and invoked the supervisor agent.
|
||||
- AIOps orchestration completed and final report extraction ran.
|
||||
- Exact trace was queried with the same `sessionId + runId`.
|
||||
|
||||
## Final Gate
|
||||
|
||||
Final Phase 5 gate:
|
||||
|
||||
```text
|
||||
mvn -q clean test-compile
|
||||
mvn -q "-Dtest=AiOpsServiceTest,ChatControllerTest,AgentLoggingHookTest,ToolInvocationRecorderTest,AiOpsRuleEvaluationServiceTest,DiagnosisTraceServiceTest" test
|
||||
openspec validate session-run-trace-isolation --strict
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Result: passed.
|
||||
|
||||
## Conclusion
|
||||
|
||||
Phase 5 satisfies AIOps run creation, SSE run metadata, run-scoped execution context propagation, run-scoped final report/evaluation/metrics writes, focused tests, and Maven E2E DB/log verification.
|
||||
@@ -42,12 +42,12 @@
|
||||
|
||||
## 5. AIOps Run Isolation
|
||||
|
||||
- [ ] 5.1 Change valid `/api/ai_ops` executions to create `diagnosis_run` with `agent_flow=AI_OPS`.
|
||||
- [ ] 5.2 Expose `runId` in the AIOps SSE-compatible metadata stream while preserving existing report streaming.
|
||||
- [ ] 5.3 Propagate `runId` through AIOps Agent hooks and tool recording.
|
||||
- [ ] 5.4 Change AIOps final report, status, counts, and `diagnosis_run.self_evaluation.aiops_rule_evaluation` writes to the current run.
|
||||
- [ ] 5.5 Add tests for repeated AIOps executions with the same `sessionId` and run-scoped rule evaluation.
|
||||
- [ ] 5.6 Phase 5 gate: run focused AIOps tests and E2E when needed, inspect DB/logs, update task status, archive phase evidence, and commit before starting Phase 6.
|
||||
- [x] 5.1 Change valid `/api/ai_ops` executions to create `diagnosis_run` with `agent_flow=AI_OPS`.
|
||||
- [x] 5.2 Expose `runId` in the AIOps SSE-compatible metadata stream while preserving existing report streaming.
|
||||
- [x] 5.3 Propagate `runId` through AIOps Agent hooks and tool recording.
|
||||
- [x] 5.4 Change AIOps final report, status, counts, and `diagnosis_run.self_evaluation.aiops_rule_evaluation` writes to the current run.
|
||||
- [x] 5.5 Add tests for repeated AIOps executions with the same `sessionId` and run-scoped rule evaluation.
|
||||
- [x] 5.6 Phase 5 gate: run focused AIOps tests and E2E when needed, inspect DB/logs, update task status, archive phase evidence, and commit before starting Phase 6.
|
||||
|
||||
## 6. Demo, Trace UI, Documentation, and Verification
|
||||
|
||||
|
||||
Reference in New Issue
Block a user