158 lines
4.6 KiB
Markdown
158 lines
4.6 KiB
Markdown
# Phase 5 Evidence: AIOps Run Isolation
|
|
|
|
## Scope
|
|
|
|
Phase 5 implements AIOps run isolation:
|
|
|
|
- `/api/ai_ops` allocates a `runId` before execution.
|
|
- The SSE stream keeps event name `message` and emits a JSON `SseMessage` with `type=metadata`, `sessionId`, and `runId` before content.
|
|
- `AiOpsService` creates `diagnosis_run` rows with `agent_flow=AI_OPS`.
|
|
- AIOps Agent hooks and tool recording receive `sessionId + runId` through `RunnableConfig.metadata` and `SessionContextHolder`.
|
|
- AIOps status, final report, metrics, and `self_evaluation.aiops_rule_evaluation` write to the current `diagnosis_run`.
|
|
- Historical `DiagnosisSession` final-report persistence remains available only through the legacy overload.
|
|
|
|
## Focused Tests
|
|
|
|
Focused tests:
|
|
|
|
```text
|
|
mvn -q "-Dtest=AiOpsServiceTest,ChatControllerTest,AgentLoggingHookTest,ToolInvocationRecorderTest,AiOpsRuleEvaluationServiceTest" test
|
|
```
|
|
|
|
Result: passed.
|
|
|
|
Coverage:
|
|
|
|
- AIOps final report updates `diagnosis_run.answer` and `diagnosis_run.self_evaluation`.
|
|
- Rule evaluation reads `tool_invocation` rows by `run_id`.
|
|
- AIOps run metrics count `agent_step` and `tool_invocation` rows by `run_id`.
|
|
- The same AIOps `sessionId` can create distinct `runId` values.
|
|
- SSE metadata uses `type=metadata` and carries `sessionId/runId`.
|
|
- Existing hook and tool-recorder tests cover `runId` propagation into `agent_step` and `tool_invocation`.
|
|
|
|
## E2E Runtime
|
|
|
|
Maven startup:
|
|
|
|
```text
|
|
mvn spring-boot:run -Dspring-boot.run.profiles=mvp-demo
|
|
```
|
|
|
|
Captured artifacts:
|
|
|
|
- `target/e2e/phase5-mvn-20260710-204732.out.log`
|
|
- `target/e2e/phase5-mvn-20260710-204732.err.log`
|
|
- `target/e2e/phase5-aiops-request.json`
|
|
- `target/e2e/phase5-aiops-sse-response.txt`
|
|
- `target/e2e/phase5-mvn-20260710-205204.out.log`
|
|
- `target/e2e/phase5-mvn-20260710-205204.err.log`
|
|
- `target/e2e/phase5-aiops-request-2053.json`
|
|
- `target/e2e/phase5-aiops-sse-response-2053.txt`
|
|
- `target/e2e/phase5-aiops-trace-2053.json`
|
|
|
|
The Maven process was stopped after evidence collection.
|
|
|
|
First E2E attempt exposed an existing AIOps runtime integration bug:
|
|
|
|
```text
|
|
AI Ops 流程失败: mainAgent (ReactAgent) must be provided for supervisor agent
|
|
```
|
|
|
|
Diagnosis result:
|
|
|
|
- feedback loop: fixed `/api/ai_ops` request with `mvp-demo` profile;
|
|
- root cause: current `SupervisorAgent` dependency validates that `mainAgent(ReactAgent)` is set;
|
|
- fix: `AiOpsService.buildSupervisorAgent(...)` now sets Planner as `mainAgent` and Executor as sub-agent;
|
|
- regression coverage: `AiOpsServiceTest.buildSupervisorAgentSetsPlannerAsMainAgent`.
|
|
|
|
Successful E2E:
|
|
|
|
```text
|
|
sessionId = e2e-phase5-aiops-codex-20260710-2053
|
|
runId = run-84b8c02b-4d1e-4b24-ac88-222c88c8db26
|
|
```
|
|
|
|
SSE response:
|
|
|
|
```text
|
|
event:message
|
|
data:{"type":"metadata","data":null,"sessionId":"e2e-phase5-aiops-codex-20260710-2053","runId":"run-84b8c02b-4d1e-4b24-ac88-222c88c8db26"}
|
|
|
|
...
|
|
|
|
event:message
|
|
data:{"type":"done","data":null,"sessionId":null,"runId":null}
|
|
```
|
|
|
|
Exact trace check:
|
|
|
|
```text
|
|
GET /api/diagnosis/e2e-phase5-aiops-codex-20260710-2053/trace?runId=run-84b8c02b-4d1e-4b24-ac88-222c88c8db26
|
|
```
|
|
|
|
Observed:
|
|
|
|
```text
|
|
code=200
|
|
runId=run-84b8c02b-4d1e-4b24-ac88-222c88c8db26
|
|
agentFlow=AI_OPS
|
|
summary.hasAiOpsRuleEvaluation=true
|
|
summary.persistedStepCount=1
|
|
summary.persistedToolCallCount=0
|
|
```
|
|
|
|
The E2E model did not call evidence tools. This is captured as rule-evaluation WARN rather than a run-isolation failure.
|
|
|
|
## Database Inspection
|
|
|
|
Queried through `scripts/query_mysql.py`.
|
|
|
|
`diagnosis_run`:
|
|
|
|
```text
|
|
run_id=run-84b8c02b-4d1e-4b24-ac88-222c88c8db26
|
|
session_id=e2e-phase5-aiops-codex-20260710-2053
|
|
status=SUCCESS
|
|
agent_flow=AI_OPS
|
|
has_answer=1
|
|
has_aiops_eval=1
|
|
step_count=1
|
|
tool_call_count=0
|
|
```
|
|
|
|
`agent_step` grouped by run:
|
|
|
|
```text
|
|
run-84b8c02b-4d1e-4b24-ac88-222c88c8db26 | step_rows=1
|
|
```
|
|
|
|
`tool_invocation` grouped by run:
|
|
|
|
```text
|
|
(empty; the successful E2E did not invoke evidence tools)
|
|
```
|
|
|
|
Log evidence from `phase5-mvn-20260710-205204.out.log`:
|
|
|
|
- AIOps request was received with the expected `sessionId` and `runId`.
|
|
- `AiOpsService` started analysis and invoked the supervisor agent.
|
|
- AIOps orchestration completed and final report extraction ran.
|
|
- Exact trace was queried with the same `sessionId + runId`.
|
|
|
|
## Final Gate
|
|
|
|
Final Phase 5 gate:
|
|
|
|
```text
|
|
mvn -q clean test-compile
|
|
mvn -q "-Dtest=AiOpsServiceTest,ChatControllerTest,AgentLoggingHookTest,ToolInvocationRecorderTest,AiOpsRuleEvaluationServiceTest,DiagnosisTraceServiceTest" test
|
|
openspec validate session-run-trace-isolation --strict
|
|
git diff --check
|
|
```
|
|
|
|
Result: passed.
|
|
|
|
## Conclusion
|
|
|
|
Phase 5 satisfies AIOps run creation, SSE run metadata, run-scoped execution context propagation, run-scoped final report/evaluation/metrics writes, focused tests, and Maven E2E DB/log verification.
|