feat: add aiops lightweight verifier

2026-07-05 13:44:30 +08:00
parent 2658742119
commit ed267d753d
15 changed files with 417 additions and 5 deletions
@@ -0,0 +1,59 @@
+## Context
+
+Chat diagnosis has a Verifier Agent that writes structured evaluation into `diagnosis_session.self_evaluation`. AIOps currently focuses on payload scoping, evidence tools, and trace persistence, but it has no quality gate that checks whether the final report stayed on target or used evidence.
+
+The next stage should add a low-risk quality gate before considering a full AIOps LLM verifier.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Evaluate AIOps final reports with deterministic rules.
+- Persist the evaluation under a dedicated `aiops_rule_evaluation` self-evaluation key.
+- Keep trace replay able to show whether AIOps output passed, warned, or failed basic quality checks.
+- Add focused unit tests without requiring live LLMs or external tools.
+
+**Non-Goals:**
+
+- Do not add an AIOps Verifier Agent yet.
+- Do not route/retry AIOps execution based on the evaluation result.
+- Do not change `tool_invocation` schema.
+- Do not require new database migrations.
+
+## Decisions
+
+### Decision 1: Rule-Based Before LLM Verifier
+
+The first AIOps verifier is a deterministic evaluator, not an LLM agent.
+
+Rationale:
+
+- AIOps quality risks are concrete at this stage: payload focus, evidence coverage, and report presence.
+- Rule evaluation is cheap, stable, and easy to explain in an interview.
+- A full verifier agent can be added later once AIOps trace expectations are stable.
+
+### Decision 2: Dedicated Self-Evaluation Channel
+
+Persist under `aiops_rule_evaluation` instead of reusing `rule_evaluation` or `verifier_evaluation`.
+
+Rationale:
+
+- `verifier_evaluation` is already associated with Chat's LLM verifier.
+- `rule_evaluation` may be used by generic diagnosis evaluation.
+- A dedicated key avoids conflating AIOps-specific checks with other evaluation channels.
+
+### Decision 3: Evaluate After Final Report Persistence
+
+Run the evaluator when `persistFinalReport(...)` is called.
+
+Rationale:
+
+- It has access to the final report and session id.
+- It can read persisted tool invocations for the same session.
+- It does not disturb the Agent execution path.
+
+## Risks / Trade-offs
+
+- [Risk] Rule evaluation can miss semantic hallucinations. -> Mitigation: position it as lightweight AIOps quality gate, not full groundedness verification.
+- [Risk] Strict keyword checks may warn on valid reports with different wording. -> Mitigation: use WARN for missing soft signals and FAIL only for critical absence.
+- [Risk] Evaluation after report persistence does not trigger retries. -> Mitigation: keep routing unchanged in this phase; later changes can consume the verdict.