feat: add chat verifier agent

2026-07-03 10:54:33 +08:00
parent 4f5316d473
commit 9050487307
28 changed files with 3200 additions and 209 deletions
@@ -0,0 +1,58 @@
+# Acceptance: chat-verifier-agent
+
+## Classification
+
+standard
+
+## Task Status
+
+| Task | Status | Notes |
+| --- | --- | --- |
+| Verifier prompt | Done | Strict JSON schema, verdict matrix, fact classifications, and `evidence_refs` are defined. |
+| VerifierInputHook | Done | Explicit verifier payload replaces raw conversation history. |
+| ChatService integration | Done | Planner, executor, and verifier are called explicitly with max two rounds. |
+| Verdict routing | Done | PASS, LOW_CONFID, and REJECT paths are handled in code. |
+| Trace summary | Done | Evidence summaries include `trace_ref` and `source_invocation_ids`. |
+| self_evaluation merge | Done | `rule_evaluation` and `verifier_evaluation` are preserved independently. |
+| Verifier observability | Done | `verifier_evaluation` persists facts, evidence refs, trace summary, rationale, score, and round. |
+
+## Static Verification
+
+- [x] OpenSpec artifacts exist: `proposal.md`, `design.md`, `specs/chat-verifier-agent/spec.md`, `tasks.md`, `.committed`.
+- [x] `change.json` exists and has `metadata.status = committed`.
+- [x] `.archive-ready` exists.
+- [x] devflow archive-prep files exist: `brief.md`, `evidence.md`, `decisions.md`, `acceptance.md`.
+- [x] `devflow/index.md` contains `chat-verifier-agent` with status `archived`.
+
+## Script Verification
+
+- [x] `mvn -q -DskipTests compile` passed.
+
+## Runtime Verification
+
+- [x] POST `/api/chat` with a complex question returned successfully.
+- [x] Runtime session `9138f064` showed planner, executor, and verifier execution in logs.
+- [x] Runtime session `9138f064` wrote `verifier_evaluation.verdict = LOW_CONFID`.
+- [x] Runtime session `9138f064` wrote `facts_checked[*].evidence_refs`.
+- [x] Runtime session `9138f064` wrote `tool_trace_summary[*].source_invocation_ids`.
+- [x] LOW_CONFID final answer included disclaimer and verifier-derived evidence gaps.
+
+## Unverified
+
+| Scenario | Reason | Risk | Follow-up |
+| --- | --- | --- | --- |
+| PASS runtime path | The exercised complex runtime case produced LOW_CONFID. | Low; PASS routing is simple pass-through after parsed verifier decision. | Add a fixture or deterministic verifier test if this becomes product-critical. |
+| REJECT runtime path | No forced contradiction case was run after traceability changes. | Medium; REJECT is the safety-critical degraded path. | Add a targeted test with a fabricated claim and evidence contradiction. |
+| Document-path-level evidence mapping | Current implementation records invocation ids and source document labels, not guaranteed canonical document paths for every retrieval mode. | Low for current audit need; medium for future UI drill-down. | Extend retrieval details with canonical document paths in a later change. |
+
+## Remaining Risks
+
+1. Verifier output still depends on model compliance with JSON schema; code falls back to LOW_CONFID on missing or invalid output.
+2. `AgentLoggingHook` is shared by several agent paths; current changes preserve compile and runtime behavior but should be watched in AiOps flows.
+3. `SupervisorAgent` construction remains as legacy residue in `ChatService`; runtime orchestration is explicit, but a later cleanup should remove unused supervisor construction.
+
+## Archive State
+
+- [x] OpenSpec change is archive-ready.
+- [x] OpenSpec change has been moved to `openspec/changes/archive/2026-07-03-chat-verifier-agent/`.
+- [x] Main spec exists at `openspec/specs/chat-verifier-agent/spec.md`.