feat(agent): harden verifier evidence references

2026-07-08 16:12:56 +08:00
parent a08672b31e
commit 7b8c75e571
29 changed files with 3426 additions and 94 deletions
@@ -0,0 +1,52 @@
+# Acceptance
+
+## Static Verification
+
+- `openspec validate verifier-evidence-reference-fidelity --strict`: passed.
+- `openspec validate --specs`: passed.
+
+## Script Verification
+
+- `mvn "-Dtest=ToolInvocationRecorderTest,ExecutorGatekeeperServiceTest,VerifierInputHookTest,QueryLogsToolsTest,ChatServiceSequentialAgentTest" test`
+  - Passed: 38 tests.
+- `mvn "-Dtest=ExecutorGatekeeperServiceTest" test`
+  - Passed: 9 tests.
+- `mvn test`
+  - Failed on unrelated environment-gated `MilvusConnectionTest.connect`: `MILVUS_TOKEN` environment variable was not set.
+  - Other executed tests in the run progressed until that single failure; focused tests for this change passed.
+
+## End-to-End Verification
+
+The Java service was restarted with `mvn spring-boot:run`; logs were written under `logs/`.
+
+| Case | Session | Result | Gatekeeper Audit |
+|---|---|---|---|
+| HikariCP positive | `iss007-hikari-positive-20260708-1553` | PASS; confirmed order-service HikariCP timeout and pool saturation logs | `pass / none`, checked_bindings=2 |
+| HikariCP negative | `iss007-hikari-negative-20260708-1555` | LOW_CONFID; no `generic-service`; no false positive for inventory-service | `fail / low_confid` |
+| HighMemoryUsage positive | `iss007-memory-positive-20260708-1558` | PASS; confirmed HighMemoryUsage 91%, did not confirm memory leak | `pass / none`, checked_bindings=1 |
+| SlowResponse positive | `iss007-slow-positive-20260708-1600` | PASS; confirmed SlowResponse and slow request logs, no DB pool root cause | `pass / none`, checked_bindings=7 |
+| Narrow HighCPUUsage | `iss007-narrow-highcpu-20260708-1602` | PASS; only covered payment-service HighCPUUsage | `pass / none`, checked_bindings=1 |
+
+## Database Audit
+
+`scripts/query_mysql.py` was used to verify:
+
+- `diagnosis_session.self_evaluation.verifier_evaluation.verdict`
+- `gatekeeper_result.status`
+- `gatekeeper_result.severity`
+- `gatekeeper_result.checked_bindings`
+- no-hit HikariCP query rows persist `evidence_status=no_evidence`
+
+## Remaining Risk
+
+- Negative no-hit claims still have incomplete precise references when Executor uses `$.logs` for empty arrays. Gatekeeper correctly downgrades to `LOW_CONFID`.
+- Prompt-only scope control is improved but not a hard contract. A future `scope_contract` may still be needed.
+- Full test suite requires `MILVUS_TOKEN` to pass `MilvusConnectionTest`.
+
+## OpenSpec Archive
+
+- `openspec archive verifier-evidence-reference-fidelity --yes`: succeeded.
+- Main specs updated:
+  - `openspec/specs/chat-verifier-agent/spec.md`
+  - `openspec/specs/evidence-trace-hardening/spec.md`
+- Non-blocking warning: proposal did not use OpenSpec's preferred `## Why` / `## What Changes` headers, but archive completed.
@@ -0,0 +1,44 @@
+# Verifier Evidence Reference Fidelity
+
+## Background
+
+ISS-007 came from end-to-end diagnosis cases where raw tool output and Executor `evidence_excerpt` contained enough facts, but Verifier still returned `LOW_CONFID` because the verifier-facing summary compressed away key details.
+
+The affected flow is:
+
+```text
+chat_planner
+  -> chat_executor
+  -> VerifierInputHook / Gatekeeper
+  -> chat_verifier
+  -> chat_composer
+```
+
+The change hardens the evidence handoff between Executor, Gatekeeper, and Verifier.
+
+## Goal
+
+Make Executor cite concrete tool evidence, make Gatekeeper validate that citation with code, and make Verifier judge whether verified evidence can derive the claim.
+
+## Scope
+
+- Persist `tool_invocation.retrieval_details.evidence_refs`.
+- Use `source_invocation_id + raw_path + evidence_excerpt` as the precise evidence binding.
+- Add Gatekeeper `severity` and checked binding audit.
+- Keep Gatekeeper in the Verifier hook path.
+- Keep `tool_trace_summary` as navigation/audit context, not the only evidence source.
+- Fix HikariCP mock positive/no-hit behavior.
+- Tighten Executor/Verifier prompts for narrow-scope evidence handling.
+
+## Non-goals
+
+- No Planner `scope_contract`.
+- No new database table.
+- No full JSONPath engine.
+- No change to external HTTP API.
+- No retry rollback from Gatekeeper to Executor in this phase.
+
+## OpenSpec
+
+- Change: `openspec/changes/verifier-evidence-reference-fidelity`
+- Source issue: `mvp/issues/ISS-007-verifier-evidence-summary-fidelity.md`
@@ -0,0 +1,54 @@
+# Decisions
+
+## Evidence Reference
+
+Use `source_invocation_id + raw_path + evidence_excerpt` as the precise evidence reference for Executor claim bindings.
+
+Reason:
+
+- Invocation ID alone only identifies a tool call, not the evidence inside it.
+- `raw_path` is enough for the first version when paired with `retrieval_details.evidence_refs`.
+- `evidence_excerpt` remains the text Verifier reads, but only after Gatekeeper validates it.
+
+## Raw Path
+
+Only support stable locators in the first version:
+
+- `$.alerts[i]`
+- `$.logs[i]`
+- `$.evidence_blocks[i]`
+
+No full JSONPath engine is introduced.
+
+## Gatekeeper Severity
+
+Gatekeeper output includes:
+
+- `status`
+- `severity`
+- `checked_bindings`
+- `failed_rules`
+- `warnings`
+- `errors`
+
+Severity meaning:
+
+- `none`: precise references passed.
+- `low_confid`: evidence is missing or incomplete, but not fabricated.
+- `reject`: fabricated ID, wrong tool, unknown raw path, or mismatched excerpt.
+
+## Verifier Boundary
+
+Verifier uses verified claim-local excerpts as primary derivability evidence. `tool_trace_summary` remains available for navigation and audit, but no longer needs to carry every concrete fact.
+
+## Hook Placement
+
+Gatekeeper remains in the Verifier input hook path. This version does not retry Executor on Gatekeeper failure.
+
+## Planner
+
+Planner is not changed. `scope_contract` remains a later-stage idea. This phase uses prompt constraints to reduce narrow-scope over-expansion.
+
+## Database
+
+No new tables. Evidence refs are stored in `tool_invocation.retrieval_details.evidence_refs`; audit is stored in `diagnosis_session.self_evaluation.verifier_evaluation.gatekeeper_result`.
@@ -0,0 +1,33 @@
+# Evidence
+
+## Existing Context
+
+- Existing `chat-verifier-agent` spec still used `source_invocation_ids` and `tool_trace_summary` as the main verifier evidence context.
+- Existing `evidence-trace-hardening` spec already established `tool_invocation.retrieval_details` as the right place for structured tool-specific facts.
+- Prior devflow projects established that runbook/skill content is guidance, not incident evidence.
+
+## Code Findings
+
+- `VerifierInputHook` previously backfilled plural `source_invocation_ids` from `tool_trace_summary` by tool name.
+- `ExecutorGatekeeperService` previously validated invocation existence and tool name, but not `raw_path` or excerpt authenticity.
+- `ToolInvocationRecorder` persisted retrieval details but did not generate claim-addressable `evidence_refs`.
+- `QueryLogsTools` could fall back to `generic-service` placeholder logs on no-hit.
+
+## Implementation Evidence
+
+- `ToolInvocationRecorder` now extracts:
+  - `$.alerts[i]` for `query_metrics`
+  - `$.logs[i]` for `query_logs`
+  - `$.evidence_blocks[i]` for `lookup_knowledge`
+- `ExecutorGatekeeperService` now validates:
+  - invocation existence
+  - tool name
+  - raw path presence
+  - `retrieval_details.evidence_refs`
+  - excerpt similarity/support
+- `VerifierInputHook` only auto-fills a singular `source_invocation_id` when exactly one candidate exists and never invents `raw_path`.
+- `QueryLogsTools` returns HikariCP mock logs for `order-service` and returns empty no-hit results for unrelated services.
+
+## Residual Finding
+
+The HikariCP negative E2E no longer has generic-service pollution, but the model still issued an extra broad HikariCP query without the service filter and used order-service as context. This is a remaining narrow-scope behavior issue, not a mock evidence pollution issue.