feat(agent): harden verifier evidence references
This commit is contained in:
aruo
@@ -0,0 +1,73 @@
|
||||
# Decisions
|
||||
|
||||
## Discover Context
|
||||
|
||||
- Source issue: `mvp/issues/ISS-007-verifier-evidence-summary-fidelity.md`.
|
||||
- Related existing specs: `chat-verifier-agent`, `evidence-trace-hardening`.
|
||||
- Related devflow records: `executor-gatekeeper-hook`, `executor-verifier-claim-checks`, `executor-composer-final-answer`, `evidence-trace-hardening`.
|
||||
- Current repo instruction requested semantic code search and LSP confirmation before code changes; those tools are not exposed in this environment, so implementation will use `rg`, direct code reading, and focused tests as fallback evidence.
|
||||
|
||||
## Classification
|
||||
|
||||
- sm-flow scale: `standard`.
|
||||
- Reason: internal contract change across recorder, hook, Gatekeeper, verifier prompt, mock tools, tests, and E2E validation.
|
||||
|
||||
## Question Pool
|
||||
|
||||
| Question | Mode | Resolution |
|
||||
|---|---|---|
|
||||
| Should Planner change or add `scope_contract`? | user-interview, already resolved in issue | No. This change is prompt-first and does not alter Planner. |
|
||||
| Should Gatekeeper move to Executor hook for retry? | user-interview, already resolved in issue | No. Gatekeeper remains in Verifier hook path for this version. |
|
||||
| Should new DB tables be added for evidence refs or audit? | user-interview, already resolved in issue | No. Use `tool_invocation.retrieval_details.evidence_refs` and existing `self_evaluation.verifier_evaluation.gatekeeper_result`. |
|
||||
| Should `tool_trace_summary` remain the primary evidence source? | user-interview, already resolved in issue | No. It becomes navigation/audit context; verified claim-local excerpts become primary evidence for derivability. |
|
||||
| Should full JSONPath be supported? | user-interview, already resolved in issue | No. Only stable raw paths listed in design are supported. |
|
||||
|
||||
## Cross-artifact Alignment
|
||||
|
||||
| Check | Status |
|
||||
|---|---|
|
||||
| Issue background and target -> proposal | aligned |
|
||||
| Proposal scope and non-goals -> design | aligned |
|
||||
| Design protocol and risks -> specs/tasks | aligned |
|
||||
| Specs observable behavior -> tasks acceptance | aligned |
|
||||
|
||||
## Architecture Audit
|
||||
|
||||
The change keeps the existing agent orchestration and only hardens the evidence payload between Executor, Gatekeeper, and Verifier. The highest coupling risk is transition compatibility from plural `source_invocation_ids` to singular `source_invocation_id`; the implementation must accept old data but only allow precise new bindings to pass. No new DB table is introduced, reducing migration risk. Verifier prompt and effective verdict guardrails must agree on `gatekeeper_result.severity`, otherwise the model could still produce a PASS that runtime later must downgrade.
|
||||
|
||||
## Commit Gate
|
||||
|
||||
Passed on 2026-07-08.
|
||||
|
||||
- `openspec validate verifier-evidence-reference-fidelity --strict`: passed.
|
||||
- `openspec validate --specs`: passed.
|
||||
- File completeness: proposal, design, specs, tasks present.
|
||||
- Consistency: proposal -> design -> specs -> tasks aligned.
|
||||
- Apply authorization: user requested continuing implementation and fixing autonomously; proceed to Apply.
|
||||
|
||||
## Apply Verification
|
||||
|
||||
Focused tests:
|
||||
|
||||
- `mvn "-Dtest=ToolInvocationRecorderTest,ExecutorGatekeeperServiceTest,VerifierInputHookTest,QueryLogsToolsTest,ChatServiceSequentialAgentTest" test`: passed, 38 tests.
|
||||
- `mvn "-Dtest=ExecutorGatekeeperServiceTest" test`: passed, 9 tests after adding the old-invocation-without-evidence-refs case.
|
||||
|
||||
OpenSpec:
|
||||
|
||||
- `openspec validate verifier-evidence-reference-fidelity --strict`: passed.
|
||||
- `openspec validate --specs`: passed.
|
||||
|
||||
End-to-end sessions:
|
||||
|
||||
| Case | Session | Result | Gatekeeper |
|
||||
|---|---|---|---|
|
||||
| HikariCP positive | `iss007-hikari-positive-20260708-1553` | `PASS`; answer confirmed order-service HikariCP timeout and pool saturation logs | `pass / none`, checked_bindings=2 |
|
||||
| HikariCP negative | `iss007-hikari-negative-20260708-1555` | `LOW_CONFID`; no `generic-service` pollution; no positive inventory-service confirmation | `fail / low_confid` due incomplete negative evidence references |
|
||||
| HighMemoryUsage positive | `iss007-memory-positive-20260708-1558` | `PASS`; answer confirmed HighMemoryUsage 91% without confirming memory leak | `pass / none`, checked_bindings=1 |
|
||||
| SlowResponse positive | `iss007-slow-positive-20260708-1600` | `PASS`; answer confirmed SlowResponse and slow request logs without DB-pool root cause | `pass / none`, checked_bindings=7 |
|
||||
| Narrow HighCPUUsage | `iss007-narrow-highcpu-20260708-1602` | `PASS`; answer only covered payment-service HighCPUUsage | `pass / none`, checked_bindings=1 |
|
||||
|
||||
Residual observation:
|
||||
|
||||
- HikariCP negative no longer returns `generic-service`, and no-hit rows persist `evidence_status=no_evidence`.
|
||||
- The model still issued an extra broad HikariCP query without the `inventory-service` filter and used order-service as a context claim. This is a remaining narrow-scope behavior issue, not a mock data pollution issue. It is acceptable for this change because the final verdict did not become a false positive for inventory-service.
|
||||
@@ -0,0 +1,180 @@
|
||||
# Design
|
||||
|
||||
## Data Flow
|
||||
|
||||
```text
|
||||
Evidence tool result
|
||||
-> ToolInvocationRecorder
|
||||
persists tool_invocation.retrieval_details.evidence_refs
|
||||
-> Executor
|
||||
emits claim-local evidence_bindings
|
||||
-> VerifierInputHook
|
||||
preserves structured payload and runs Gatekeeper
|
||||
-> ExecutorGatekeeperService
|
||||
validates reference authenticity
|
||||
-> Verifier
|
||||
checks derivability from verified excerpts
|
||||
-> ChatService / Composer
|
||||
enforces effective verdict and safe final answer
|
||||
```
|
||||
|
||||
## Evidence Reference Contract
|
||||
|
||||
`tool_invocation.retrieval_details.evidence_refs` is an array of minimal evidence references:
|
||||
|
||||
```json
|
||||
{
|
||||
"evidence_refs": [
|
||||
{
|
||||
"raw_path": "$.alerts[0]",
|
||||
"text": "HighMemoryUsage firing, service=order-service, current=91%, duration=15m"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Supported `raw_path` formats in this change:
|
||||
|
||||
- `$.alerts[i]` for `query_metrics`
|
||||
- `$.logs[i]` for `query_logs`
|
||||
- `$.evidence_blocks[i]` for `lookup_knowledge`
|
||||
|
||||
Unsupported in this change:
|
||||
|
||||
- Deep JSONPath such as `$.alerts[1].description`
|
||||
- Nested aliases such as `$.retrieval_details.evidence_blocks[0]`
|
||||
- Filter expressions
|
||||
- Tool-specific metadata beyond `raw_path` and `text`
|
||||
|
||||
## Executor Binding Contract
|
||||
|
||||
Executor V2 claim bindings should use:
|
||||
|
||||
```json
|
||||
{
|
||||
"tool_name": "query_metrics",
|
||||
"source_invocation_id": 12345,
|
||||
"raw_path": "$.alerts[0]",
|
||||
"evidence_excerpt": "HighMemoryUsage firing, service=order-service, current=91%, duration=15m"
|
||||
}
|
||||
```
|
||||
|
||||
Compatibility:
|
||||
|
||||
- Legacy `source_invocation_ids` may still be read for transition.
|
||||
- New precise validation requires singular `source_invocation_id` plus `raw_path`.
|
||||
- Missing `raw_path` is `LOW_CONFID`, not `PASS`.
|
||||
|
||||
## Gatekeeper Severity
|
||||
|
||||
Gatekeeper output includes:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "fail",
|
||||
"severity": "reject",
|
||||
"checked_bindings": [],
|
||||
"failed_rules": [],
|
||||
"warnings": [],
|
||||
"errors": []
|
||||
}
|
||||
```
|
||||
|
||||
Severity mapping:
|
||||
|
||||
- `status=pass`, `severity=none`: all checked bindings are authentic.
|
||||
- `status=fail`, `severity=low_confid`: evidence is missing or incomplete but not fabricated.
|
||||
- `status=fail`, `severity=reject`: Executor cites fabricated or mismatched evidence.
|
||||
|
||||
`reject` cases:
|
||||
|
||||
- Invocation ID does not exist.
|
||||
- Invocation belongs to another session.
|
||||
- Tool name mismatches persisted invocation.
|
||||
- `raw_path` is not present in `retrieval_details.evidence_refs`.
|
||||
- `evidence_excerpt` clearly mismatches the system-side `text`.
|
||||
|
||||
`low_confid` cases:
|
||||
|
||||
- Confirmed claim has empty `evidence_bindings`.
|
||||
- `source_invocation_id` exists but `raw_path` is missing.
|
||||
- Old invocation lacks `evidence_refs`.
|
||||
- `evidence_excerpt` is too short or generic to compare.
|
||||
- Output is incomplete without concrete fabricated IDs or paths.
|
||||
|
||||
## Verifier Behavior
|
||||
|
||||
Verifier should treat `gatekeeper_result.severity` as a hard boundary:
|
||||
|
||||
- `reject`: effective result cannot be `PASS`; fabricated-reference cases should become `REJECT`.
|
||||
- `low_confid`: effective result cannot be `PASS`; missing-reference cases should become `LOW_CONFID`.
|
||||
- `none`: Verifier judges whether claim text is derivable from verified excerpts.
|
||||
|
||||
`tool_trace_summary` remains available as navigation and audit context, but claim-local verified excerpts are the primary evidence for derivability.
|
||||
|
||||
## Hook Placement
|
||||
|
||||
Gatekeeper remains in the Verifier hook path for this change. Retry or rollback into Executor is not implemented in this stage.
|
||||
|
||||
Existing ad hoc verifier-hook validation should be replaced by Gatekeeper output. The hook may normalize compatibility fields, but it should not independently decide pass/fail outside Gatekeeper semantics.
|
||||
|
||||
## Auto-backfill
|
||||
|
||||
`VerifierInputHook` may only auto-fill a missing `source_invocation_id` when exactly one invocation candidate exists for the binding's `tool_name`.
|
||||
|
||||
Rules:
|
||||
|
||||
- Never bulk-fill multiple IDs.
|
||||
- Never auto-fill `raw_path`.
|
||||
- Add a warning when auto-fill occurs.
|
||||
- Auto-filled binding without `raw_path` must remain `LOW_CONFID`.
|
||||
|
||||
## Prompt Constraints
|
||||
|
||||
Executor prompt changes are prompt-first, contract-later:
|
||||
|
||||
- Executor is an evidence collector and micro-fact extractor.
|
||||
- Narrow-scope questions should usually produce one claim and at most two claims.
|
||||
- Do not limit evidence binding count.
|
||||
- Emit observation or negative observation, not root-cause certainty, for narrow confirmation questions.
|
||||
- Runbook and skill content cannot become current incident facts.
|
||||
- Recommended actions, if any, are evidence-collection next steps, not remediation actions.
|
||||
|
||||
## HikariCP Mock Behavior
|
||||
|
||||
`query_logs` should support positive mock hits for:
|
||||
|
||||
- `HikariCP`
|
||||
- `HikariPool`
|
||||
- `connection pool`
|
||||
- `数据库连接池`
|
||||
- `连接池耗尽`
|
||||
- `active=50/50`
|
||||
- `waiting`
|
||||
- `request timed out after 30000ms`
|
||||
- `order-service`
|
||||
|
||||
Positive output should include order-service HikariCP log records. No-hit should return `logs=[]` and `evidence_status=no_evidence`, not `generic-service` placeholder logs.
|
||||
|
||||
## Audit
|
||||
|
||||
Gatekeeper result must be persisted under:
|
||||
|
||||
```text
|
||||
diagnosis_session.self_evaluation.verifier_evaluation.gatekeeper_result
|
||||
```
|
||||
|
||||
The minimum persisted fields are:
|
||||
|
||||
- `status`
|
||||
- `severity`
|
||||
- `checked_bindings`
|
||||
- `failed_rules`
|
||||
- `warnings`
|
||||
- `errors`
|
||||
|
||||
## Interface Impact
|
||||
|
||||
Impact level: L2 internal contract change.
|
||||
|
||||
The change affects internal Agent payload JSON, persisted audit JSON, prompts, and test fixtures. It does not add public HTTP endpoints or new database tables.
|
||||
@@ -0,0 +1,62 @@
|
||||
# verifier-evidence-reference-fidelity
|
||||
|
||||
## Problem
|
||||
|
||||
Recent end-to-end checks show that some narrow diagnosis questions still become `LOW_CONFID` even when the raw tool output and Executor evidence excerpts contain enough concrete evidence. The failure is caused by evidence being compressed or lost before Verifier reasoning, plus mock log no-hit behavior that can return `generic-service` placeholder logs.
|
||||
|
||||
Current weak points:
|
||||
|
||||
- Verifier still relies too heavily on `tool_trace_summary.output_summary`.
|
||||
- Executor evidence bindings identify tool invocations but do not precisely locate evidence inside the invocation output.
|
||||
- Gatekeeper validates invocation IDs and tool names, but does not yet verify `raw_path` and excerpt fidelity.
|
||||
- `query_logs` mock data cannot reliably produce a positive HikariCP connection-pool exhaustion path and can pollute no-hit results with placeholder logs.
|
||||
- Narrow-scope Executor answers can still over-expand into unrelated claims.
|
||||
|
||||
## Proposed Change
|
||||
|
||||
Introduce a claim-local evidence reference protocol:
|
||||
|
||||
```text
|
||||
tool raw output
|
||||
-> ToolInvocationRecorder stores retrieval_details.evidence_refs
|
||||
-> Executor outputs claim + source_invocation_id + raw_path + evidence_excerpt
|
||||
-> Gatekeeper verifies that the reference is real
|
||||
-> Verifier judges whether verified evidence can derive the claim
|
||||
-> Composer only expresses Verifier-allowed material
|
||||
```
|
||||
|
||||
The first implementation keeps the orchestration unchanged. Gatekeeper remains in the Verifier input hook path. Planner `scope_contract` is out of scope for this change.
|
||||
|
||||
## Scope
|
||||
|
||||
- Add minimal `retrieval_details.evidence_refs` extraction for `query_metrics`, `query_logs`, and `lookup_knowledge`.
|
||||
- Tighten Executor evidence bindings to prefer singular `source_invocation_id`, stable `raw_path`, and `evidence_excerpt`.
|
||||
- Extend Gatekeeper to validate `source_invocation_id + raw_path + evidence_excerpt`.
|
||||
- Add Gatekeeper severity: `none`, `low_confid`, `reject`.
|
||||
- Make Verifier consume verified `evidence_excerpt` as the primary claim-local evidence.
|
||||
- Tighten `VerifierInputHook` auto-backfill: only unique invocation candidate, never `raw_path`, and no `PASS` without a precise reference.
|
||||
- Fix HikariCP mock log matching and no-hit behavior.
|
||||
- Update Executor and Verifier prompts for narrow-scope and derivability behavior.
|
||||
- Add focused tests and end-to-end checks for the minimum acceptance matrix.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Do not change Planner output.
|
||||
- Do not implement Planner `scope_contract`.
|
||||
- Do not add new database tables.
|
||||
- Do not implement a general JSONPath engine.
|
||||
- Do not make `tool_trace_summary` the primary evidence source again.
|
||||
- Do not allow Executor `diagnosis_summary` or `user_facing_answer` to re-enter the V2 contract.
|
||||
|
||||
## Context Constraints
|
||||
|
||||
- `tool_invocation.retrieval_details` is the preferred place for tool-specific structured details.
|
||||
- `DiagnosisSession.selfEvaluation.verifier_evaluation.gatekeeper_result` is the existing audit container and must be preserved.
|
||||
- `read_skill` / runbook guidance is not incident evidence.
|
||||
- Existing Composer routing must keep raw Executor JSON out of normal user answers.
|
||||
|
||||
## Risks
|
||||
|
||||
- Existing tests or prompts may still assume plural `source_invocation_ids`.
|
||||
- Some old invocations will not have `evidence_refs`; those must downgrade to `LOW_CONFID`, not `PASS`.
|
||||
- Similarity checks must tolerate formatting changes without accepting unrelated text.
|
||||
+110
@@ -0,0 +1,110 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Executor evidence bindings SHALL support precise evidence references
|
||||
Executor V2 evidence bindings SHALL support precise evidence references that locate evidence inside a persisted tool invocation.
|
||||
|
||||
#### Scenario: Precise evidence binding contains invocation path and excerpt
|
||||
- **WHEN** Executor binds evidence to a claim
|
||||
- **THEN** the binding SHOULD include singular `source_invocation_id`
|
||||
- **AND** the binding SHOULD include `raw_path`
|
||||
- **AND** the binding SHALL include `tool_name` and `evidence_excerpt`
|
||||
- **AND** the `raw_path` SHALL be interpreted relative to the referenced tool invocation's `retrieval_details.evidence_refs`
|
||||
|
||||
#### Scenario: Legacy plural invocation ids remain compatibility only
|
||||
- **WHEN** Executor emits legacy `source_invocation_ids`
|
||||
- **THEN** the system MAY read them for compatibility
|
||||
- **AND** they SHALL NOT be sufficient for a precise Gatekeeper pass without `raw_path`
|
||||
|
||||
### Requirement: Gatekeeper SHALL validate evidence reference fidelity
|
||||
Gatekeeper SHALL validate that Executor evidence bindings point to real current-session evidence references before Verifier uses them as primary evidence.
|
||||
|
||||
#### Scenario: Valid precise binding passes
|
||||
- **WHEN** a binding's `source_invocation_id` exists in the current session
|
||||
- **AND** the binding's `tool_name` matches the persisted invocation
|
||||
- **AND** the binding's `raw_path` exists in `retrieval_details.evidence_refs`
|
||||
- **AND** the binding's `evidence_excerpt` is supported by the matching evidence ref text
|
||||
- **THEN** Gatekeeper SHALL return `status=pass`
|
||||
- **AND** Gatekeeper SHALL return `severity=none`
|
||||
|
||||
#### Scenario: Missing raw path is low confidence
|
||||
- **WHEN** a binding references an existing invocation
|
||||
- **AND** the binding omits `raw_path`
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=low_confid`
|
||||
- **AND** the effective verifier result SHALL NOT be `PASS`
|
||||
|
||||
#### Scenario: Old invocation without evidence refs is low confidence
|
||||
- **WHEN** a binding references an existing invocation
|
||||
- **AND** the invocation does not contain `retrieval_details.evidence_refs`
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=low_confid`
|
||||
- **AND** the effective verifier result SHALL NOT be `PASS`
|
||||
|
||||
#### Scenario: Unknown raw path is rejected
|
||||
- **WHEN** a binding references an existing invocation
|
||||
- **AND** the binding's `raw_path` is absent from that invocation's `retrieval_details.evidence_refs`
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=reject`
|
||||
- **AND** `failed_rules` SHALL include `evidence.raw_path`
|
||||
|
||||
#### Scenario: Mismatched excerpt is rejected
|
||||
- **WHEN** a binding references an existing invocation and raw path
|
||||
- **AND** the binding's `evidence_excerpt` is not supported by the matching system-side evidence ref text
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=reject`
|
||||
- **AND** `failed_rules` SHALL include `evidence.excerpt_mismatch`
|
||||
|
||||
### Requirement: Verifier SHALL use verified claim-local evidence for derivability
|
||||
Verifier SHALL judge structured claims primarily against Gatekeeper-verified claim-local evidence excerpts.
|
||||
|
||||
#### Scenario: Verified excerpt supports direct observation
|
||||
- **WHEN** `gatekeeper_result.severity=none`
|
||||
- **AND** a claim's verified evidence excerpts directly contain the claim's concrete facts
|
||||
- **THEN** Verifier MAY classify that claim as `direct_observation`
|
||||
|
||||
#### Scenario: Tool trace summary is navigation context
|
||||
- **WHEN** `executor_structured_output.claims[].evidence_bindings` are available
|
||||
- **THEN** Verifier SHALL use `tool_trace_summary` as navigation and audit context
|
||||
- **AND** it SHALL NOT require `tool_trace_summary.output_summary` to contain every fact already present in verified claim-local evidence
|
||||
|
||||
### Requirement: Gatekeeper severity SHALL constrain effective verdict
|
||||
Runtime effective verdict calculation SHALL treat Gatekeeper severity as a hard upper bound.
|
||||
|
||||
#### Scenario: Reject severity prevents PASS
|
||||
- **WHEN** `gatekeeper_result.severity=reject`
|
||||
- **AND** the Verifier model returns `verdict=PASS`
|
||||
- **THEN** ChatService SHALL downgrade the effective verdict
|
||||
- **AND** the effective verdict SHALL be `REJECT`
|
||||
|
||||
#### Scenario: Low confidence severity prevents PASS
|
||||
- **WHEN** `gatekeeper_result.severity=low_confid`
|
||||
- **AND** the Verifier model returns `verdict=PASS`
|
||||
- **THEN** ChatService SHALL downgrade the effective verdict
|
||||
- **AND** the effective verdict SHALL be `LOW_CONFID`
|
||||
|
||||
#### Scenario: Gatekeeper audit includes severity
|
||||
- **WHEN** verifier evaluation is persisted
|
||||
- **THEN** `diagnosis_session.self_evaluation.verifier_evaluation.gatekeeper_result` SHALL include `status`, `severity`, `checked_bindings`, `failed_rules`, `warnings`, and `errors`
|
||||
|
||||
### Requirement: Verifier input hook SHALL only perform narrow compatibility backfill
|
||||
The verifier input hook SHALL avoid converting broad tool summaries into precise evidence references.
|
||||
|
||||
#### Scenario: Unique invocation candidate may be backfilled
|
||||
- **WHEN** an evidence binding omits `source_invocation_id`
|
||||
- **AND** exactly one current-session invocation exists for the binding's `tool_name`
|
||||
- **THEN** the hook MAY backfill `source_invocation_id`
|
||||
- **AND** it SHALL add a Gatekeeper warning describing the auto-backfill
|
||||
|
||||
#### Scenario: Raw path is never backfilled
|
||||
- **WHEN** an evidence binding omits `raw_path`
|
||||
- **THEN** the hook SHALL NOT synthesize `raw_path`
|
||||
- **AND** Gatekeeper SHALL treat the binding as not precise enough to pass
|
||||
|
||||
### Requirement: Executor prompt SHALL constrain narrow-scope over-expansion
|
||||
The Executor prompt SHALL instruct Executor to keep narrow confirmation questions focused on observation-level claims.
|
||||
|
||||
#### Scenario: Narrow scope produces minimal observation claims
|
||||
- **WHEN** the user asks to confirm one specific service, alert, or symptom
|
||||
- **THEN** Executor SHOULD output the minimum necessary claims, normally one and at most two
|
||||
- **AND** those claims SHALL be `observation` or `negative_observation` unless current-session evidence proves more
|
||||
- **AND** Executor SHALL NOT emit unrelated root-cause, remediation, or excluded-topic claims as confirmed facts
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Evidence tools SHALL persist minimal evidence refs
|
||||
Evidence-bearing tool invocations SHALL persist claim-addressable evidence references in `tool_invocation.retrieval_details.evidence_refs`.
|
||||
|
||||
#### Scenario: Metrics alerts produce evidence refs
|
||||
- **WHEN** a `query_metrics` invocation returns alert entries
|
||||
- **THEN** the persisted retrieval details SHALL include one `evidence_refs` item per usable alert
|
||||
- **AND** each item SHALL include `raw_path` formatted as `$.alerts[i]`
|
||||
- **AND** each item SHALL include bounded `text` containing concrete alert facts such as alert name, state, service, current value, and duration when available
|
||||
|
||||
#### Scenario: Logs produce evidence refs
|
||||
- **WHEN** a `query_logs` invocation returns log entries
|
||||
- **THEN** the persisted retrieval details SHALL include one `evidence_refs` item per usable log
|
||||
- **AND** each item SHALL include `raw_path` formatted as `$.logs[i]`
|
||||
- **AND** each item SHALL include bounded `text` containing concrete log facts such as timestamp, level, service, and message when available
|
||||
|
||||
#### Scenario: Knowledge lookup produces evidence refs
|
||||
- **WHEN** a `lookup_knowledge` invocation returns evidence blocks
|
||||
- **THEN** the persisted retrieval details SHALL include one `evidence_refs` item per usable evidence block
|
||||
- **AND** each item SHALL include `raw_path` formatted as `$.evidence_blocks[i]`
|
||||
- **AND** each item SHALL include bounded `text` containing concrete block content, title, or source when available
|
||||
|
||||
#### Scenario: Evidence ref extraction does not infer diagnosis
|
||||
- **WHEN** the recorder creates `evidence_refs`
|
||||
- **THEN** it SHALL only copy or format concrete tool output fields
|
||||
- **AND** it SHALL NOT infer root cause, remediation, or diagnosis conclusions
|
||||
|
||||
### Requirement: Log mock no-hit semantics SHALL avoid placeholder evidence
|
||||
The log query mock SHALL distinguish positive mock evidence from no-hit results without using placeholder service logs as evidence.
|
||||
|
||||
#### Scenario: HikariCP positive query returns order-service pool evidence
|
||||
- **WHEN** a `query_logs` request targets `order-service` and HikariCP connection-pool exhaustion terms
|
||||
- **THEN** the tool SHALL return order-service HikariCP-related log entries
|
||||
- **AND** the returned evidence SHALL include concrete terms such as `HikariPool`, `active=50/50`, `waiting`, or `request timed out after 30000ms`
|
||||
- **AND** it SHALL NOT return `generic-service` placeholder logs
|
||||
|
||||
#### Scenario: HikariCP no-hit query returns no evidence
|
||||
- **WHEN** a `query_logs` request targets a service without matching HikariCP mock evidence
|
||||
- **THEN** the tool SHALL return an empty `logs` array
|
||||
- **AND** it SHALL mark the output as `evidence_status=no_evidence`
|
||||
- **AND** it SHALL NOT return `generic-service` placeholder logs
|
||||
@@ -0,0 +1,79 @@
|
||||
# Tasks
|
||||
|
||||
## 1. Evidence reference extraction
|
||||
|
||||
- [x] Add `evidence_refs` extraction in `ToolInvocationRecorder` for `query_metrics` alert arrays.
|
||||
- [x] Add `evidence_refs` extraction in `ToolInvocationRecorder` for `query_logs` log arrays.
|
||||
- [x] Add `evidence_refs` extraction in `ToolInvocationRecorder` for `lookup_knowledge` evidence blocks.
|
||||
- [x] Add focused recorder tests for `$.alerts[i]`, `$.logs[i]`, and `$.evidence_blocks[i]`.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Persisted `retrieval_details` contains minimal `raw_path` and `text`.
|
||||
- Extraction does not infer root cause or diagnosis.
|
||||
|
||||
## 2. Gatekeeper reference fidelity
|
||||
|
||||
- [x] Extend `ExecutorGatekeeperService` to read singular `source_invocation_id`, `raw_path`, and `evidence_excerpt`.
|
||||
- [x] Keep legacy `source_invocation_ids` compatibility where needed, but require `raw_path` for precise pass.
|
||||
- [x] Validate invocation existence, session ownership, tool name, `raw_path`, and excerpt similarity.
|
||||
- [x] Add `severity` and checked binding details to Gatekeeper output.
|
||||
- [x] Add tests for valid reference, missing `raw_path`, missing `evidence_refs`, unknown `raw_path`, mismatched excerpt, fabricated invocation ID, and tool mismatch.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Valid precise references pass.
|
||||
- Missing precision downgrades to `LOW_CONFID`.
|
||||
- Fabricated or mismatched references become `REJECT`.
|
||||
|
||||
## 3. Verifier input hook and prompts
|
||||
|
||||
- [x] Tighten `VerifierInputHook` auto-backfill to only unique single invocation candidates.
|
||||
- [x] Prevent auto-filled bindings without `raw_path` from passing Gatekeeper.
|
||||
- [x] Add auto-backfill warnings into Gatekeeper/audit output.
|
||||
- [x] Update `chat-executor-prompt.md` with evidence-reference and narrow-scope constraints.
|
||||
- [x] Update `chat-verifier-prompt.md` so verified excerpts are the primary derivability evidence.
|
||||
- [x] Add focused hook and prompt-sensitive tests where practical.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Hook no longer bulk-fills invocation IDs.
|
||||
- Verifier can see `gatekeeper_result.severity`.
|
||||
- Executor is instructed to output precise references and avoid over-expansion.
|
||||
|
||||
## 4. Effective verdict and audit persistence
|
||||
|
||||
- [x] Ensure `severity=reject` prevents effective `PASS` and maps to `REJECT` when appropriate.
|
||||
- [x] Ensure `severity=low_confid` prevents effective `PASS` and maps to `LOW_CONFID`.
|
||||
- [x] Ensure `gatekeeper_result` with `severity` is persisted under `verifier_evaluation`.
|
||||
- [x] Add ChatService or integration tests for effective verdict guardrails.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Gatekeeper fail cannot become final PASS.
|
||||
- Audit JSON contains the minimum Gatekeeper fields.
|
||||
|
||||
## 5. HikariCP mock quality
|
||||
|
||||
- [x] Add positive HikariCP mock logs for `order-service`.
|
||||
- [x] Support HikariCP synonym matching.
|
||||
- [x] Remove `generic-service` placeholder evidence for no-hit cases.
|
||||
- [x] Add tests for HikariCP positive and negative no-hit behavior.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Positive HikariCP query returns order-service logs.
|
||||
- Negative HikariCP query returns `logs=[]` and `evidence_status=no_evidence`.
|
||||
|
||||
## 6. Verification
|
||||
|
||||
- [x] Run focused unit tests for recorder, Gatekeeper, hook, ChatService guardrails, and query log mock behavior.
|
||||
- [x] Run `openspec validate verifier-evidence-reference-fidelity --strict`.
|
||||
- [x] Run `openspec validate --specs`.
|
||||
- [x] Start the Java project using `mvn spring-boot:run`.
|
||||
- [x] Run end-to-end checks for HighMemoryUsage positive, SlowResponse positive, HikariCP positive, HikariCP negative, and narrow forbidden claim.
|
||||
- [x] Query MySQL audit data with `scripts/query_mysql.py` to confirm persisted `gatekeeper_result`.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Minimum E2E matrix passes or any failure is classified as code issue, mock quality issue, retrieval/tool quality issue, or model nondeterminism with evidence.
|
||||
@@ -1,4 +1,4 @@
|
||||
# chat-verifier-agent Specification
|
||||
# chat-verifier-agent Specification
|
||||
|
||||
## Purpose
|
||||
TBD - created by archiving change chat-verifier-agent. Update Purpose after archive.
|
||||
@@ -119,6 +119,7 @@ The system SHALL use ChatService for explicit single-round `Planner -> Executor
|
||||
- **THEN** the system SHALL output a degraded result indicating the answer cannot be reliably generated
|
||||
- **AND** it SHALL NOT pass through the raw Executor answer
|
||||
- **AND** it SHALL NOT include a root-cause conclusion
|
||||
|
||||
### Requirement: Verifier SHALL be observable
|
||||
The Verifier's verdict and downstream final-answer composition SHALL be persisted for observability.
|
||||
|
||||
@@ -421,3 +422,112 @@ The system SHALL use Composer or fixed safe templates for PASS, LOW_CONFID, and
|
||||
- **AND** it SHALL include only confirmed facts, evidence gaps, and next-step suggestions
|
||||
- **AND** it SHALL NOT include unverified raw answer content
|
||||
- **AND** it SHALL NOT include a root-cause conclusion
|
||||
|
||||
### Requirement: Executor evidence bindings SHALL support precise evidence references
|
||||
Executor V2 evidence bindings SHALL support precise evidence references that locate evidence inside a persisted tool invocation.
|
||||
|
||||
#### Scenario: Precise evidence binding contains invocation path and excerpt
|
||||
- **WHEN** Executor binds evidence to a claim
|
||||
- **THEN** the binding SHOULD include singular `source_invocation_id`
|
||||
- **AND** the binding SHOULD include `raw_path`
|
||||
- **AND** the binding SHALL include `tool_name` and `evidence_excerpt`
|
||||
- **AND** the `raw_path` SHALL be interpreted relative to the referenced tool invocation's `retrieval_details.evidence_refs`
|
||||
|
||||
#### Scenario: Legacy plural invocation ids remain compatibility only
|
||||
- **WHEN** Executor emits legacy `source_invocation_ids`
|
||||
- **THEN** the system MAY read them for compatibility
|
||||
- **AND** they SHALL NOT be sufficient for a precise Gatekeeper pass without `raw_path`
|
||||
|
||||
### Requirement: Gatekeeper SHALL validate evidence reference fidelity
|
||||
Gatekeeper SHALL validate that Executor evidence bindings point to real current-session evidence references before Verifier uses them as primary evidence.
|
||||
|
||||
#### Scenario: Valid precise binding passes
|
||||
- **WHEN** a binding's `source_invocation_id` exists in the current session
|
||||
- **AND** the binding's `tool_name` matches the persisted invocation
|
||||
- **AND** the binding's `raw_path` exists in `retrieval_details.evidence_refs`
|
||||
- **AND** the binding's `evidence_excerpt` is supported by the matching evidence ref text
|
||||
- **THEN** Gatekeeper SHALL return `status=pass`
|
||||
- **AND** Gatekeeper SHALL return `severity=none`
|
||||
|
||||
#### Scenario: Missing raw path is low confidence
|
||||
- **WHEN** a binding references an existing invocation
|
||||
- **AND** the binding omits `raw_path`
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=low_confid`
|
||||
- **AND** the effective verifier result SHALL NOT be `PASS`
|
||||
|
||||
#### Scenario: Old invocation without evidence refs is low confidence
|
||||
- **WHEN** a binding references an existing invocation
|
||||
- **AND** the invocation does not contain `retrieval_details.evidence_refs`
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=low_confid`
|
||||
- **AND** the effective verifier result SHALL NOT be `PASS`
|
||||
|
||||
#### Scenario: Unknown raw path is rejected
|
||||
- **WHEN** a binding references an existing invocation
|
||||
- **AND** the binding's `raw_path` is absent from that invocation's `retrieval_details.evidence_refs`
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=reject`
|
||||
- **AND** `failed_rules` SHALL include `evidence.raw_path`
|
||||
|
||||
#### Scenario: Mismatched excerpt is rejected
|
||||
- **WHEN** a binding references an existing invocation and raw path
|
||||
- **AND** the binding's `evidence_excerpt` is not supported by the matching system-side evidence ref text
|
||||
- **THEN** Gatekeeper SHALL return `status=fail`
|
||||
- **AND** Gatekeeper SHALL return `severity=reject`
|
||||
- **AND** `failed_rules` SHALL include `evidence.excerpt_mismatch`
|
||||
|
||||
### Requirement: Verifier SHALL use verified claim-local evidence for derivability
|
||||
Verifier SHALL judge structured claims primarily against Gatekeeper-verified claim-local evidence excerpts.
|
||||
|
||||
#### Scenario: Verified excerpt supports direct observation
|
||||
- **WHEN** `gatekeeper_result.severity=none`
|
||||
- **AND** a claim's verified evidence excerpts directly contain the claim's concrete facts
|
||||
- **THEN** Verifier MAY classify that claim as `direct_observation`
|
||||
|
||||
#### Scenario: Tool trace summary is navigation context
|
||||
- **WHEN** `executor_structured_output.claims[].evidence_bindings` are available
|
||||
- **THEN** Verifier SHALL use `tool_trace_summary` as navigation and audit context
|
||||
- **AND** it SHALL NOT require `tool_trace_summary.output_summary` to contain every fact already present in verified claim-local evidence
|
||||
|
||||
### Requirement: Gatekeeper severity SHALL constrain effective verdict
|
||||
Runtime effective verdict calculation SHALL treat Gatekeeper severity as a hard upper bound.
|
||||
|
||||
#### Scenario: Reject severity prevents PASS
|
||||
- **WHEN** `gatekeeper_result.severity=reject`
|
||||
- **AND** the Verifier model returns `verdict=PASS`
|
||||
- **THEN** ChatService SHALL downgrade the effective verdict
|
||||
- **AND** the effective verdict SHALL be `REJECT`
|
||||
|
||||
#### Scenario: Low confidence severity prevents PASS
|
||||
- **WHEN** `gatekeeper_result.severity=low_confid`
|
||||
- **AND** the Verifier model returns `verdict=PASS`
|
||||
- **THEN** ChatService SHALL downgrade the effective verdict
|
||||
- **AND** the effective verdict SHALL be `LOW_CONFID`
|
||||
|
||||
#### Scenario: Gatekeeper audit includes severity
|
||||
- **WHEN** verifier evaluation is persisted
|
||||
- **THEN** `diagnosis_session.self_evaluation.verifier_evaluation.gatekeeper_result` SHALL include `status`, `severity`, `checked_bindings`, `failed_rules`, `warnings`, and `errors`
|
||||
|
||||
### Requirement: Verifier input hook SHALL only perform narrow compatibility backfill
|
||||
The verifier input hook SHALL avoid converting broad tool summaries into precise evidence references.
|
||||
|
||||
#### Scenario: Unique invocation candidate may be backfilled
|
||||
- **WHEN** an evidence binding omits `source_invocation_id`
|
||||
- **AND** exactly one current-session invocation exists for the binding's `tool_name`
|
||||
- **THEN** the hook MAY backfill `source_invocation_id`
|
||||
- **AND** it SHALL add a Gatekeeper warning describing the auto-backfill
|
||||
|
||||
#### Scenario: Raw path is never backfilled
|
||||
- **WHEN** an evidence binding omits `raw_path`
|
||||
- **THEN** the hook SHALL NOT synthesize `raw_path`
|
||||
- **AND** Gatekeeper SHALL treat the binding as not precise enough to pass
|
||||
|
||||
### Requirement: Executor prompt SHALL constrain narrow-scope over-expansion
|
||||
The Executor prompt SHALL instruct Executor to keep narrow confirmation questions focused on observation-level claims.
|
||||
|
||||
#### Scenario: Narrow scope produces minimal observation claims
|
||||
- **WHEN** the user asks to confirm one specific service, alert, or symptom
|
||||
- **THEN** Executor SHOULD output the minimum necessary claims, normally one and at most two
|
||||
- **AND** those claims SHALL be `observation` or `negative_observation` unless current-session evidence proves more
|
||||
- **AND** Executor SHALL NOT emit unrelated root-cause, remediation, or excluded-topic claims as confirmed facts
|
||||
|
||||
@@ -108,3 +108,44 @@ The persisted trace SHALL make it possible to audit model step counts separately
|
||||
- **AND** `diagnosis_session.tool_call_count` SHALL count persisted evidence-tool invocation rows
|
||||
- **AND** helper workflow calls that are not evidence rows SHALL be auditable from agent steps or logs without inflating `tool_invocation`
|
||||
|
||||
### Requirement: Evidence tools SHALL persist minimal evidence refs
|
||||
Evidence-bearing tool invocations SHALL persist claim-addressable evidence references in `tool_invocation.retrieval_details.evidence_refs`.
|
||||
|
||||
#### Scenario: Metrics alerts produce evidence refs
|
||||
- **WHEN** a `query_metrics` invocation returns alert entries
|
||||
- **THEN** the persisted retrieval details SHALL include one `evidence_refs` item per usable alert
|
||||
- **AND** each item SHALL include `raw_path` formatted as `$.alerts[i]`
|
||||
- **AND** each item SHALL include bounded `text` containing concrete alert facts such as alert name, state, service, current value, and duration when available
|
||||
|
||||
#### Scenario: Logs produce evidence refs
|
||||
- **WHEN** a `query_logs` invocation returns log entries
|
||||
- **THEN** the persisted retrieval details SHALL include one `evidence_refs` item per usable log
|
||||
- **AND** each item SHALL include `raw_path` formatted as `$.logs[i]`
|
||||
- **AND** each item SHALL include bounded `text` containing concrete log facts such as timestamp, level, service, and message when available
|
||||
|
||||
#### Scenario: Knowledge lookup produces evidence refs
|
||||
- **WHEN** a `lookup_knowledge` invocation returns evidence blocks
|
||||
- **THEN** the persisted retrieval details SHALL include one `evidence_refs` item per usable evidence block
|
||||
- **AND** each item SHALL include `raw_path` formatted as `$.evidence_blocks[i]`
|
||||
- **AND** each item SHALL include bounded `text` containing concrete block content, title, or source when available
|
||||
|
||||
#### Scenario: Evidence ref extraction does not infer diagnosis
|
||||
- **WHEN** the recorder creates `evidence_refs`
|
||||
- **THEN** it SHALL only copy or format concrete tool output fields
|
||||
- **AND** it SHALL NOT infer root cause, remediation, or diagnosis conclusions
|
||||
|
||||
### Requirement: Log mock no-hit semantics SHALL avoid placeholder evidence
|
||||
The log query mock SHALL distinguish positive mock evidence from no-hit results without using placeholder service logs as evidence.
|
||||
|
||||
#### Scenario: HikariCP positive query returns order-service pool evidence
|
||||
- **WHEN** a `query_logs` request targets `order-service` and HikariCP connection-pool exhaustion terms
|
||||
- **THEN** the tool SHALL return order-service HikariCP-related log entries
|
||||
- **AND** the returned evidence SHALL include concrete terms such as `HikariPool`, `active=50/50`, `waiting`, or `request timed out after 30000ms`
|
||||
- **AND** it SHALL NOT return `generic-service` placeholder logs
|
||||
|
||||
#### Scenario: HikariCP no-hit query returns no evidence
|
||||
- **WHEN** a `query_logs` request targets a service without matching HikariCP mock evidence
|
||||
- **THEN** the tool SHALL return an empty `logs` array
|
||||
- **AND** it SHALL mark the output as `evidence_status=no_evidence`
|
||||
- **AND** it SHALL NOT return `generic-service` placeholder logs
|
||||
|
||||
|
||||
Reference in New Issue
Block a user