feat(agent): add executor gatekeeper hook
This commit is contained in:
aruo
@@ -7,6 +7,7 @@
|
||||
| 2026-07-05 | diagnosis-playbook-skills | Agent Skill/Playbook | read_skill, diagnosis playbook, progressive disclosure, payment timeout, MySQL pool, Redis timeout | openspec/changes/diagnosis-playbook-skills | implemented |
|
||||
| 2026-07-07 | executor-evidence-output-contract | Chat质量门禁/证据归因 | Executor structured output, evidence bindings, Verifier structured claims, LOW_CONFID, hallucination | openspec/changes/archive/2026-07-07-executor-evidence-output-contract | archived |
|
||||
| 2026-07-07 | executor-v2-output-contract | Chat质量门禁/证据归因 | executor_evidence_v2, user_facing_answer removal, diagnosis_summary removal, structured renderer | openspec/changes/archive/2026-07-07-executor-v2-output-contract | archived |
|
||||
| 2026-07-07 | executor-gatekeeper-hook | Chat质量门禁/证据归因 | Gatekeeper, verifier payload, source_invocation_ids, tool_name match, self_evaluation | openspec/changes/archive/2026-07-07-executor-gatekeeper-hook | archived |
|
||||
| 2026-07-06 | rag-eval-pipeline-closure | RAG/评测/回归闭环 | lookupResult fixture, LookupKnowledgeTool snapshot, evidenceBlocks, contextPack, retrievalTrace, rerankTrace, baseline diff, fallback case | devflow/projects/2026-07-06-rag-eval-pipeline-closure | archived |
|
||||
| 2026-07-06 | modular-rag-pipeline | RAG/Agent工具/证据链 | modular RAG, lookup_knowledge, evidenceBlocks, contextPack, rerank, retrievalTrace, L0 hint, unfiltered retry | openspec/changes/archive/2026-07-06-modular-rag-pipeline | archived |
|
||||
| 2026-07-05 | mvp-demo-interview-runbook | MVP Demo/Interview | Plan C, payment timeout, runbook, trace checklist, demo script | openspec/changes/archive/2026-07-05-mvp-demo-interview-runbook | archived |
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
# Acceptance: executor-gatekeeper-hook
|
||||
|
||||
## Implementation Result
|
||||
|
||||
Completed stage two of Executor Structured Output V2.
|
||||
|
||||
- Added `ExecutorGatekeeperService`.
|
||||
- Added initial `schema.executor_v2` and `evidence.invocation_ref` rules.
|
||||
- Added `gatekeeper_result` to Verifier payload.
|
||||
- Stored `gatekeeper_result` in `VerifierContextHolder`.
|
||||
- Persisted `gatekeeper_result` under `diagnosis_session.self_evaluation.verifier_evaluation`.
|
||||
- Updated Verifier prompt so Gatekeeper fail must not produce PASS.
|
||||
- Added focused tests for schema failure, valid pass, fabricated invocation ids, tool name mismatch, hook payload, and persistence.
|
||||
|
||||
## Static Verification
|
||||
|
||||
- `cmd /c openspec validate executor-gatekeeper-hook`
|
||||
- Result: passed.
|
||||
- Coverage: OpenSpec change validity.
|
||||
|
||||
## Script Verification
|
||||
|
||||
- `mvn "-Dtest=ExecutorGatekeeperServiceTest,VerifierInputHookTest,ChatServiceSequentialAgentTest" test`
|
||||
- Result: passed.
|
||||
- Coverage: 23 focused tests for Gatekeeper service, hook integration, and ChatService persistence.
|
||||
|
||||
## Browser / Manual Verification
|
||||
|
||||
Not run. This stage changes backend validation and audit behavior only.
|
||||
|
||||
## Unverified
|
||||
|
||||
- Full live application run with a real LLM.
|
||||
- MySQL trace inspection after a real chat session.
|
||||
- Excerpt similarity or phrase/utilization rules.
|
||||
|
||||
Reason: This phase intentionally covers deterministic schema and invocation-reference validation. Full live verification is better after Verifier V2 and Composer are implemented.
|
||||
|
||||
## Remaining Work
|
||||
|
||||
- Phase three: Verifier V2 `claim_checks`.
|
||||
- Phase four: Composer final-answer generation.
|
||||
- Phase five: eval fixtures and full audit closure.
|
||||
|
||||
## Archive Status
|
||||
|
||||
Devflow archive files created for stage two. OpenSpec archive is expected before moving to stage three.
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
# Brief: executor-gatekeeper-hook
|
||||
|
||||
## Background
|
||||
|
||||
Stage one of Executor Structured Output V2 changed Chat Executor output to `executor_evidence_v2`, removing final-expression fields from Executor. That made the output structured, but it did not yet prevent deterministic evidence attribution failures such as fabricated invocation ids, removed fields, empty evidence bindings, or mismatched tool names.
|
||||
|
||||
## Goal
|
||||
|
||||
Add a deterministic Gatekeeper between Executor output parsing and Verifier model execution.
|
||||
|
||||
The Gatekeeper should:
|
||||
|
||||
- Validate the initial Executor V2 schema.
|
||||
- Validate `claims[].evidence_bindings[].source_invocation_ids` against current-session `tool_invocation` rows.
|
||||
- Validate evidence binding `tool_name` against the persisted invocation tool name.
|
||||
- Expose a small `gatekeeper_result` to Verifier and audit persistence.
|
||||
|
||||
## Scope
|
||||
|
||||
Included:
|
||||
|
||||
- New Gatekeeper validation service.
|
||||
- `schema.executor_v2` initial rule.
|
||||
- `evidence.invocation_ref` initial rule.
|
||||
- `VerifierInputHook` payload integration.
|
||||
- `VerifierContextHolder` storage.
|
||||
- `ChatService` verifier evaluation persistence.
|
||||
- Minimal verifier prompt update.
|
||||
- Focused tests for Gatekeeper, hook payload, fabricated invocation ids, tool name mismatch, and persistence.
|
||||
|
||||
Excluded:
|
||||
|
||||
- No Executor retry on Gatekeeper failure.
|
||||
- No excerpt similarity rule in this phase.
|
||||
- No hallucination phrase or evidence utilization rule in this phase.
|
||||
- No Verifier V2 `claim_checks`.
|
||||
- No Composer.
|
||||
- No database schema changes.
|
||||
|
||||
## OpenSpec
|
||||
|
||||
- Change: `openspec/changes/executor-gatekeeper-hook`
|
||||
- Parent stage: `openspec/changes/archive/2026-07-07-executor-v2-output-contract`
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
# Decisions: executor-gatekeeper-hook
|
||||
|
||||
## Key Decisions
|
||||
|
||||
### Gatekeeper stays in VerifierInputHook
|
||||
|
||||
Decision: Gatekeeper is integrated inside `VerifierInputHook`, after Executor output parsing and before Verifier model execution.
|
||||
|
||||
Reason: The user explicitly chose to keep this version in the Verifier hook and not move validation into Executor hook. This preserves the current workflow orchestration.
|
||||
|
||||
### No retry in this phase
|
||||
|
||||
Decision: Gatekeeper failure does not trigger automatic Executor retry.
|
||||
|
||||
Reason: Retry behavior is intentionally deferred. This phase only validates, exposes, and audits deterministic failures.
|
||||
|
||||
### Initial rule set is intentionally small
|
||||
|
||||
Decision: Stage two implements only `schema.executor_v2` and `evidence.invocation_ref` as hard checks.
|
||||
|
||||
Reason: These rules catch the highest-confidence physical failures with low implementation risk. Excerpt similarity, hallucination phrases, and evidence utilization remain later enhancements.
|
||||
|
||||
### No new database schema
|
||||
|
||||
Decision: Persist `gatekeeper_result` in existing `diagnosis_session.self_evaluation.verifier_evaluation`.
|
||||
|
||||
Reason: The user asked to keep database fields minimal. Existing JSON audit storage is enough for this phase.
|
||||
|
||||
### Internal interface impact
|
||||
|
||||
Decision: This is an L2 internal interface extension.
|
||||
|
||||
Impact:
|
||||
|
||||
- Verifier payload gains `gatekeeper_result`.
|
||||
- `VerifierContextHolder` gains Gatekeeper result storage.
|
||||
- `self_evaluation.verifier_evaluation` gains `gatekeeper_result`.
|
||||
- No external API, DTO, database table, or schema migration changes.
|
||||
|
||||
## Deferred Decisions
|
||||
|
||||
- Whether Gatekeeper should later trigger Executor retry.
|
||||
- Whether `evidence.excerpt_similarity` should be hard fail or warn-only.
|
||||
- Whether hallucination phrase and evidence utilization rules should be config-driven from metadata files.
|
||||
- How Verifier V2 `claim_checks` should enforce Gatekeeper failures in code, beyond prompt instruction.
|
||||
|
||||
## Remaining Risks
|
||||
|
||||
- Verifier prompt compliance is not a deterministic guarantee; stage three should make Gatekeeper fail incompatible with PASS in Verifier V2 behavior.
|
||||
- Excerpt authenticity is not checked in this phase, so real invocation ids can still be paired with misleading excerpt text until a later rule is implemented.
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
# Evidence: executor-gatekeeper-hook
|
||||
|
||||
## Context Evidence
|
||||
|
||||
- `executor-v2-output-contract` established `executor_evidence_v2` and removed Executor final-expression fields.
|
||||
- `VerifierInputHook` is the existing integration point for explicit Verifier payload construction.
|
||||
- `ChatService.persistVerifierEvaluation(...)` is the existing persistence path for verifier audit snapshots.
|
||||
- `ToolInvocationRepository.findBySessionIdOrderByIdAsc(...)` provides the current-session invocation pool used by Gatekeeper.
|
||||
|
||||
## Implementation Evidence
|
||||
|
||||
- `src/main/java/com/superbiz/agent/service/ExecutorGatekeeperService.java`
|
||||
- Implements `schema.executor_v2`.
|
||||
- Implements `evidence.invocation_ref`.
|
||||
- Returns `status`, `failed_rules`, `warnings`, and `errors`.
|
||||
|
||||
- `src/main/java/com/superbiz/agent/hook/VerifierInputHook.java`
|
||||
- Runs Gatekeeper after parsing Executor output and building trace summary.
|
||||
- Adds `gatekeeper_result` to Verifier payload.
|
||||
- Stores `gatekeeper_result` in `VerifierContextHolder`.
|
||||
|
||||
- `src/main/java/com/superbiz/agent/util/VerifierContextHolder.java`
|
||||
- Stores per-request Gatekeeper result for later persistence.
|
||||
|
||||
- `src/main/java/com/superbiz/agent/service/ChatService.java`
|
||||
- Wires `ExecutorGatekeeperService` into verifier hook construction.
|
||||
- Persists `gatekeeper_result` under `diagnosis_session.self_evaluation.verifier_evaluation`.
|
||||
|
||||
- `src/main/resources/prompts/chat-verifier-prompt.md`
|
||||
- Documents `gatekeeper_result` as an input.
|
||||
- States Gatekeeper fail must not produce PASS.
|
||||
|
||||
## Test Evidence
|
||||
|
||||
- `src/test/java/com/superbiz/agent/service/ExecutorGatekeeperServiceTest.java`
|
||||
- Covers schema failure and valid pass behavior.
|
||||
- Covers fabricated invocation ids and tool name mismatch.
|
||||
|
||||
- `src/test/java/com/superbiz/agent/hook/VerifierInputHookTest.java`
|
||||
- Covers Verifier payload containing `gatekeeper_result`.
|
||||
- Covers hook behavior for fabricated invocation ids.
|
||||
|
||||
- `src/test/java/com/superbiz/agent/service/ChatServiceSequentialAgentTest.java`
|
||||
- Covers persistence of `gatekeeper_result` into verifier evaluation.
|
||||
|
||||
## Validation Evidence
|
||||
|
||||
- `mvn "-Dtest=ExecutorGatekeeperServiceTest,VerifierInputHookTest,ChatServiceSequentialAgentTest" test`
|
||||
- Result: passed.
|
||||
- Coverage: 23 focused tests.
|
||||
|
||||
- `cmd /c openspec validate executor-gatekeeper-hook`
|
||||
- Result: passed.
|
||||
|
||||
Reference in New Issue
Block a user