feat(eval): add evidence pipeline acceptance closure

2026-07-09 00:47:48 +08:00
parent a77c947cd4
commit db0f229285
46 changed files with 1434 additions and 53 deletions
@@ -0,0 +1,48 @@
+# diagnosis-eval-demo-gatekeeper-closure Acceptance
+
+## Static / Structure Verification
+
+- `cmd /c openspec validate diagnosis-eval-demo-gatekeeper-closure --strict`
+  - Result: passed.
+- `cmd /c openspec validate --specs`
+  - Result: passed, 10 specs passed.
+
+## Script Verification
+
+- `mvn "-Dtest=ExecutorGatekeeperServiceTest,DiagnosisTraceEvaluatorTest,DiagnosisEvalBaselineDiffTest,VerifierInputHookTest" test`
+  - Result: 36 tests, 0 failures, 0 errors.
+- `mvn "-Dtest=DiagnosisTraceEvaluatorTest,DiagnosisEvalBaselineDiffTest,ExecutorGatekeeperServiceTest,VerifierInputHookTest,ChatServiceSequentialAgentTest,ToolInvocationRecorderTest,QueryLogsToolsTest" test`
+  - Result: 61 tests, 0 failures, 0 errors.
+- `mvn "-Dtest=ExecutorGatekeeperServiceTest,VerifierInputHookTest" test`
+  - Result after E2E startup fix: 23 tests, 0 failures, 0 errors.
+
+## Live E2E Verification
+
+- Start command: `mvn spring-boot:run "-Dspring-boot.run.profiles=mvp-demo"`.
+- Demo command: `powershell -ExecutionPolicy Bypass -File mvp/demo/scripts/run-payment-timeout-demo.ps1`.
+- Result: chat, trace, and feedback requests completed successfully.
+- Output files:
+  - `mvp/demo/output/chat-response.json`
+  - `mvp/demo/output/trace-response.json`
+  - `mvp/demo/output/feedback-response.json`
+- Trace observations:
+  - `hasVerifierEvaluation=true`
+  - `gatekeeper_result.rule_set_version=gatekeeper-rules-v1`
+
+## Fixed During Verification
+
+- E2E startup initially failed because Spring could not instantiate `ExecutorGatekeeperService`.
+- Root cause: two public constructors and no explicit `@Autowired` constructor.
+- Fix: annotate the production constructor with `@Autowired`.
+
+## Residual Risk
+
+- The live payment-timeout path can still produce `LOW_CONFID` because model-generated evidence bindings may omit some explicit `source_invocation_id` values.
+- This is not a blocker for this change because deterministic matrix behavior is covered by saved fixtures and baseline evaluation.
+- Existing Maven warnings remain: duplicate `spring-boot-starter-test` declaration and Lombok `@Builder` default warnings.
+
+## Archive Status
+
+- Devflow archive artifacts created.
+- OpenSpec change archived to `openspec/changes/archive/2026-07-08-diagnosis-eval-demo-gatekeeper-closure`.
+- Main specs synced by `cmd /c openspec archive diagnosis-eval-demo-gatekeeper-closure --yes`.