osiman/SuperBizAgent-java
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(eval): add evidence pipeline acceptance closure

Browse Source
This commit is contained in:
aruo
2026-07-09 00:47:48 +08:00
parent a77c947cd4
commit db0f229285
46 changed files with 1434 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
devflow/index.md
+1
View File
@@ -10,6 +10,7 @@
| 2026-07-07 | executor-gatekeeper-hook | Chat质量门禁/证据归因 | Gatekeeper, verifier payload, source_invocation_ids, tool_name match, self_evaluation | openspec/changes/archive/2026-07-07-executor-gatekeeper-hook | archived |
| 2026-07-07 | executor-verifier-claim-checks | Chat质量门禁/证据归因 | Verifier claim_checks, facts_checked compatibility, effective verdict guardrail, malformed output downgrade | openspec/changes/archive/2026-07-07-executor-verifier-claim-checks | archived |
| 2026-07-08 | executor-composer-final-answer | Chat quality gate/evidence attribution | chat_composer, final answer, allowed_claims, allowed_hypotheses, safe fallback, composer_output | openspec/changes/archive/2026-07-08-executor-composer-final-answer | archived |
| 2026-07-08 | diagnosis-eval-demo-gatekeeper-closure | Agent eval/demo/Gatekeeper | diagnosis eval matrix, stable demo scenarios, Gatekeeper rule set version, audit metadata | openspec/changes/archive/2026-07-08-diagnosis-eval-demo-gatekeeper-closure | archived |
| 2026-07-08 | verifier-evidence-reference-fidelity | Chat质量门禁/证据归因 | evidence_refs, raw_path, Gatekeeper severity, verifier evidence excerpt, HikariCP mock, no_evidence | openspec/changes/archive/2026-07-08-verifier-evidence-reference-fidelity | archived |
| 2026-07-06 | rag-eval-pipeline-closure | RAG/评测/回归闭环 | lookupResult fixture, LookupKnowledgeTool snapshot, evidenceBlocks, contextPack, retrievalTrace, rerankTrace, baseline diff, fallback case | devflow/projects/2026-07-06-rag-eval-pipeline-closure | archived |
| 2026-07-06 | modular-rag-pipeline | RAG/Agent工具/证据链 | modular RAG, lookup_knowledge, evidenceBlocks, contextPack, rerank, retrievalTrace, L0 hint, unfiltered retry | openspec/changes/archive/2026-07-06-modular-rag-pipeline | archived |
devflow/projects/2026-07-08-diagnosis-eval-demo-gatekeeper-closure/acceptance.md
+48
View File
@@ -0,0 +1,48 @@
# diagnosis-eval-demo-gatekeeper-closure Acceptance
## Static / Structure Verification
- `cmd /c openspec validate diagnosis-eval-demo-gatekeeper-closure --strict`
- Result: passed.
- `cmd /c openspec validate --specs`
- Result: passed, 10 specs passed.
## Script Verification
- `mvn "-Dtest=ExecutorGatekeeperServiceTest,DiagnosisTraceEvaluatorTest,DiagnosisEvalBaselineDiffTest,VerifierInputHookTest" test`
- Result: 36 tests, 0 failures, 0 errors.
- `mvn "-Dtest=DiagnosisTraceEvaluatorTest,DiagnosisEvalBaselineDiffTest,ExecutorGatekeeperServiceTest,VerifierInputHookTest,ChatServiceSequentialAgentTest,ToolInvocationRecorderTest,QueryLogsToolsTest" test`
- Result: 61 tests, 0 failures, 0 errors.
- `mvn "-Dtest=ExecutorGatekeeperServiceTest,VerifierInputHookTest" test`
- Result after E2E startup fix: 23 tests, 0 failures, 0 errors.
## Live E2E Verification
- Start command: `mvn spring-boot:run "-Dspring-boot.run.profiles=mvp-demo"`.
- Demo command: `powershell -ExecutionPolicy Bypass -File mvp/demo/scripts/run-payment-timeout-demo.ps1`.
- Result: chat, trace, and feedback requests completed successfully.
- Output files:
- `mvp/demo/output/chat-response.json`
- `mvp/demo/output/trace-response.json`
- `mvp/demo/output/feedback-response.json`
- Trace observations:
- `hasVerifierEvaluation=true`
- `gatekeeper_result.rule_set_version=gatekeeper-rules-v1`
## Fixed During Verification
- E2E startup initially failed because Spring could not instantiate `ExecutorGatekeeperService`.
- Root cause: two public constructors and no explicit `@Autowired` constructor.
- Fix: annotate the production constructor with `@Autowired`.
## Residual Risk
- The live payment-timeout path can still produce `LOW_CONFID` because model-generated evidence bindings may omit some explicit `source_invocation_id` values.
- This is not a blocker for this change because deterministic matrix behavior is covered by saved fixtures and baseline evaluation.
- Existing Maven warnings remain: duplicate `spring-boot-starter-test` declaration and Lombok `@Builder` default warnings.
## Archive Status
- Devflow archive artifacts created.
- OpenSpec change archived to `openspec/changes/archive/2026-07-08-diagnosis-eval-demo-gatekeeper-closure`.
- Main specs synced by `cmd /c openspec archive diagnosis-eval-demo-gatekeeper-closure --yes`.
devflow/projects/2026-07-08-diagnosis-eval-demo-gatekeeper-closure/brief.md
+33
View File
@@ -0,0 +1,33 @@
# diagnosis-eval-demo-gatekeeper-closure Brief
## Background
The Chat evidence pipeline already had Executor V2 structured output, deterministic Gatekeeper validation, Verifier claim checks, and Composer final rendering. The missing piece was an interview-ready acceptance story that made the anti-hallucination behavior easy to demonstrate and regress.
## Goal
Close the next three interview-readiness gaps together:
- diagnosis eval fixture matrix
- stable demo data set
- Gatekeeper rule configuration and audit version
## Scope
- Expand `mvp/eval` with matrix-oriented cases, fixtures, and baseline reports.
- Add stable demo request payloads and scenario documentation.
- Add a lightweight local Gatekeeper rule catalog with `rule_set_version` and rule metadata in `gatekeeper_result`.
- Update architecture, demo, and eval docs to describe the current implementation.
## Non-goals
- No new public HTTP endpoint.
- No new database table.
- No Planner `scope_contract`.
- No Gatekeeper retry loop.
- No remote or dynamic rule execution engine.
## OpenSpec
- Change: `openspec/changes/diagnosis-eval-demo-gatekeeper-closure`
- Interface impact: L2 internal contract change.
devflow/projects/2026-07-08-diagnosis-eval-demo-gatekeeper-closure/decisions.md
+144
View File
@@ -0,0 +1,144 @@
# diagnosis-eval-demo-gatekeeper-closure Decisions
## Clarify
- Entry summary: implement the next three interview-readiness items together: diagnosis eval fixture matrix, stable demo data set, and Gatekeeper rule configuration/audit version.
- Slug: `diagnosis-eval-demo-gatekeeper-closure`
- Devflow scale: `standard`
- Interface impact: expected L2 internal contract change because `gatekeeper_result` audit JSON will gain rule metadata/version fields.
## Context
- `devflow/index.md` used: related entries found for diagnosis eval harness, fixture expansion, MVP demo runbook, Gatekeeper hook, and verifier evidence reference fidelity.
- Relevant glossary:
- Evidence Tools produce incident facts and must be recorded in `tool_invocation`.
- Verifier should not use skills/runbooks as incident evidence.
- `tool_invocation.retrieval_details` is the structured evidence/audit home for tool-specific details.
- Historical constraints that must enter OpenSpec:
- Diagnosis eval is offline and deterministic; no LLM-as-judge.
- Demo assets should be runnable, but fixed regression should use saved fixtures.
- Gatekeeper remains in the Verifier hook path.
- No new database table for Gatekeeper audit; use `self_evaluation.verifier_evaluation.gatekeeper_result`.
- `$.no_evidence` is a query no-hit signal, not proof that a problem is impossible.
## Question Pool
| ID | Dimension | Mode | Question | Status |
|---|---|---|---|---|
| Q1 | Terminology | evidence-driven | What names should this change use for the matrix, demo set, and Gatekeeper rule metadata? | Resolved |
| Q2 | Boundary | evidence-driven | Should this change alter public APIs, database schema, Planner output, or retry behavior? | Resolved |
| Q3 | Acceptance | evidence-driven | Which existing tests and baseline assets define the current acceptance style? | Resolved |
| Q4 | Technical | evidence-driven | Where should Gatekeeper rule metadata live with minimal implementation risk? | Pending code research |
| Q5 | Scope | user-interview | Should the stable demo set be documentation/payloads only, or should it include live E2E scripts for all scenarios? | Confirmed |
## Evidence-driven Conclusions
- Q1 conclusion: use `diagnosis eval matrix`, `stable demo scenarios`, and `Gatekeeper rule set version` as terms.
- Q2 conclusion: keep this as an internal contract change. Do not add public endpoints, tables, Planner `scope_contract`, or Gatekeeper retry.
- Q3 conclusion: existing `DiagnosisTraceEvaluatorTest`, `ExecutorGatekeeperServiceTest`, `VerifierInputHookTest`, `ToolInvocationRecorderTest`, and `mvp/eval/reports` define the current acceptance style.
- Q4 conclusion: Gatekeeper metadata should live behind a small rule catalog loaded by `ExecutorGatekeeperService`; the audit output should include a rule set version and enabled rule metadata summary, without adding tables or remote registry.
## User-interview Confirmations
- Q5 confirmed by resumed objective: complete items 1/2/3 with sm-flow, archive, submit, and run end-to-end if necessary.
- Implementation interpretation: stable demo scenarios will be fixed request payloads and runbook docs plus deterministic fixture-backed eval. Live E2E remains necessary only for at least one main path or where unit/fixture evidence is insufficient.
## OpenSpec Backfill
- Created Draft proposal at `openspec/changes/diagnosis-eval-demo-gatekeeper-closure/proposal.md`.
- Context constraints from historical devflow entries were written into the proposal.
- Scope confirmation and Gatekeeper catalog placement were written into the proposal/design.
## Current Checkpoint
- Discover completed.
- No implementation files changed yet.
## Specify / Alignment
### Cross-artifact Alignment
| Check | Status | Notes |
|---|---|---|
| brief/proposal goals -> proposal | Aligned | Proposal covers eval matrix, stable demo scenarios, and Gatekeeper rule catalog/audit version. |
| proposal scope/constraints -> design | Aligned | Design records offline deterministic eval, fixture-backed demo distinction, local rule catalog, and no new table/API. |
| design decisions -> specs/tasks | Aligned | Specs cover eval matrix, rule set version validation, demo scenarios, and Gatekeeper rule metadata; tasks cover matching implementation slices. |
| specs observable behavior -> tasks | Aligned | Each requirement has an executable task and acceptance check. |
### Interface Impact
- Level: L2 internal contract change.
- Reason: `gatekeeper_result` internal audit JSON gains `rule_set_version` and rule metadata summary. Eval case/result fields may gain optional rule set checks. No public HTTP API, database schema, or external DTO contract changes.
## Audit
Input -> processing -> output chain:
```text
mvp/demo request docs + mvp/eval fixtures
-> DiagnosisTraceEvaluator
-> baseline reports
-> interview/demo evidence
Gatekeeper rule catalog
-> ExecutorGatekeeperService
-> VerifierInputHook / ChatService persisted self_evaluation
-> Trace and eval audit
```
Architecture risk assessment:
1. The change is intentionally internal and should not add new public consumers.
2. Gatekeeper catalog must stay metadata-only; dynamic rule execution would be a different, riskier architecture.
3. Fixture-backed demo scenarios should be documented as deterministic regression artifacts, not live LLM guarantees.
4. Baseline report churn is expected and must be committed with case/fixture changes.
5. No devflow/OpenSpec conflict found.
## Commit Gate
- `cmd /c openspec validate diagnosis-eval-demo-gatekeeper-closure --strict`: passed.
- `cmd /c openspec validate --specs`: passed, 10 specs passed.
- File completeness:
- proposal.md: present.
- design.md: present.
- specs: present for `diagnosis-eval-harness`, `mvp-demo-trace-acceptance`, `chat-verifier-agent`.
- tasks.md: present.
- Consistency:
- Proposal concepts have corresponding design sections.
- Design decisions are reflected in specs/tasks.
- Task acceptance checks are verifiable.
## Current Checkpoint
- Commit completed.
- `.committed` marker created.
## Apply Verification
- Focused verification passed:
- `mvn "-Dtest=ExecutorGatekeeperServiceTest,DiagnosisTraceEvaluatorTest,DiagnosisEvalBaselineDiffTest,VerifierInputHookTest" test`
- Result: 36 tests, 0 failures, 0 errors.
- Broader relevant regression passed:
- `mvn "-Dtest=DiagnosisTraceEvaluatorTest,DiagnosisEvalBaselineDiffTest,ExecutorGatekeeperServiceTest,VerifierInputHookTest,ChatServiceSequentialAgentTest,ToolInvocationRecorderTest,QueryLogsToolsTest" test`
- Result: 61 tests, 0 failures, 0 errors.
- E2E startup repro found a Spring bean construction issue:
- Command: `mvn spring-boot:run "-Dspring-boot.run.profiles=mvp-demo"`
- Failure: `ExecutorGatekeeperService` had two public constructors and no annotated constructor, so Spring attempted a no-arg constructor and failed with `No default constructor found`.
- Classification: code deviation from OpenSpec implementation intent, not a spec gap.
- Fix: annotate the production constructor with `@Autowired`.
- Post-fix focused regression passed:
- `mvn "-Dtest=ExecutorGatekeeperServiceTest,VerifierInputHookTest" test`
- Result: 23 tests, 0 failures, 0 errors.
- Live E2E passed for demo compatibility:
- Start: `mvn spring-boot:run "-Dspring-boot.run.profiles=mvp-demo"`
- Run: `powershell -ExecutionPolicy Bypass -File mvp/demo/scripts/run-payment-timeout-demo.ps1`
- Result: `/api/chat`, `/api/diagnosis/{sessionId}/trace`, and `/api/feedback` completed successfully.
- Trace summary included `hasVerifierEvaluation=true`.
- Persisted Gatekeeper audit included `rule_set_version=gatekeeper-rules-v1`.
- Residual quality note: the live payment-timeout response remained `LOW_CONFID` because some model-produced evidence bindings still lacked explicit `source_invocation_id`; deterministic PASS/LOW_CONFID/REJECT claims are covered by fixture-backed eval.
## Archive Readiness
- OpenSpec tasks 1-4 completed.
- Verification is recorded in devflow acceptance artifacts.
- Remaining known risk: live LLM output is not deterministic and may still produce LOW_CONFID on the payment-timeout path; this is intentionally documented as demo compatibility, not a fixed PASS guarantee.
devflow/projects/2026-07-08-diagnosis-eval-demo-gatekeeper-closure/evidence.md
+22
View File
@@ -0,0 +1,22 @@
# diagnosis-eval-demo-gatekeeper-closure Evidence
## Code And Artifact Evidence
- Gatekeeper rule metadata lives in `src/main/resources/gatekeeper/gatekeeper-rules.json`.
- `ExecutorGatekeeperService` loads the local catalog, uses configured threshold parameters, and emits `rule_set_version` plus enabled rule metadata.
- `VerifierInputHook` and `ChatService` preserve Gatekeeper audit metadata in fallback/default paths.
- `DiagnosisTraceEvaluator` can optionally validate expected Gatekeeper rule set version.
- `mvp/eval/cases/diagnosis-cases.json` now includes narrow-scope and no-evidence matrix cases.
- `mvp/eval/reports/baseline-report.json` and `.md` were regenerated for the expanded fixed matrix.
- `mvp/demo/evidence-pipeline-scenarios.md` documents live vs fixture-backed demo scenarios.
## Decisions
- Keep this phase internal: no public API, no DB schema, no Planner output change.
- Keep Gatekeeper deterministic Java validation; the catalog is metadata/config only.
- Treat live demo as compatibility evidence and fixture-backed eval as deterministic regression evidence.
- Persist audit under the existing `self_evaluation.verifier_evaluation.gatekeeper_result` structure.
## Runtime Finding
The first Maven E2E startup found a real integration issue: `ExecutorGatekeeperService` had multiple public constructors without an annotated constructor, so Spring could not instantiate the service. The fix was to annotate the production constructor with `@Autowired`.