59 lines
3.1 KiB
Markdown
59 lines
3.1 KiB
Markdown
# Evidence
|
|
|
|
## Context Files Read
|
|
|
|
- `devflow/index.md`
|
|
- `devflow/glossary/CONTEXT.md`
|
|
- `devflow/projects/2026-07-08-diagnosis-eval-demo-gatekeeper-closure/decisions.md`
|
|
- `devflow/projects/2026-07-08-executor-composer-final-answer/decisions.md`
|
|
- `mvp/architecture/current-mvp-architecture.md`
|
|
- `mvp/architecture/agent-orchestration.md`
|
|
- `mvp/architecture/executor-evidence-pipeline-refactor.md`
|
|
- `mvp/architecture/harness-quality-gates.md`
|
|
- `mvp/demo/README.md`
|
|
- `mvp/demo/ten-minute-interview-demo.md`
|
|
- `mvp/eval/README.md`
|
|
- `mvp/eval/cases/diagnosis-cases.json`
|
|
- `src/main/java/com/superbiz/agent/service/ChatService.java`
|
|
- `src/main/java/com/superbiz/agent/service/ExecutorGatekeeperService.java`
|
|
- `src/main/java/com/superbiz/agent/eval/DiagnosisTraceEvaluator.java`
|
|
- `src/main/resources/gatekeeper/gatekeeper-rules.json`
|
|
|
|
## Tooling Note
|
|
|
|
The required `codebase-retrieval` and LSP tools were not exposed in this session. Impact analysis used `rg`, direct file reads, existing OpenSpec/devflow artifacts, and targeted tests instead.
|
|
|
|
## Implementation Evidence
|
|
|
|
- `src/main/java/com/superbiz/agent/service/ChatService.java`
|
|
- Adds `prompt_audit` under `verifier_evaluation` through the shared `persistVerifierEvaluation(...)` path.
|
|
- Uses compact metadata only: audit version, prompt names, prompt versions, and resource paths.
|
|
- `src/main/java/com/superbiz/agent/eval/DiagnosisTraceEvaluator.java`
|
|
- Adds deterministic checks for `requirePromptAudit`, `expectedPromptAuditVersion`, `expectedPromptVersions`, and `requireGatekeeperRules`.
|
|
- `src/main/java/com/superbiz/agent/eval/DiagnosisEvalReportWriter.java`
|
|
- Adds Prompt Audit and Gatekeeper rule count columns to Markdown reports.
|
|
- `mvp/eval/cases/diagnosis-cases.json`
|
|
- Expands fixed baseline to 12 fixture-backed cases.
|
|
- `mvp/eval/fixtures/prompt-gatekeeper-audit-closure-pass.json`
|
|
- Positive PASS fixture proving Prompt audit and Gatekeeper rule metadata closure.
|
|
- `mvp/eval/fixtures/audit-metadata-low-confid.json`
|
|
- LOW_CONFID fixture proving safe answer behavior while audit metadata remains present.
|
|
- `mvp/demo/scripts/run-interview-demo-check.ps1`
|
|
- Adds service readiness, Chat, Trace, feedback, and summary output for interview preflight.
|
|
|
|
## Verification Evidence
|
|
|
|
- OpenSpec:
|
|
- `openspec validate interview-demo-quality-audit --strict`: passed before archive.
|
|
- `openspec validate --specs --strict`: 10 specs passed after merging deltas into main specs.
|
|
- Unit/eval:
|
|
- `mvn -q "-Dtest=DiagnosisTraceEvaluatorTest" test`: passed.
|
|
- `mvn -q "-Dtest=ChatServiceSequentialAgentTest" test`: passed.
|
|
- `mvn -q "-Dtest=DiagnosisTraceEvaluatorTest,DiagnosisEvalBaselineDiffTest,ChatServiceSequentialAgentTest,ExecutorGatekeeperServiceTest,VerifierInputHookTest" test`: passed.
|
|
- Compile:
|
|
- `mvn -q -DskipTests compile`: passed.
|
|
- E2E:
|
|
- Started `mvn spring-boot:run -Dspring-boot.run.profiles=mvp-demo`.
|
|
- Ran `mvp/demo/scripts/run-interview-demo-check.ps1` against `http://localhost:9900`.
|
|
- Summary recorded `chatSuccess=true`, `verdict=LOW_CONFID`, `gatekeeperRuleSetVersion=gatekeeper-rules-v1`, and `promptAuditVersion=chat-prompts-v1`.
|