feat(graph): complete stategraph cleanup and acceptance
This commit is contained in:
zhuyongxin
+1
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
committed_at: 2026-07-17
|
||||
checkpoint: Commit
|
||||
authorization: user-requested-direct-implementation
|
||||
@@ -0,0 +1,127 @@
|
||||
## Context
|
||||
|
||||
ISS-011 的运行时和测试迁移已经完成：复杂 Chat 通过真实 Diagnosis StateGraph，Graph Nodes 使用显式 RunnableConfig/verified state，Run 保存 orchestration trace，三层权威 tests 已建立。剩余工作跨越代码清理、current docs、demo script、eval、live process、日志、数据库和 Issue 生命周期，必须按“静态/自动化先完成，live E2E 最后执行”的顺序收口。
|
||||
|
||||
全仓引用证明旧闭包为 `VerifierInputHook -> VerifierContextHolder + ToolTraceSummaryService`，外加 `ToolTraceSummaryServiceTest`。新 Graph 不依赖该闭包。current architecture docs 仍描述 Sequential/Hook/full trace；历史 issues/design-notes/fixtures 则应保留当时语义或历史兼容数据。
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
|
||||
- 删除旧闭包且保持 Graph parser/Gatekeeper/projection/evaluation 行为。
|
||||
- 把 architecture index 声明的 current docs、active eval/demo docs 与真实 StateGraph/Run trace 对齐。
|
||||
- 让 interview demo check 将 exact `run.orchestrationTrace` 作为强制验收字段和 summary 输出。
|
||||
- 运行确定性 Graph/test/eval gates，再启动 Maven 完成唯一最终 live E2E。
|
||||
- 对本次 E2E 的日志和数据库做 exact session/run 证据核验并清理进程。
|
||||
- 全部通过后关闭/归档 ISS-011 和阶段 5 OpenSpec。
|
||||
|
||||
**Non-Goals:**
|
||||
|
||||
- 不修改 Graph 路由、Prompt、公开 API、DTO 或 DB schema。
|
||||
- 不重写历史 archived/design-note 文档和 legacy eval fixtures。
|
||||
- 不删除 UI/eval 对历史 `tool_trace_summary` 的兼容读取。
|
||||
- 不治理仓库凭据、外部基础设施或其他 active issues。
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. 删除完整旧闭包，不留下空壳 Hook
|
||||
|
||||
删除 `VerifierInputHook.java`、`VerifierContextHolder.java`、`ToolTraceSummaryService.java` 和 `ToolTraceSummaryServiceTest.java`。保留空类或 deprecated wrapper 会继续让维护者误认为存在第二条 Verifier 输入路径，也会让 Spring component 扫描注册无消费者 service。
|
||||
|
||||
删除前后以全仓 definition/import/instantiation/type-use 搜索、test compilation、Graph suite、Gatekeeper/protocol tests 证明闭包；负向契约 test 可保留名称字符串，任何可执行类型依赖都视为 cleanup blocker。
|
||||
|
||||
### 2. current docs 改为显式 StateGraph，历史 docs 保留
|
||||
|
||||
必须更新：
|
||||
|
||||
- `mvp/architecture/README.md`
|
||||
- `agent-orchestration.md`
|
||||
- `session-trace-lifecycle.md`
|
||||
- `current-mvp-architecture.md`
|
||||
- `executor-evidence-pipeline-refactor.md`
|
||||
- `harness-quality-gates.md`
|
||||
- `feedback-architecture.md`
|
||||
- `retrieval-observability.md`
|
||||
- `mvp/eval/README.md`
|
||||
- `mvp/demo/README.md` / trace checklist
|
||||
|
||||
历史 archived issues/design-notes 和 legacy fixtures不批量替换：它们记录演进阶段或兼容旧 Trace。current docs 若提到 `tool_trace_summary`，只能标注为历史读取兼容，不能描述为新 Verifier 输入/持久化源。
|
||||
|
||||
### 3. Demo script 是 final E2E executable contract
|
||||
|
||||
扩展 `run-interview-demo-check.ps1`：
|
||||
|
||||
1. 从 Chat response 获取 runId。
|
||||
2. 查询 exact Trace。
|
||||
3. 要求 `trace.data.run.orchestrationTrace` 非空。
|
||||
4. 要求 version/final_node/termination_reason 非空，transitions 为数组。
|
||||
5. 把 finalNode、terminationReason、degraded、transitionCount、evidenceRetryCount 写入 summary。
|
||||
6. 保留 feedback 和 Prompt/Gatekeeper summary。
|
||||
|
||||
脚本必须 fail fast，不能用 session self-evaluation 或日志合成缺失的 Graph trace。
|
||||
|
||||
### 4. 自动化门禁先于 live E2E
|
||||
|
||||
顺序固定：cleanup source check → docs/script tests → Graph/Chat/Trace/Gatekeeper/Composer/Controller/Repository tests → diagnosis eval baseline/diff → test compilation → OpenSpec/static gates → live startup/E2E → logs → DB → process cleanup。
|
||||
|
||||
在 live 前发现的失败按 OpenSpec/code/test/documentation分类；live 后失败使用 diagnose loop，以 exact run evidence 定位，不通过放宽断言绕过。
|
||||
|
||||
### 5. Live E2E 使用唯一 identity 和可回收后台 Maven
|
||||
|
||||
- sessionId：`iss-011-stage5-<timestamp>`。
|
||||
- Maven：`spring-boot:run` + `mvp-demo` profile，后台 hidden process，stdout/stderr 写入 `target/` 临时文件。
|
||||
- readiness：轮询 9900，最长明确超时，不阻塞超过 60 秒且持续汇报。
|
||||
- 执行：复用 interview demo script，输出写入 `target/iss-011-stage5-output/`，避免覆盖已提交 demo sample。
|
||||
- 最终：停止应用进程树，确认 9900 无监听，删除临时启动/output文件。
|
||||
|
||||
如果 9900 启动前已被其他进程占用，先识别而不是杀掉未知进程；只有本轮启动的 PID 可被清理。
|
||||
|
||||
### 6. 日志与数据库证据只认本次 Run
|
||||
|
||||
启动前记录 `logs/application.log`、`application-error.log`、`chat.log` 长度和时间；E2E 后只读新增片段，并搜索 sessionId/runId、Graph执行、Flyway/JPA 和 ERROR。测试阶段写入的旧日志不计入 live 证据。
|
||||
|
||||
数据库通过 `scripts/query_mysql.py` 执行 read-only queries：
|
||||
|
||||
- `SHOW COLUMNS ... orchestration_trace`。
|
||||
- exact `diagnosis_run` status/flow/answer/metrics/self_evaluation/orchestration trace/feedback。
|
||||
- JSON_EXTRACT version/final_node/termination_reason/degraded/evidence_retry_count。
|
||||
- exact run AgentStep/ToolInvocation count 和 distinct ownership。
|
||||
- wrong session ownership count=0。
|
||||
|
||||
所有查询必须带 E2E sessionId/runId 或 schema column 条件；不接受全局 latest 代替。
|
||||
|
||||
### 7. Issue 归档是最后一个实现动作
|
||||
|
||||
E2E/log/DB 任一失败时 ISS-011 保持 active。全部通过后更新验收 checkbox和状态，将文件 move 到 `mvp/issues/archived/`，并把 issues index 从 active 移到 archived。OpenSpec archive 在 Issue move 后执行，Git commit 是阶段 5最后门禁。
|
||||
|
||||
## Interface Impact
|
||||
|
||||
- 等级：L2 内部类型删除 + demo/docs 增强。
|
||||
- 外部 `/api/chat`、Trace、feedback、DB schema 和 JSON 字段：不变。
|
||||
- 内部消费者：旧 Hook/ThreadLocal/service 无消费者，删除无需迁移调用点。
|
||||
- Demo script 行为：新增 fail-fast orchestration trace contract和 summary 字段；旧 Chat/Trace/feedback outputs保留。
|
||||
- 回滚：revert 阶段 5代码/文档；DB/生产数据无变更。E2E产生的 demo Run 是正常审计数据，不执行破坏性回滚。
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [隐藏反射/配置消费者未被 rg发现] → test compilation + Spring startup 是最终验证；发现即恢复并修正规格。
|
||||
- [文档仍有历史术语] → current docs 白名单扫描；history/design-notes允许但索引明确非当前真理源。
|
||||
- [live LLM 输出波动] → mock logs/metrics提供稳定工具证据；Graph允许安全 Fallback，但验收仍要求非空真实 orchestration trace和一致 Run生命周期。
|
||||
- [外部 DB/Redis/Milvus/LLM不可用] → readiness/log/DB diagnose；不伪造通过，不修改验收口径。
|
||||
- [停止进程误伤] → 只记录和终止本轮 Maven/Java PID，结束后用端口复核。
|
||||
- [E2E output覆盖仓库样例] → 输出放 target临时目录，证据摘要提炼进 devflow 后删除临时文件。
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. 删除旧闭包，运行 source/test compilation/Graph安全回归。
|
||||
2. 更新 current docs 和 demo script/checklist，增加静态脚本契约 test。
|
||||
3. 运行完整 focused tests、diagnosis eval baseline/diff、OpenSpec/static gates。
|
||||
4. Maven `mvp-demo` startup，运行 unique session E2E。
|
||||
5. 检查新增日志片段和 exact DB数据，停止进程、清理临时文件。
|
||||
6. 更新/移动 ISS-011，回填 devflow，archive OpenSpec，独立提交。
|
||||
|
||||
运行时回滚为 Git revert；数据库 V012列和 E2E Run可安全保留。
|
||||
|
||||
## Open Questions
|
||||
|
||||
无。清理闭包、current/history docs边界、E2E身份/证据和 Issue关闭门禁均已确定。
|
||||
+69
@@ -0,0 +1,69 @@
|
||||
# Chat Diagnosis StateGraph Cleanup, Final Acceptance And Documentation
|
||||
|
||||
## Why
|
||||
|
||||
阶段 0–4 已完成设计冻结、Graph 骨架、真实 Nodes、ChatService 生产切换和新测试体系，但仓库仍保留只服务旧 Sequential/Hook 链路的生产类，当前架构/评测/demo 文档仍把 Chat 描述为固定顺序、Hook Gatekeeper 和 full `tool_trace_summary`。阶段 5 需要删除这组死代码、将当前文档与真实 StateGraph 对齐，并首次运行统一 Maven live E2E、日志和数据库验收，形成 ISS-011 最终闭环。
|
||||
|
||||
## What Changes
|
||||
|
||||
- 删除无生产消费者的 `VerifierInputHook`、`VerifierContextHolder`、`ToolTraceSummaryService` 及其 focused test；保留仍被 Graph 使用的共享 parser/Gatekeeper/verified input 组件。
|
||||
- 更新当前架构、Trace 生命周期、证据管线、质量门禁、反馈、评测和 demo 文档，明确 StateGraph 条件边、verified-only Verifier、Run `orchestration_trace` 和三层测试体系。
|
||||
- 更新 interview demo check，使其强制读取 `run.orchestrationTrace`，并在 summary 输出 final node、termination reason、degraded、transition count 和 evidence retry count。
|
||||
- 运行新的 Graph/Chat/Trace/Controller/Repository/Gatekeeper/Composer/Eval 回归和固定 diagnosis eval baseline。
|
||||
- 使用 `mvn spring-boot:run` 启动 `mvp-demo` profile，运行唯一 session 的 Chat→exact Trace→feedback E2E。
|
||||
- 只检查本次 E2E 新增日志片段，并用 `scripts/query_mysql.py` 核验 V012、当前 Run、AgentStep/ToolInvocation、self-evaluation/orchestration trace、feedback 和跨 Run 隔离。
|
||||
- E2E 全部通过后将 ISS-011 从 active 移到 archived，更新 issues index 和验收 checkbox。
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `chat-diagnosis-stategraph-cleanup-docs`：规定旧编排死代码清理、当前架构文档对齐、最终自动化回归、Maven live E2E、日志/数据库证据和 Issue 关闭门禁。
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- `mvp-demo-trace-acceptance`：最终 interview demo 必须断言精确 Run 的非空 orchestration trace，并把 StateGraph 路由摘要纳入可复现证据包和检查清单。
|
||||
|
||||
## Scope
|
||||
|
||||
### In Scope
|
||||
|
||||
- 旧 Hook/ThreadLocal/trace-summary producer 闭包删除和引用清理。
|
||||
- 当前（非 archive/design history）架构、eval、demo 与 Issue 文档更新。
|
||||
- demo preflight 脚本的 orchestration trace assertion/summary 扩展。
|
||||
- 所有阶段 5单元/集成/评测门禁和最终 live E2E/log/DB 验收。
|
||||
- ISS-011 状态关闭、归档和阶段 5独立 Git commit。
|
||||
|
||||
### Out of Scope
|
||||
|
||||
- 不重写历史 archived issues、旧 design-notes 或旧 fixture payload；它们保留时间点语义。
|
||||
- 不删除 Trace UI/eval 对历史 `tool_trace_summary` 的只读兼容支持。
|
||||
- 不改变 Graph 路由、Prompt、API、DTO、DB schema 或诊断业务行为；发现生产偏差时按实现期冲突规则处理。
|
||||
- 不 push，不清理仓库已有凭据/基础设施配置；这些不属于 ISS-011。
|
||||
|
||||
## Context Constraints
|
||||
|
||||
- `VerifierInputHook` 与 `VerifierContextHolder` 当前只互相引用；`ToolTraceSummaryService` 只被该 Hook 和自身测试使用，因此四文件构成可删除闭包。
|
||||
- `ExecutorEvidenceParser`、`ExecutorGatekeeperService`、`GatekeeperNode`、`VerifiedInputNode`、`VerifierNodeAdapter` 和 `DiagnosisGraphResultMapper` 是新链路真理源，不得随旧闭包删除。
|
||||
- 新 StateGraph Chat run 的 `run.orchestrationTrace` 必须非空；顶层/session 不重复，历史 null 仍兼容。
|
||||
- demo summary 必须从 exact run trace 读取 Graph 摘要，不能从日志反推路由。
|
||||
- 日志验收只分析本次启动/Run 的新增内容；数据库查询必须使用 exact sessionId+runId，不能以 latest 全局记录代替。
|
||||
- live E2E 必须在所有实现、单元测试、eval、static gates 通过后执行，并在结束后关闭应用进程/端口。
|
||||
|
||||
## Acceptance
|
||||
|
||||
- 生产/测试源码中不存在 `VerifierInputHook`、`VerifierContextHolder`、`ToolTraceSummaryService` 定义、import、实例化或类型依赖；负向契约测试 MAY 保留名称字符串以阻止回归；Graph verified-only、安全 Fallback 和 Gatekeeper tests仍全绿。
|
||||
- 当前架构/Trace/eval/demo 文档不再把 Chat 描述为 SequentialAgent/Hook Gatekeeper/full tool trace Verifier。
|
||||
- demo check 对 `run.orchestrationTrace` 缺失 fail fast，并把路由摘要写入 interview summary。
|
||||
- 新 Graph suite、保留安全回归、test compilation、fixed diagnosis eval baseline、OpenSpec/static gates全部通过。
|
||||
- Maven `mvp-demo` live startup 成功；Chat/Trace/feedback 请求成功并返回精确 runId。
|
||||
- 新日志片段可关联本次 sessionId/runId，无未解释 ERROR；数据库证明 Run=SUCCESS/CHAT、answer/metrics/evaluation/trace 非空、feedback=useful、步骤/工具属于当前 run、V012 列存在且 trace JSON 可解析。
|
||||
- 应用进程和 9900 端口最终清理；ISS-011 移到 archived 并更新 index/checklist。
|
||||
|
||||
## Risks
|
||||
|
||||
- 删除 `ToolTraceSummaryService` 可能遗漏隐藏消费者；删除前后使用全仓引用搜索、test compilation 和 retained tests验证。
|
||||
- 当前架构文档引用面广；只更新 architecture index 声明的 current docs 和 active demo/eval docs，历史 archive/design-notes 保持不变。
|
||||
- live LLM/外部基础设施存在波动；先做 readiness，失败时按日志/Run/Trace/DB 证据 diagnose，禁止把环境失败伪装为验收通过。
|
||||
- E2E 可能污染已有 demo session；使用时间戳唯一 sessionId 和 exact runId 查询，并在证据中记录。
|
||||
- Maven/Java 子进程可能遗留；以端口和进程双重检查清理，避免影响后续工作。
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Legacy implicit verifier orchestration SHALL be removed
|
||||
|
||||
The final StateGraph implementation SHALL have no production or test definition, import, instantiation, or executable type dependency for the legacy Verifier Hook, its ThreadLocal context, or its dedicated full-trace summary producer. Negative contract tests MAY retain identifier strings solely to prevent regression.
|
||||
|
||||
#### Scenario: Legacy source inventory is inspected
|
||||
- **WHEN** stage 5 cleanup completes
|
||||
- **THEN** `VerifierInputHook`, `VerifierContextHolder`, and `ToolTraceSummaryService` source files SHALL NOT exist
|
||||
- **AND** no production or test source SHALL import, instantiate, extend, or type-reference those types
|
||||
- **AND** identifier string literals SHALL only be allowed in negative source/Prompt contract assertions
|
||||
|
||||
#### Scenario: Graph safety contracts are rerun
|
||||
- **WHEN** the legacy closure is removed
|
||||
- **THEN** Executor parser, Gatekeeper service/node, Verified Input, Verifier, Composer, Fallback, Trace, and Chat integration tests SHALL pass
|
||||
- **AND** Maven test compilation and live Spring startup SHALL succeed without the removed beans
|
||||
|
||||
### Requirement: Current documentation SHALL describe the real StateGraph architecture
|
||||
|
||||
Current architecture, eval, and demo documentation SHALL describe complex Chat as explicit bounded Diagnosis StateGraph orchestration with run-owned events and verified-only Verifier input.
|
||||
|
||||
#### Scenario: Current docs are inspected
|
||||
- **WHEN** a maintainer follows the architecture index and active eval/demo guides
|
||||
- **THEN** Chat orchestration SHALL be described as conditional StateGraph Nodes rather than SequentialAgent or Hook Gatekeeper
|
||||
- **AND** Verifier input SHALL use verified Executor projection/evidence rather than full tool trace
|
||||
- **AND** `run.orchestrationTrace` SHALL be documented separately from self-evaluation and detailed Agent/tool Trace
|
||||
|
||||
#### Scenario: Historical docs are inspected
|
||||
- **WHEN** a maintainer reads archived issues, design notes, or legacy fixtures
|
||||
- **THEN** those materials MAY retain their original Hook/full-trace terminology
|
||||
- **AND** they SHALL NOT be indexed as the current implementation truth
|
||||
|
||||
### Requirement: Final deterministic regression SHALL remain green
|
||||
|
||||
The project SHALL run the authoritative Graph suite, retained public/security contracts, Maven test compilation, and fixed diagnosis eval baseline before live acceptance.
|
||||
|
||||
#### Scenario: Final automated gates run
|
||||
- **WHEN** stage 5 implementation and documentation are complete
|
||||
- **THEN** Workflow, Node Contract, Chat Integration, Trace, Controller, Repository, Gatekeeper, Composer, parser/projection, and Eval tests SHALL pass
|
||||
- **AND** the fixed diagnosis eval report/diff SHALL show no unexpected regression
|
||||
- **AND** OpenSpec strict and source/whitespace checks SHALL pass
|
||||
|
||||
### Requirement: Final live E2E SHALL prove exact StateGraph Run ownership
|
||||
|
||||
The final acceptance SHALL start the application through Maven with the `mvp-demo` profile and SHALL execute Chat, exact Trace, and feedback requests using one unique sessionId and the returned runId.
|
||||
|
||||
#### Scenario: Live Chat Graph completes
|
||||
- **WHEN** the unique payment-timeout demo request completes
|
||||
- **THEN** the response SHALL be successful with a non-empty answer/sessionId/runId
|
||||
- **AND** exact Trace SHALL return the same runId with a non-empty `run.orchestrationTrace`
|
||||
- **AND** orchestration trace SHALL include version, final node, termination reason, transitions, degraded, and evidence retry count
|
||||
- **AND** feedback SHALL attach to the same runId
|
||||
|
||||
#### Scenario: Live application is cleaned up
|
||||
- **WHEN** live verification finishes or fails
|
||||
- **THEN** the Maven/Java processes started by this stage SHALL be stopped
|
||||
- **AND** port 9900 SHALL no longer be owned by the stage 5 process
|
||||
- **AND** temporary target output/log files SHALL be removed after evidence extraction
|
||||
|
||||
### Requirement: Final logs and database SHALL corroborate the E2E Run
|
||||
|
||||
Stage 5 SHALL inspect only the current live run's new log segment and SHALL query exact database rows with the repository MySQL tool.
|
||||
|
||||
#### Scenario: New logs are inspected
|
||||
- **WHEN** E2E returns sessionId and runId
|
||||
- **THEN** new application/chat logs SHALL contain evidence for the current request/run lifecycle
|
||||
- **AND** no unexplained ERROR in the new stage 5 segment SHALL invalidate acceptance
|
||||
|
||||
#### Scenario: Exact database run is queried
|
||||
- **WHEN** `scripts/query_mysql.py` queries the E2E sessionId/runId
|
||||
- **THEN** V012 orchestration column SHALL exist
|
||||
- **AND** DiagnosisRun SHALL be CHAT/SUCCESS with non-empty answer, metrics, self-evaluation, orchestration trace, and useful feedback
|
||||
- **AND** orchestration JSON fields SHALL match the exact Trace response
|
||||
- **AND** AgentStep/ToolInvocation ownership checks SHALL contain no row from another run/session
|
||||
|
||||
### Requirement: ISS-011 SHALL close only after all final gates pass
|
||||
|
||||
The Issue SHALL remain active until cleanup, docs, deterministic regression, live E2E, logs, database, process cleanup, and OpenSpec validation are complete.
|
||||
|
||||
#### Scenario: Any final gate fails
|
||||
- **WHEN** a required stage 5 gate is incomplete or failed
|
||||
- **THEN** ISS-011 SHALL remain active
|
||||
- **AND** acceptance SHALL record the blocker without marking the OpenSpec complete
|
||||
|
||||
#### Scenario: All final gates pass
|
||||
- **WHEN** all stage 5 acceptance evidence is archived
|
||||
- **THEN** ISS-011 checkboxes/status SHALL be completed
|
||||
- **AND** the Issue SHALL move from active to archived with its index entry updated
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: End-to-end MVP acceptance case is documented
|
||||
The project SHALL include an end-to-end acceptance case that demonstrates Maven start-up, chat diagnosis, exact trace query, orchestration trace inspection, feedback submission, log inspection, and exact database verification using the same `sessionId + runId`.
|
||||
|
||||
#### Scenario: Reviewer follows the acceptance case
|
||||
- **WHEN** a reviewer follows the documented MVP demo acceptance steps
|
||||
- **THEN** they can run the application, submit a diagnosis question, query the exact trace endpoint, inspect `run.orchestrationTrace`, and submit feedback for the same run
|
||||
- **AND** they can correlate that run with new logs and exact DiagnosisRun/AgentStep/ToolInvocation database records
|
||||
|
||||
### Requirement: MVP demo SHALL be reproducible for interviews
|
||||
The MVP demo SHALL provide a repeatable way to show a diagnosis answer, exact run trace, StateGraph orchestration summary, verifier evaluation, and feedback.
|
||||
|
||||
#### Scenario: interview demo check script records an evidence bundle
|
||||
- **WHEN** the user runs the interview demo check script against a running `mvp-demo` service
|
||||
- **THEN** the script SHALL submit a fixed Chat diagnosis request
|
||||
- **AND** it SHALL fetch the trace for the same `sessionId + runId`
|
||||
- **AND** it SHALL fail if `run.orchestrationTrace` or its version/final node/termination reason is missing
|
||||
- **AND** it SHALL submit useful feedback for that run
|
||||
- **AND** it SHALL write chat, trace, feedback, and summary outputs under the configured output directory
|
||||
- **AND** summary SHALL include final node, termination reason, degraded, transition count, and evidence retry count
|
||||
|
||||
#### Scenario: interview demo check fails with actionable readiness output
|
||||
- **WHEN** the target service is not reachable
|
||||
- **THEN** the script SHALL fail before issuing diagnosis requests
|
||||
- **AND** the failure message SHALL name the base URL and the expected startup profile
|
||||
|
||||
#### Scenario: interview documentation explains audit fields
|
||||
- **WHEN** an interviewer asks how Prompt, Gatekeeper, or Graph routing changes are audited
|
||||
- **THEN** the demo documentation SHALL point to `prompt_audit.version`, `gatekeeper_result.rule_set_version`, and `run.orchestrationTrace`
|
||||
- **AND** it SHALL explain that deterministic eval fixtures and the Graph test suite are the regression source of truth
|
||||
|
||||
### Requirement: MVP demo SHALL provide a trace inspection checklist
|
||||
The MVP demo SHALL document which trace fields to inspect for StateGraph routing, evidence, verifier behavior, and run-level auditability.
|
||||
|
||||
#### Scenario: Checklist maps fields to interview claims
|
||||
- **WHEN** a developer reviews an exact trace response
|
||||
- **THEN** the checklist SHALL map `run.orchestrationTrace` version/transitions/final node/termination reason/degraded/evidence retry count to Graph routing claims
|
||||
- **AND** it SHALL map AgentStep, ToolInvocation, self-evaluation, Prompt audit, Gatekeeper audit, answer, and feedback paths to their separate responsibilities
|
||||
@@ -0,0 +1,52 @@
|
||||
## 1. Legacy Orchestration Closure Removal
|
||||
|
||||
- [x] 1.1 Delete `VerifierInputHook`, `VerifierContextHolder`, `ToolTraceSummaryService`, and `ToolTraceSummaryServiceTest` as one proven-unused closure.
|
||||
- [x] 1.2 Run whole-repository definition/import/instantiation/type-use searches proving no executable production/test source depends on the removed types; allow only negative guard strings.
|
||||
- [x] 1.3 Run Executor parser, Gatekeeper service/node, VerifiedInput, Verifier, Composer, Fallback, Chat integration, and test compilation regressions after deletion.
|
||||
- [x] 1.4 Confirm no Graph/shared protocol component or Spring bean required by the current path was removed.
|
||||
|
||||
## 2. Current Architecture And Eval Documentation
|
||||
|
||||
- [x] 2.1 Update architecture index and `agent-orchestration.md` to explicit bounded StateGraph, verified-only Verifier input, conditional retry/Fallback, and orchestration trace.
|
||||
- [x] 2.2 Update `session-trace-lifecycle.md` and `current-mvp-architecture.md` for StateGraph Run lifecycle, `orchestration_trace`, explicit Gatekeeper Node, and new test layers.
|
||||
- [x] 2.3 Update current evidence-pipeline/quality-gate/feedback/retrieval docs so full `tool_trace_summary` is historical compatibility rather than new Verifier input.
|
||||
- [x] 2.4 Update `mvp/eval/README.md` regression commands and architecture descriptions to the authoritative Graph suite.
|
||||
- [x] 2.5 Run a current-doc source scan proving no current truth document describes Chat as SequentialAgent/Hook Gatekeeper/full-trace Verifier.
|
||||
|
||||
## 3. Demo Orchestration Trace Contract
|
||||
|
||||
- [x] 3.1 Extend `run-interview-demo-check.ps1` to fail fast when exact `run.orchestrationTrace` or required routing fields are missing.
|
||||
- [x] 3.2 Add final node, termination reason, degraded, transition count, and evidence retry count to interview summary output.
|
||||
- [x] 3.3 Update demo README and trace checklist with exact run orchestration trace fields and responsibility separation.
|
||||
- [x] 3.4 Add/extend static script contract tests for exact runId use, orchestration assertion, and summary fields without requiring a live service.
|
||||
|
||||
## 4. Final Deterministic Regression
|
||||
|
||||
- [x] 4.1 Run authoritative Workflow/Node Contract/Chat Integration and all focused Graph/Trace/Gatekeeper/Composer/Controller/Repository tests.
|
||||
- [x] 4.2 Run fixed `DiagnosisTraceEvaluatorTest` and `DiagnosisEvalBaselineDiffTest`; confirm baseline count/distribution and no unexpected diff.
|
||||
- [x] 4.3 Run Maven test compilation and any docs/script contract tests touched by cleanup.
|
||||
- [x] 4.4 Run current change strict validation, all main specs strict validation, `git diff --check`, legacy-source scan, current-doc scan, and no-unexpected-schema check.
|
||||
|
||||
## 5. Maven Live E2E Startup And Requests
|
||||
|
||||
- [x] 5.1 Confirm port 9900 is free, record pre-start log lengths/timestamps, and create a unique `iss-011-stage5-<timestamp>` sessionId.
|
||||
- [x] 5.2 Start Maven `spring-boot:run` with the `mvp-demo` profile as a hidden tracked background process and wait for readiness with bounded polling.
|
||||
- [x] 5.3 Run the interview demo check into a target-only output directory and capture Chat/Trace/feedback/summary for the exact returned runId.
|
||||
- [x] 5.4 Assert live response success, non-empty answer, exact runId, Graph summary fields, Agent/tool evidence, evaluation audit, and useful feedback.
|
||||
|
||||
## 6. Live Logs And Database Evidence
|
||||
|
||||
- [x] 6.1 Read only the new stage-5 application/chat/error log segments and correlate the current sessionId/runId/Graph lifecycle.
|
||||
- [x] 6.2 Classify every ERROR in the new segment; leave no unexplained error in accepted evidence.
|
||||
- [x] 6.3 Use `scripts/query_mysql.py` to prove V012 column existence and exact Run CHAT/SUCCESS/answer/metrics/self-evaluation/orchestration trace/feedback fields.
|
||||
- [x] 6.4 Query JSON routing fields and compare version/final node/termination/degraded/evidence retry count with the exact Trace response.
|
||||
- [x] 6.5 Query AgentStep/ToolInvocation counts and distinct session/run ownership; prove wrong-session/run rows are zero.
|
||||
|
||||
## 7. Cleanup, Issue Closure And Handoff
|
||||
|
||||
- [x] 7.1 Stop only the Maven/Java processes started by stage 5 and confirm port 9900 is released even on failure.
|
||||
- [x] 7.2 Remove target-only E2E output/startup files after extracting durable acceptance evidence.
|
||||
- [x] 7.3 Update ISS-011 final checklist/status and move it from `mvp/issues/active` to `mvp/issues/archived` only after every gate passes.
|
||||
- [x] 7.4 Update `mvp/issues/README.md` and current documentation links/status for archived ISS-011.
|
||||
- [x] 7.5 Record deterministic, live E2E, log, database, process cleanup, known limits, and exact identities in devflow acceptance/evidence.
|
||||
- [x] 7.6 Run final OpenSpec/static/worktree scope checks and prepare stage 5 Archive/independent Git commit without pushing.
|
||||
@@ -0,0 +1,91 @@
|
||||
# chat-diagnosis-stategraph-cleanup-docs Specification
|
||||
|
||||
## Purpose
|
||||
TBD - created by archiving change chat-diagnosis-stategraph-cleanup-docs. Update Purpose after archive.
|
||||
## Requirements
|
||||
### Requirement: Legacy implicit verifier orchestration SHALL be removed
|
||||
|
||||
The final StateGraph implementation SHALL have no production or test definition, import, instantiation, or executable type dependency for the legacy Verifier Hook, its ThreadLocal context, or its dedicated full-trace summary producer. Negative contract tests MAY retain identifier strings solely to prevent regression.
|
||||
|
||||
#### Scenario: Legacy source inventory is inspected
|
||||
- **WHEN** stage 5 cleanup completes
|
||||
- **THEN** `VerifierInputHook`, `VerifierContextHolder`, and `ToolTraceSummaryService` source files SHALL NOT exist
|
||||
- **AND** no production or test source SHALL import, instantiate, extend, or type-reference those types
|
||||
- **AND** identifier string literals SHALL only be allowed in negative source/Prompt contract assertions
|
||||
|
||||
#### Scenario: Graph safety contracts are rerun
|
||||
- **WHEN** the legacy closure is removed
|
||||
- **THEN** Executor parser, Gatekeeper service/node, Verified Input, Verifier, Composer, Fallback, Trace, and Chat integration tests SHALL pass
|
||||
- **AND** Maven test compilation and live Spring startup SHALL succeed without the removed beans
|
||||
|
||||
### Requirement: Current documentation SHALL describe the real StateGraph architecture
|
||||
|
||||
Current architecture, eval, and demo documentation SHALL describe complex Chat as explicit bounded Diagnosis StateGraph orchestration with run-owned events and verified-only Verifier input.
|
||||
|
||||
#### Scenario: Current docs are inspected
|
||||
- **WHEN** a maintainer follows the architecture index and active eval/demo guides
|
||||
- **THEN** Chat orchestration SHALL be described as conditional StateGraph Nodes rather than SequentialAgent or Hook Gatekeeper
|
||||
- **AND** Verifier input SHALL use verified Executor projection/evidence rather than full tool trace
|
||||
- **AND** `run.orchestrationTrace` SHALL be documented separately from self-evaluation and detailed Agent/tool Trace
|
||||
|
||||
#### Scenario: Historical docs are inspected
|
||||
- **WHEN** a maintainer reads archived issues, design notes, or legacy fixtures
|
||||
- **THEN** those materials MAY retain their original Hook/full-trace terminology
|
||||
- **AND** they SHALL NOT be indexed as the current implementation truth
|
||||
|
||||
### Requirement: Final deterministic regression SHALL remain green
|
||||
|
||||
The project SHALL run the authoritative Graph suite, retained public/security contracts, Maven test compilation, and fixed diagnosis eval baseline before live acceptance.
|
||||
|
||||
#### Scenario: Final automated gates run
|
||||
- **WHEN** stage 5 implementation and documentation are complete
|
||||
- **THEN** Workflow, Node Contract, Chat Integration, Trace, Controller, Repository, Gatekeeper, Composer, parser/projection, and Eval tests SHALL pass
|
||||
- **AND** the fixed diagnosis eval report/diff SHALL show no unexpected regression
|
||||
- **AND** OpenSpec strict and source/whitespace checks SHALL pass
|
||||
|
||||
### Requirement: Final live E2E SHALL prove exact StateGraph Run ownership
|
||||
|
||||
The final acceptance SHALL start the application through Maven with the `mvp-demo` profile and SHALL execute Chat, exact Trace, and feedback requests using one unique sessionId and the returned runId.
|
||||
|
||||
#### Scenario: Live Chat Graph completes
|
||||
- **WHEN** the unique payment-timeout demo request completes
|
||||
- **THEN** the response SHALL be successful with a non-empty answer/sessionId/runId
|
||||
- **AND** exact Trace SHALL return the same runId with a non-empty `run.orchestrationTrace`
|
||||
- **AND** orchestration trace SHALL include version, final node, termination reason, transitions, degraded, and evidence retry count
|
||||
- **AND** feedback SHALL attach to the same runId
|
||||
|
||||
#### Scenario: Live application is cleaned up
|
||||
- **WHEN** live verification finishes or fails
|
||||
- **THEN** the Maven/Java processes started by this stage SHALL be stopped
|
||||
- **AND** port 9900 SHALL no longer be owned by the stage 5 process
|
||||
- **AND** temporary target output/log files SHALL be removed after evidence extraction
|
||||
|
||||
### Requirement: Final logs and database SHALL corroborate the E2E Run
|
||||
|
||||
Stage 5 SHALL inspect only the current live run's new log segment and SHALL query exact database rows with the repository MySQL tool.
|
||||
|
||||
#### Scenario: New logs are inspected
|
||||
- **WHEN** E2E returns sessionId and runId
|
||||
- **THEN** new application/chat logs SHALL contain evidence for the current request/run lifecycle
|
||||
- **AND** no unexplained ERROR in the new stage 5 segment SHALL invalidate acceptance
|
||||
|
||||
#### Scenario: Exact database run is queried
|
||||
- **WHEN** `scripts/query_mysql.py` queries the E2E sessionId/runId
|
||||
- **THEN** V012 orchestration column SHALL exist
|
||||
- **AND** DiagnosisRun SHALL be CHAT/SUCCESS with non-empty answer, metrics, self-evaluation, orchestration trace, and useful feedback
|
||||
- **AND** orchestration JSON fields SHALL match the exact Trace response
|
||||
- **AND** AgentStep/ToolInvocation ownership checks SHALL contain no row from another run/session
|
||||
|
||||
### Requirement: ISS-011 SHALL close only after all final gates pass
|
||||
|
||||
The Issue SHALL remain active until cleanup, docs, deterministic regression, live E2E, logs, database, process cleanup, and OpenSpec validation are complete.
|
||||
|
||||
#### Scenario: Any final gate fails
|
||||
- **WHEN** a required stage 5 gate is incomplete or failed
|
||||
- **THEN** ISS-011 SHALL remain active
|
||||
- **AND** acceptance SHALL record the blocker without marking the OpenSpec complete
|
||||
|
||||
#### Scenario: All final gates pass
|
||||
- **WHEN** all stage 5 acceptance evidence is archived
|
||||
- **THEN** ISS-011 checkboxes/status SHALL be completed
|
||||
- **AND** the Issue SHALL move from active to archived with its index entry updated
|
||||
@@ -37,11 +37,12 @@ The system SHALL provide an `mvp-demo` Spring profile that documents the demo ru
|
||||
- **THEN** `prometheus.mock-enabled` and `cls.mock-enabled` are enabled by profile configuration
|
||||
|
||||
### Requirement: End-to-end MVP acceptance case is documented
|
||||
The project SHALL include an end-to-end acceptance case that demonstrates start-up, chat diagnosis, trace query, and feedback submission using the same `sessionId + runId`.
|
||||
The project SHALL include an end-to-end acceptance case that demonstrates Maven start-up, chat diagnosis, exact trace query, orchestration trace inspection, feedback submission, log inspection, and exact database verification using the same `sessionId + runId`.
|
||||
|
||||
#### Scenario: Reviewer follows the acceptance case
|
||||
- **WHEN** a reviewer follows the documented MVP demo acceptance steps
|
||||
- **THEN** they can run the application, submit a diagnosis question, query the exact trace endpoint, and submit feedback for the same run
|
||||
- **THEN** they can run the application, submit a diagnosis question, query the exact trace endpoint, inspect `run.orchestrationTrace`, and submit feedback for the same run
|
||||
- **AND** they can correlate that run with new logs and exact DiagnosisRun/AgentStep/ToolInvocation database records
|
||||
|
||||
### Requirement: MVP demo SHALL provide an interview runbook
|
||||
The MVP demo SHALL include a concise interview runbook that explains how to demonstrate the Agent flow and how to narrate the engineering value.
|
||||
@@ -67,14 +68,16 @@ The MVP demo SHALL provide scripts and request payloads for running the payment-
|
||||
- **THEN** it SHALL write chat, exact trace, and feedback responses under a demo output directory
|
||||
|
||||
### Requirement: MVP demo SHALL be reproducible for interviews
|
||||
The MVP demo SHALL provide a repeatable way to show a diagnosis answer, trace, verifier evaluation, and feedback.
|
||||
The MVP demo SHALL provide a repeatable way to show a diagnosis answer, exact run trace, StateGraph orchestration summary, verifier evaluation, and feedback.
|
||||
|
||||
#### Scenario: interview demo check script records an evidence bundle
|
||||
- **WHEN** the user runs the interview demo check script against a running `mvp-demo` service
|
||||
- **THEN** the script SHALL submit a fixed Chat diagnosis request
|
||||
- **AND** it SHALL fetch the trace for the same `sessionId + runId`
|
||||
- **AND** it SHALL fail if `run.orchestrationTrace` or its version/final node/termination reason is missing
|
||||
- **AND** it SHALL submit useful feedback for that run
|
||||
- **AND** it SHALL write chat, trace, feedback, and summary outputs under `mvp/demo/output/`
|
||||
- **AND** it SHALL write chat, trace, feedback, and summary outputs under the configured output directory
|
||||
- **AND** summary SHALL include final node, termination reason, degraded, transition count, and evidence retry count
|
||||
|
||||
#### Scenario: interview demo check fails with actionable readiness output
|
||||
- **WHEN** the target service is not reachable
|
||||
@@ -82,16 +85,17 @@ The MVP demo SHALL provide a repeatable way to show a diagnosis answer, trace, v
|
||||
- **AND** the failure message SHALL name the base URL and the expected startup profile
|
||||
|
||||
#### Scenario: interview documentation explains audit fields
|
||||
- **WHEN** an interviewer asks how prompt or Gatekeeper changes are audited
|
||||
- **THEN** the demo documentation SHALL point to `prompt_audit.version` and `gatekeeper_result.rule_set_version`
|
||||
- **AND** it SHALL explain that deterministic eval fixtures are the regression source of truth
|
||||
- **WHEN** an interviewer asks how Prompt, Gatekeeper, or Graph routing changes are audited
|
||||
- **THEN** the demo documentation SHALL point to `prompt_audit.version`, `gatekeeper_result.rule_set_version`, and `run.orchestrationTrace`
|
||||
- **AND** it SHALL explain that deterministic eval fixtures and the Graph test suite are the regression source of truth
|
||||
|
||||
### Requirement: MVP demo SHALL provide a trace inspection checklist
|
||||
The MVP demo SHALL document which trace fields to inspect for evidence, verifier behavior, and run-level auditability.
|
||||
The MVP demo SHALL document which trace fields to inspect for StateGraph routing, evidence, verifier behavior, and run-level auditability.
|
||||
|
||||
#### Scenario: Checklist maps fields to interview claims
|
||||
- **WHEN** a developer reviews a trace response
|
||||
- **THEN** the checklist SHALL map concrete JSON paths to the claims made in the interview walkthrough
|
||||
- **WHEN** a developer reviews an exact trace response
|
||||
- **THEN** the checklist SHALL map `run.orchestrationTrace` version/transitions/final node/termination reason/degraded/evidence retry count to Graph routing claims
|
||||
- **AND** it SHALL map AgentStep, ToolInvocation, self-evaluation, Prompt audit, Gatekeeper audit, answer, and feedback paths to their separate responsibilities
|
||||
|
||||
### Requirement: MVP demo SHALL provide a browser trace workbench
|
||||
The MVP demo SHALL provide a browser-accessible static page for inspecting one
|
||||
|
||||
Reference in New Issue
Block a user