test(graph): establish diagnosis stategraph suite

This commit is contained in:
zhuyongxin
2026-07-17 18:53:29 +08:00
parent 99e490f227
commit 208a231113
19 changed files with 868 additions and 395 deletions
@@ -0,0 +1,3 @@
committed_at: 2026-07-17
checkpoint: Commit
authorization: user-requested-direct-implementation
@@ -0,0 +1,83 @@
## Context
阶段 3 已将复杂 Chat 单轨切换到 Diagnosis StateGraph,并以 119 个 focused tests 证明生产入口、路由、Node、Trace 和 Eval。当前测试实现按阶段累积:`DiagnosisGraphRoutingTest` 是完整 Fake workflow,`DiagnosisRealGraphIntegrationTest` 验证真实 Node assembly,多个专用 Node/protocol tests 锁定局部契约,`ChatServiceGraphIntegrationTest` 验证 Run 生命周期;另有 `VerifierInputHookTest` 仍锁定已退出生产路径的 raw/full-trace/ThreadLocal payload。
ISS-011 阶段 4只改变测试架构。生产源码、API、DTO、DB、Prompt 和路由均不得修改。目标是建立三个权威入口,同时保留小而精确的专用 tests,避免把全部覆盖合并成难维护的大类。
## Goals / Non-Goals
**Goals:**
- 用 `DiagnosisGraphWorkflowTest` 表达所有条件边、有限重试、终止和 event 顺序。
- 用 `DiagnosisGraphNodeContractTest` 表达真实 Node assembly 的输入投影、config identity、Gatekeeper/verified evidence、status/verdict 和安全材料边界。
- 用 `ChatServiceGraphIntegrationTest` 表达 public ChatResult、Run/Trace/evaluation、SUCCESS/FAILED 和 multi-run 隔离。
- 建立 Issue 阶段 4路径矩阵,补齐 partial-pass REJECT、tool-failure-but-valid-output、invalid Verifier 无伪 verdict 等高价值缺口。
- 删除 `VerifierInputHookTest`,并证明其仍有价值的 parser/Gatekeeper/投影行为已由独立 tests承接。
**Non-Goals:**
- 不修改生产实现或为了测试引入额外 public seam。
- 不删除生产 `VerifierInputHook`/`VerifierContextHolder` 类型;阶段 5清理。
- 不强制删除所有细粒度 Node/protocol tests。
- 不运行 Maven live E2E、日志或真实数据库验收。
## Decisions
### 1. 重命名现有权威类,不复制路由测试
将 `DiagnosisGraphRoutingTest` 原样演进为 `DiagnosisGraphWorkflowTest`。它已经使用 `ScriptedDiagnosisGraphActions` 驱动真实 CompiledGraph,覆盖 counter、edge、sequence 和 trace;复制一份新类只会形成两个路由真理源。
替代方案是保留旧类再加 facade/suite class,但 Maven/JUnit suite 依赖和重复发现没有新增验证价值,因此拒绝。
### 2. 真实 Node assembly integration 成为 Node Contract 权威入口
将 `DiagnosisRealGraphIntegrationTest` 演进为 `DiagnosisGraphNodeContractTest`。该类跨 Planner/Executor/Gatekeeper/VerifiedInput/Verifier/Composer 观察 invoker input、Gatekeeper调用次数、完整 snapshot 和 safe fallback,最符合“节点输入输出契约”的架构边界。
专用 `PlannerNodeAdapterTest`、`VerifiedInputNodeTest` 等继续保留,用于精确定位单组件失败;Node Contract 不复制它们全部断言,只补跨节点组合缺口。
### 3. 覆盖矩阵按行为归属,不按历史类迁移
- Workflow:PASS、Planner/Verifier/Composer retry 与 exhaustion、Executor status、Gatekeeper outcome、evidence retry、Verifier REJECT、终止/trace。
- Node Contract:adapter/config、合法 no-evidence/工具失败说明、partial-pass REJECT、verified-only projection、ceiling、完整 snapshot revalidation、Composer/Fallback材料。
- Chat Integration:Run start/complete/fail、metrics/Eval、orchestration trace、self-evaluation、cleanup、multi-run。
同一安全边界可以有 unit + integration 两层证据,但不得复制大段 fixture;复用现有 helpers 和 constants。
### 4. 删除 Hook test,不迁移旧 payload
`VerifierInputHookTest` 的 raw Executor、full tool trace、ThreadLocal 和 Hook Gatekeeper payload 与阶段 3 verified-only生产协议冲突。其 parser sanitization 由 `ExecutorEvidenceParserTest`,引用真实性由 `ExecutorGatekeeperServiceTest`,Graph status/audit 由 `GatekeeperNodeTest`,passed-binding projection 由 `VerifiedInputNodeTest` 覆盖。
阶段 4删除测试但保留生产类型,避免把阶段 5清理提前混入测试重构。若阶段 5删除类型,已经没有测试迁移阻力。
### 5. 测试失败按三类处理,默认 production diff=0
若新矩阵发现失败:规格缺口则先修正 OpenSpec;生产实现偏离既有 specs 才允许修复生产代码;测试命名/fixture 偏差只修测试。阶段 4正常验收要求 `src/main` 无 diff,从而保持提交职责单一。
## Interface Impact
- 等级:L1 test-only。
- 外部行为、API、DTO、DB、Prompt、Graph State 和消费者:无变化。
- 测试发现生产偏差时才可能升级影响级别,并必须回写 OpenSpec;当前没有此类发现。
- 回滚:revert 测试重构提交;不涉及数据或运行时迁移。
## Risks / Trade-offs
- [类重命名让历史链接失效] → devflow/archive 记录 old→new 映射,Git rename 可追踪历史。
- [三个权威类变成巨型文件] → 保留细粒度专用 unit tests,权威类只承载跨组件/路径矩阵。
- [删除 Hook test 丢失安全边界] → 删除前运行 parser/Gatekeeper/VerifiedInput保留集合并在 acceptance 记录映射。
- [仅有测试名没有真实矩阵] → tasks 明确补 3 个跨节点缺口并对照 Issue 全列表审计。
- [repository/integration tests 写本地日志被误认为 live E2E] → 验收明确区分 Maven tests 与 Maven 应用启动;阶段 4不检查日志/数据库。
## Migration Plan
1. Rename Routing→Workflow 和 RealGraphIntegration→NodeContract,不改变测试逻辑,先跑 GREEN。
2. 以 Issue 矩阵审计现有 method coverage,补 partial-pass REJECT、tool failure但合法结构、Verifier failure无伪 verdict 等缺口。
3. 删除 `VerifierInputHookTest`,运行对应 parser/Gatekeeper/VerifiedInput回归证明安全覆盖未丢失。
4. 扩展/复核 ChatService integration 与保留 Controller/Trace/Repository/Composer/Eval tests。
5. 运行 test compilation、OpenSpec/static gates,归档并独立提交。
回滚只需 revert 阶段 4提交,不影响生产运行。
## Open Questions
无。测试职责、旧 Hook test 退役和 E2E 阶段边界均由代码证据、阶段 0 spec 与用户规则确定。
@@ -0,0 +1,63 @@
# Chat Diagnosis StateGraph Test Suite
## Why
阶段 1–3 已建立完整 Graph 路由、真实 Node 和 ChatService cutover 测试,但核心覆盖仍分散在 `DiagnosisGraphRoutingTest`、多个单 Node test、真实 Graph integration 和已迁移前的 `VerifierInputHookTest`。阶段 4 需要让测试结构与 ISS-011 的正式架构一致:以 Graph workflow、Node contract、ChatService 外部生命周期三层为权威入口,不继续让 Sequential/Hook 内部实现成为长期约束。
## What Changes
- 将纯条件边/重试/终止矩阵收敛为 `DiagnosisGraphWorkflowTest`,保留 Fake Node、精确事件序列和有界循环断言。
- 建立 `DiagnosisGraphNodeContractTest`,以显式输入白名单、status/verdict 分离、Gatekeeper/verified evidence、retry snapshot 和 Fallback 安全材料为跨节点契约。
- 扩展 `ChatServiceGraphIntegrationTest`,覆盖 public ChatResult、Run/Trace/self-evaluation/Eval、失败生命周期和同 session 多 run 隔离。
- 删除已被显式 Graph Node 契约替代的 `VerifierInputHookTest`,但阶段 4 不删除生产 Hook/ThreadLocal 类型;生产清理仍属于阶段 5。
- 保留 Gatekeeper、Controller、Trace、Repository、Composer、protocol parser、no-evidence、REJECT 和 Eval 的独立安全回归。
- 增加覆盖矩阵/源级检查,证明 Issue 阶段 4 的必需路径均有权威测试且不存在固定 Sequential 顺序断言。
## Capabilities
### New Capabilities
- `chat-diagnosis-stategraph-test-suite`:规定 Diagnosis Graph 的 Workflow、Node Contract、ChatService Integration 三层测试体系、必需路径矩阵和旧实现测试退役边界。
### Modified Capabilities
- `chat-diagnosis-stategraph-design-freeze`:将已冻结的“route/node-contract/Chat integration 替换旧 Sequential/Hook 测试”要求落实为具体权威测试类和保留回归集合。
## Scope
### In Scope
- 测试类重命名/收敛、跨节点契约测试、ChatService integration 分支补齐、覆盖矩阵和测试辅助夹具复用。
- 删除 `VerifierInputHookTest`,避免已退出生产路径的隐式 Gatekeeper/ThreadLocal payload 继续约束新架构。
- 运行新的 Graph suite 与必须保留的 Controller/Trace/Repository/Gatekeeper/Composer/Eval 回归。
### Out of Scope
- 不改变生产 Graph 路由、Node、ChatService、Prompt、数据库或 API 行为;若测试发现生产偏差,按实现期冲突规则单独分类。
- 不删除 `VerifierInputHook`、`VerifierContextHolder` 或其他生产兼容代码;阶段 5 统一清理。
- 不运行 Maven live E2E、检查 `logs/` 或查询真实数据库;仍保留到阶段 5。
- 不把所有细粒度 unit test 强制合并成一个巨型文件;三层权威入口与可复用小测试可以共存。
## Context Constraints
- `ChatServiceSequentialAgentTest` 已在阶段 3 由 Graph integration 替代,阶段 4 不恢复任何固定 Agent 顺序断言。
- Workflow 只验证 Graph 节点/条件边/计数/事件;Node Contract 只验证输入投影、标准输出和安全边界;Chat integration 只从 public service/Run persistence 观察行为。
- 必须复用 `ScriptedDiagnosisGraphActions` 和现有 protocol/node helpers,禁止复制大段 JSON fixture 或另造第二套 Graph factory。
- Gatekeeper ceiling 导致的 LOW_CONFID 不得触发 evidence retry;只有有效 critical evidence gap 且总轮次未耗尽才允许补证据。
- 前置 Fallback 不得泄漏任何 Executor claim;后置 Composer Fallback 只能使用 Verifier 允许材料。
## Acceptance
- `DiagnosisGraphWorkflowTest` 覆盖 PASS、Planner/Verifier/Composer 技术重试与耗尽、Executor failures/no-evidence、Gatekeeper PASS/LOW_CONFID/REJECT、evidence retry、Verifier REJECT 和全部安全终止。
- `DiagnosisGraphNodeContractTest` 覆盖四类 Agent Adapter、Gatekeeper、Verified Input、Evidence Retry、Fallback、config identity、unknown/failure fail-closed 和 verified-only 输入。
- `ChatServiceGraphIntegrationTest` 覆盖 SUCCESS、handled Fallback、unhandled/no-answer FAILED、Run metrics/Eval/trace、clean-up 和同 session 多 run 隔离。
- `VerifierInputHookTest` 不再存在;保留回归集合全部通过,且生产源码在本阶段无行为 diff。
- Maven test compilation、OpenSpec strict、主 specs strict、`git diff --check` 和测试体系源级检查通过。
- 阶段 4明确记录未运行最终 live E2E/log/DB 验收。
## Risks
- 仅重命名测试可能掩盖覆盖缺口;必须建立 Issue 路径到 test method 的显式矩阵并补齐缺失分支。
- 过度合并会降低失败定位;保留专用 unit tests,只让三层类成为架构入口而非唯一文件。
- 删除 Hook test 可能丢失 parser/Gatekeeper 边界;删除前必须证明对应行为已由 protocol parser、Gatekeeper Node/service 和 verified input tests覆盖。
- 测试重构若意外修改生产代码会模糊阶段边界;默认生产源码 diff 必须为零。
@@ -0,0 +1,25 @@
## MODIFIED Requirements
### Requirement: Test migration design SHALL preserve safety behavior
The project SHALL replace Sequential/Hook implementation tests with authoritative `DiagnosisGraphWorkflowTest`, `DiagnosisGraphNodeContractTest`, and `ChatServiceGraphIntegrationTest` layers while retaining focused public/security contract tests. Fixed Agent call order and legacy Hook payload shape SHALL NOT remain correctness criteria.
#### Scenario: Old tests are replaced
- **WHEN** StateGraph tests become authoritative
- **THEN** `ChatServiceSequentialAgentTest` SHALL NOT exist
- **AND** `VerifierInputHookTest` SHALL NOT exist after explicit Gatekeeper/Verified Input coverage is established
- **AND** no replacement test SHALL restore fixed Sequential Agent order as a public requirement
#### Scenario: Safety tests are retained
- **WHEN** the new suite is assembled
- **THEN** Gatekeeper, Controller, Run/Trace, Repository, Composer, parser/projection, no-evidence, REJECT, Eval, and multi-run safety contracts SHALL remain covered
- **AND** Workflow and Node Contract matrices SHALL cover bounded retry, fail-closed routing, verified-only inputs, and deterministic Fallback
#### Scenario: Authoritative test layers are inspected
- **WHEN** a maintainer needs to locate ISS-011 orchestration verification
- **THEN** Workflow tests SHALL own route/loop/event behavior
- **AND** Node Contract tests SHALL own input/output/security projection behavior
- **AND** Chat integration tests SHALL own Run lifecycle and public result behavior
@@ -0,0 +1,121 @@
## ADDED Requirements
### Requirement: Diagnosis Graph tests SHALL have three authoritative layers
The test suite SHALL use Workflow, Node Contract, and ChatService Integration layers as the authoritative ISS-011 verification structure while retaining focused component tests for failure localization.
#### Scenario: Test architecture is inspected
- **WHEN** the Diagnosis Graph test sources are listed
- **THEN** `DiagnosisGraphWorkflowTest`, `DiagnosisGraphNodeContractTest`, and `ChatServiceGraphIntegrationTest` SHALL exist
- **AND** no `ChatServiceSequentialAgentTest` SHALL exist
#### Scenario: Focused unit tests are retained
- **WHEN** a single parser, adapter, Gatekeeper, projection, retry preparer, trace builder, or Fallback contract fails
- **THEN** a focused component test SHALL be able to identify that boundary
- **AND** the suite SHALL NOT require all component assertions to be duplicated in the three authoritative classes
### Requirement: Workflow tests SHALL cover every bounded routing class
`DiagnosisGraphWorkflowTest` SHALL execute the real compiled topology with scripted nodes and SHALL verify path order, retry ownership, bounded termination, and orchestration events without invoking models or tools.
#### Scenario: Normal and Planner paths are tested
- **WHEN** the Workflow suite runs
- **THEN** it SHALL cover PASS, Planner INVALID_OUTPUT/RETRYABLE_FAILED one-time retry success, second technical failure, NON_RETRYABLE_FAILED, and fail-closed unknown status
- **AND** Planner terminal failures SHALL skip Executor and reach Fallback
#### Scenario: Executor and Gatekeeper paths are tested
- **WHEN** the Workflow suite runs
- **THEN** it SHALL cover Executor FAILED, TOOL_BLOCKED, INVALID_OUTPUT, legal no-evidence, Gatekeeper PASS, LOW_CONFID with and without verified binding, REJECT, and unknown result
- **AND** unsafe pre-verification outcomes SHALL skip Verifier
#### Scenario: Verifier evidence retry paths are tested
- **WHEN** the Workflow suite runs
- **THEN** it SHALL cover Verifier one-time technical retry, retry exhaustion, NON_RETRYABLE_FAILED, REJECT, critical evidence retry, no valid gap, non-critical gap, ceiling-driven LOW_CONFID, and second LOW_CONFID termination
- **AND** evidence retry SHALL occur at most once
#### Scenario: Composer paths are tested
- **WHEN** the Workflow suite runs
- **THEN** it SHALL cover Composer one-time technical retry, retry exhaustion, NON_RETRYABLE_FAILED, normal completion, and deterministic post-verification Fallback
- **AND** every terminal path SHALL have events matching the executed node sequence
### Requirement: Node Contract tests SHALL enforce explicit safe state projection
`DiagnosisGraphNodeContractTest` and focused component tests SHALL verify that Nodes receive only allowed state, use the current RunnableConfig identity, return standardized statuses, and never promote unverified material.
#### Scenario: Agent and Gatekeeper config is inspected
- **WHEN** Planner, Executor, Verifier, Composer, or Gatekeeper is invoked
- **THEN** the exact current RunnableConfig SHALL be preserved
- **AND** Gatekeeper SHALL validate the current run exactly once per Executor round
#### Scenario: Executor returns a legal limited result
- **WHEN** tool data is empty or a tool failed but Executor still returns a legal `executor_evidence_v2` with limitations
- **THEN** Executor status SHALL be COMPLETED
- **AND** the workflow SHALL continue to Gatekeeper
#### Scenario: Gatekeeper returns partial or unsafe evidence
- **WHEN** Gatekeeper is REJECT with any partial passed binding, LOW_CONFID with zero passed binding, missing, or unknown
- **THEN** the path SHALL fail closed to pre-verification Fallback
- **AND** no Executor claim from a passed or failed binding SHALL appear in the answer
#### Scenario: Verified-only input is projected
- **WHEN** Gatekeeper PASS or continuable LOW_CONFID reaches Verified Input
- **THEN** Verifier SHALL receive only claims/bindings matched to passed checked bindings and their `matched_text`
- **AND** it SHALL NOT receive unreferenced tool results, raw Executor text, or full tool trace
#### Scenario: Verifier execution fails
- **WHEN** Verifier output is invalid or invocation fails
- **THEN** only `verifier_status` SHALL express the execution failure
- **AND** no model/effective diagnostic verdict SHALL be fabricated
#### Scenario: Evidence retry revalidates the full snapshot
- **WHEN** one critical evidence retry occurs
- **THEN** the second Executor input SHALL contain prior verified material and incremental-query constraints
- **AND** the second complete Executor snapshot SHALL pass through Gatekeeper again without reusing the first verdict
### Requirement: Chat integration tests SHALL verify Run-owned public behavior
`ChatServiceGraphIntegrationTest` SHALL verify the public ChatResult and current DiagnosisRun lifecycle without binding to internal Agent call order.
#### Scenario: Safe answer completes
- **WHEN** Graph returns a Composer or handled Fallback answer
- **THEN** ChatResult SHALL preserve answer/sessionId/runId
- **AND** the current Run SHALL persist SUCCESS, agent_flow, metrics, self-evaluation, non-empty orchestration trace, and Eval invocation
#### Scenario: Unsafe completion fails
- **WHEN** Graph throws an unhandled failure, returns no state, or has a blank final answer
- **THEN** the current Run SHALL persist FAILED when possible
- **AND** Eval SHALL NOT run
- **AND** only a real partial trace SHALL be retained
#### Scenario: Multiple runs share one session
- **WHEN** two complex Chat requests use the same sessionId
- **THEN** each SHALL receive a distinct runId
- **AND** trace/evaluation/metrics SHALL remain owned by their current run
### Requirement: Legacy implementation tests SHALL retire without losing safety regressions
The stage 4 suite SHALL remove tests that make Sequential or Hook payload internals correctness criteria while preserving equivalent public and security contracts.
#### Scenario: Hook implementation test is retired
- **WHEN** explicit Gatekeeper and Verified Input Nodes are authoritative
- **THEN** `VerifierInputHookTest` SHALL NOT exist
- **AND** parser sanitization, Gatekeeper reference fidelity, passed-binding projection, no-evidence, REJECT, and safe Composer behavior SHALL remain covered by independent tests
#### Scenario: Retained regression suite runs
- **WHEN** stage 4 is accepted
- **THEN** Controller, Trace, Repository, Gatekeeper service, Composer/protocol, Eval, Workflow, Node Contract, and Chat integration tests SHALL pass
- **AND** Maven test compilation SHALL pass
### Requirement: Stage 4 SHALL remain a test-only change
The change SHALL reorganize and strengthen automated tests without changing production runtime behavior and SHALL defer final live verification to stage 5.
#### Scenario: Source diff is inspected
- **WHEN** stage 4 implementation completes
- **THEN** no file under `src/main` SHALL be changed by this stage
- **AND** OpenSpec/devflow/test files MAY change
#### Scenario: Stage 4 verification completes
- **WHEN** the test suite and static gates pass
- **THEN** Maven live startup, `logs/` inspection, and `scripts/query_mysql.py` database verification SHALL remain not run
- **AND** acceptance SHALL record them as reserved for stage 5
@@ -0,0 +1,35 @@
## 1. Authoritative Test Layer Renames
- [x] 1.1 Rename `DiagnosisGraphRoutingTest` to `DiagnosisGraphWorkflowTest` with Git history preserved and no assertion loss.
- [x] 1.2 Rename `DiagnosisRealGraphIntegrationTest` to `DiagnosisGraphNodeContractTest` with existing real-node assertions preserved.
- [x] 1.3 Run the two renamed classes alone and prove the rename baseline is green before adding coverage.
## 2. Workflow Coverage Matrix
- [x] 2.1 Audit Workflow methods against every stage-4 Planner, Executor, Gatekeeper, Verifier, evidence-retry, Composer, and termination scenario.
- [x] 2.2 Add explicit ceiling-driven LOW_CONFID and second-LOW_CONFID no-more-retry assertions where current routing coverage is only implicit.
- [x] 2.3 Assert all unsafe pre-verification paths skip Verifier and every terminal path emits only its actual ordered events.
- [x] 2.4 Keep `ScriptedDiagnosisGraphActions` as the only scripted workflow fixture and avoid duplicate Graph topology/factory setup.
## 3. Node Contract Coverage Matrix
- [x] 3.1 Add a legal tool-failure/no-evidence Executor snapshot case proving COMPLETED continues to Gatekeeper.
- [x] 3.2 Strengthen Gatekeeper REJECT with partial passed binding and prove pre-verification Fallback exposes no Executor claim.
- [x] 3.3 Add Verifier invalid/failure contract assertions proving execution status is set without model/effective verdict fabrication.
- [x] 3.4 Preserve current config identity, checked-binding projection, LOW_CONFID ceiling, full-snapshot revalidation, and Composer safe-material assertions.
- [x] 3.5 Run all focused Adapter, Gatekeeper, VerifiedInput, protocol parser, retry preparer, and Fallback tests.
## 4. Chat Integration And Legacy Test Retirement
- [x] 4.1 Confirm `ChatServiceGraphIntegrationTest` covers compatible ChatResult, SUCCESS/Fallback/FAILED, metrics/Eval, self-evaluation/trace, cleanup, and multi-run isolation.
- [x] 4.2 Delete `VerifierInputHookTest` without changing production Hook/ThreadLocal types in stage 4.
- [x] 4.3 Run replacement parser, Gatekeeper service/node, VerifiedInput, no-evidence, REJECT, Composer safety, Controller, Trace, Repository, and Eval regressions.
- [x] 4.4 Add a source/test inventory check proving the three authoritative classes exist and Sequential/Hook implementation tests do not.
## 5. Verification And Handoff
- [x] 5.1 Run the complete new Graph Workflow/Node Contract/Chat Integration suite and record exact test counts.
- [x] 5.2 Run Maven test compilation and all retained public/security contract tests touched by the migration.
- [x] 5.3 Run current change strict validation, all main specs strict validation, `git diff --check`, and prove `src/main` has no stage-4 diff.
- [x] 5.4 Record the Issue requirement-to-test coverage matrix and any intentional overlap/known limits in devflow acceptance evidence.
- [x] 5.5 Record that Maven live E2E, `logs/`, and `scripts/query_mysql.py` verification were not run and remain reserved for stage 5.