Compare commits

..
Author SHA1 Message Date
zhuyongxin a6afbfaa9d chore: add editorconfig 2026-07-07 21:08:51 +08:00
zhuyongxin 0ee27eb523 feat(trace): improve session workbench review 2026-07-07 19:06:18 +08:00
zhuyongxin 3b62a8940c chore(openspec): archive executor evidence output contract 2026-07-07 19:04:24 +08:00
zhuyongxin 04eb50e2b4 feat(agent): add executor evidence output contract 2026-07-07 19:02:02 +08:00
zhuyongxin 7f2e47ca38 docs(mvp): record executor evidence loop design 2026-07-07 18:51:47 +08:00
aruo aa035b828c feat(trace): add diagnosis trace workbench 2026-07-07 01:24:56 +08:00
aruo b3315ead52 fix(agent): harden live diagnosis skill observability 2026-07-07 00:20:34 +08:00
zhuyongxin 64adb998cf chore(rag): add eval knowledge base mirror 2026-07-06 21:48:21 +08:00
zhuyongxin ed7efc58b7 feat(rag): close eval pipeline with live snapshots 2026-07-06 21:39:27 +08:00
zhuyongxin cf3333d607 feat(rag): modularize knowledge retrieval pipeline 2026-07-06 17:06:05 +08:00
zhuyongxin a375daead7 fix: clean up aiops mojibake text 2026-07-06 11:38:01 +08:00
zhuyongxin a5a0e0c6be Merge remote-tracking branch 'origin/refactor/mvp1.0' into refactor/mvp1.0 2026-07-06 10:54:45 +08:00
zhuyongxin 9e8e20b3b5 chore: update agent instructions 2026-07-06 10:47:54 +08:00
aruo 3dfe3dbe53 Update devflow glossary for skills 2026-07-06 10:43:21 +08:00
zhuyongxin 37083fc92a chore: add agent skills 2026-07-06 10:18:10 +08:00
aruo 3e5c6a159c Update sm-flow workflow docs 2026-07-06 09:15:51 +08:00
aruo 6ccfd33ec5 Add diagnosis playbook skills 2026-07-06 08:35:54 +08:00
aruo 88e0a6c944 docs: reorganize MVP interview documentation 2026-07-05 15:29:28 +08:00
aruo b22f2d22c8 docs: archive historical openspec changes 2026-07-05 14:04:32 +08:00
aruo 63b62b28a2 docs: archive aiops lightweight verifier change 2026-07-05 14:01:33 +08:00
aruo d5902a0499 docs: update mvp architecture snapshot 2026-07-05 13:56:51 +08:00
aruo ed267d753d feat: add aiops lightweight verifier 2026-07-05 13:44:30 +08:00
aruo 2658742119 docs: archive rag and aiops query changes 2026-07-05 13:12:47 +08:00
aruo 72a3dbf8c5 feat: add aiops payload query augmentation 2026-07-05 12:56:20 +08:00
aruo 674dd27a48 feat: add rag post-reindex acceptance 2026-07-05 12:29:24 +08:00
aruo c7e2fc2ee2 feat: include breadcrumb in embedding text 2026-07-05 12:15:27 +08:00
aruo 1bfe1a17b4 docs: add rag refactor story 2026-07-05 12:09:32 +08:00
aruo 9dd6823fe7 docs: add rag retrieval quality report 2026-07-05 11:29:16 +08:00
aruo 9376448804 docs: add rag vectorstore interview notes 2026-07-05 11:13:50 +08:00
aruo f2bae0382c fix: align vectorstore live retrieval 2026-07-05 10:53:45 +08:00
aruo 5c71f5fc79 feat: integrate spring ai vectorstore fallback 2026-07-05 10:20:29 +08:00
aruo b9ec07de57 feat: add spring ai retrieval sidecar 2026-07-05 03:22:03 +08:00
aruo 5197712719 feat: add rag evidence postprocess blocks 2026-07-05 03:03:18 +08:00
aruo 4a94c14feb feat: treat l0 retrieval as domain hint 2026-07-05 02:18:40 +08:00
aruo 9a2a44d1b5 test: add rag retrieval baseline 2026-07-05 02:02:27 +08:00
aruo 79feed3314 Merge branch 'emdash/shy-items-fry-f4zze' into refactor/mvp1.0
# Conflicts:
#	mvp/issues/README.md
2026-07-05 01:42:42 +08:00
aruo 98155ae1d8 Merge branch 'aiops-trace-scope' into refactor/mvp1.0 2026-07-05 01:42:23 +08:00
aruo 2609c5a5ab docs: consolidate rag refactor issues 2026-07-05 01:40:08 +08:00
aruo bf5286c8f4 docs: add interview project materials 2026-07-05 01:39:37 +08:00
aruo 26e12a8d6b Archive MVP demo interview runbook 2026-07-05 01:34:04 +08:00
aruo cbef3ddd3c Add MVP demo interview runbook 2026-07-05 01:25:20 +08:00
aruo 69deb15330 Add diagnosis eval baseline diff 2026-07-05 00:59:53 +08:00
aruo 4c7c53b024 Expand diagnosis eval fixtures 2026-07-05 00:27:57 +08:00
aruo ca5c61fabf Add diagnosis eval harness 2026-07-04 23:51:43 +08:00
aruo 23ee05c7c3 feat: add traceable scoped AIOps diagnosis 2026-07-04 22:57:28 +08:00
aruo dc6cd32a67 Harden evidence trace semantics 2026-07-04 22:36:30 +08:00
424 changed files with 29944 additions and 1289 deletions
+117
View File
@@ -0,0 +1,117 @@
---
name: diagnose
description: Disciplined diagnosis loop for hard bugs and performance regressions. Reproduce → minimise → hypothesise → instrument → fix → regression-test. Use when user says "diagnose this" / "debug this", reports a bug, says something is broken/throwing/failing, or describes a performance regression.
---
# Diagnose
A discipline for hard bugs. Skip phases only when explicitly justified.
When exploring the codebase, use the project's domain glossary to get a clear mental model of the relevant modules, and check ADRs in the area you're touching.
## Phase 1 — Build a feedback loop
**This is the skill.** Everything else is mechanical. If you have a fast, deterministic, agent-runnable pass/fail signal for the bug, you will find the cause — bisection, hypothesis-testing, and instrumentation all just consume that signal. If you don't have one, no amount of staring at code will save you.
Spend disproportionate effort here. **Be aggressive. Be creative. Refuse to give up.**
### Ways to construct one — try them in roughly this order
1. **Failing test** at whatever seam reaches the bug — unit, integration, e2e.
2. **Curl / HTTP script** against a running dev server.
3. **CLI invocation** with a fixture input, diffing stdout against a known-good snapshot.
4. **Headless browser script** (Playwright / Puppeteer) — drives the UI, asserts on DOM/console/network.
5. **Replay a captured trace.** Save a real network request / payload / event log to disk; replay it through the code path in isolation.
6. **Throwaway harness.** Spin up a minimal subset of the system (one service, mocked deps) that exercises the bug code path with a single function call.
7. **Property / fuzz loop.** If the bug is "sometimes wrong output", run 1000 random inputs and look for the failure mode.
8. **Bisection harness.** If the bug appeared between two known states (commit, dataset, version), automate "boot at state X, check, repeat" so you can `git bisect run` it.
9. **Differential loop.** Run the same input through old-version vs new-version (or two configs) and diff outputs.
10. **HITL bash script.** Last resort. If a human must click, drive _them_ with `scripts/hitl-loop.template.sh` so the loop is still structured. Captured output feeds back to you.
Build the right feedback loop, and the bug is 90% fixed.
### Iterate on the loop itself
Treat the loop as a product. Once you have _a_ loop, ask:
- Can I make it faster? (Cache setup, skip unrelated init, narrow the test scope.)
- Can I make the signal sharper? (Assert on the specific symptom, not "didn't crash".)
- Can I make it more deterministic? (Pin time, seed RNG, isolate filesystem, freeze network.)
A 30-second flaky loop is barely better than no loop. A 2-second deterministic loop is a debugging superpower.
### Non-deterministic bugs
The goal is not a clean repro but a **higher reproduction rate**. Loop the trigger 100×, parallelise, add stress, narrow timing windows, inject sleeps. A 50%-flake bug is debuggable; 1% is not — keep raising the rate until it's debuggable.
### When you genuinely cannot build a loop
Stop and say so explicitly. List what you tried. Ask the user for: (a) access to whatever environment reproduces it, (b) a captured artifact (HAR file, log dump, core dump, screen recording with timestamps), or (c) permission to add temporary production instrumentation. Do **not** proceed to hypothesise without a loop.
Do not proceed to Phase 2 until you have a loop you believe in.
## Phase 2 — Reproduce
Run the loop. Watch the bug appear.
Confirm:
- [ ] The loop produces the failure mode the **user** described — not a different failure that happens to be nearby. Wrong bug = wrong fix.
- [ ] The failure is reproducible across multiple runs (or, for non-deterministic bugs, reproducible at a high enough rate to debug against).
- [ ] You have captured the exact symptom (error message, wrong output, slow timing) so later phases can verify the fix actually addresses it.
Do not proceed until you reproduce the bug.
## Phase 3 — Hypothesise
Generate **3–5 ranked hypotheses** before testing any of them. Single-hypothesis generation anchors on the first plausible idea.
Each hypothesis must be **falsifiable**: state the prediction it makes.
> Format: "If <X> is the cause, then <changing Y> will make the bug disappear / <changing Z> will make it worse."
If you cannot state the prediction, the hypothesis is a vibe — discard or sharpen it.
**Show the ranked list to the user before testing.** They often have domain knowledge that re-ranks instantly ("we just deployed a change to #3"), or know hypotheses they've already ruled out. Cheap checkpoint, big time saver. Don't block on it — proceed with your ranking if the user is AFK.
## Phase 4 — Instrument
Each probe must map to a specific prediction from Phase 3. **Change one variable at a time.**
Tool preference:
1. **Debugger / REPL inspection** if the env supports it. One breakpoint beats ten logs.
2. **Targeted logs** at the boundaries that distinguish hypotheses.
3. Never "log everything and grep".
**Tag every debug log** with a unique prefix, e.g. `[DEBUG-a4f2]`. Cleanup at the end becomes a single grep. Untagged logs survive; tagged logs die.
**Perf branch.** For performance regressions, logs are usually wrong. Instead: establish a baseline measurement (timing harness, `performance.now()`, profiler, query plan), then bisect. Measure first, fix second.
## Phase 5 — Fix + regression test
Write the regression test **before the fix** — but only if there is a **correct seam** for it.
A correct seam is one where the test exercises the **real bug pattern** as it occurs at the call site. If the only available seam is too shallow (single-caller test when the bug needs multiple callers, unit test that can't replicate the chain that triggered the bug), a regression test there gives false confidence.
**If no correct seam exists, that itself is the finding.** Note it. The codebase architecture is preventing the bug from being locked down. Flag this for the next phase.
If a correct seam exists:
1. Turn the minimised repro into a failing test at that seam.
2. Watch it fail.
3. Apply the fix.
4. Watch it pass.
5. Re-run the Phase 1 feedback loop against the original (un-minimised) scenario.
## Phase 6 — Cleanup + post-mortem
Required before declaring done:
- [ ] Original repro no longer reproduces (re-run the Phase 1 loop)
- [ ] Regression test passes (or absence of seam is documented)
- [ ] All `[DEBUG-...]` instrumentation removed (`grep` the prefix)
- [ ] Throwaway prototypes deleted (or moved to a clearly-marked debug location)
- [ ] The hypothesis that turned out correct is stated in the commit / PR message — so the next debugger learns
**Then ask: what would have prevented this bug?** If the answer involves architectural change (no good test seam, tangled callers, hidden coupling) hand off to the `/improve-codebase-architecture` skill with the specifics. Make the recommendation **after** the fix is in, not before — you have more information now than when you started.
@@ -0,0 +1,47 @@
# ADR Format
ADRs live in `docs/adr/` and use sequential numbering: `0001-slug.md`, `0002-slug.md`, etc.
Create the `docs/adr/` directory lazily — only when the first ADR is needed.
## Template
```md
# {Short title of the decision}
{1-3 sentences: what's the context, what did we decide, and why.}
```
That's it. An ADR can be a single paragraph. The value is in recording *that* a decision was made and *why* — not in filling out sections.
## Optional sections
Only include these when they add genuine value. Most ADRs won't need them.
- **Status** frontmatter (`proposed | accepted | deprecated | superseded by ADR-NNNN`) — useful when decisions are revisited
- **Considered Options** — only when the rejected alternatives are worth remembering
- **Consequences** — only when non-obvious downstream effects need to be called out
## Numbering
Scan `docs/adr/` for the highest existing number and increment by one.
## When to offer an ADR
All three of these must be true:
1. **Hard to reverse** — the cost of changing your mind later is meaningful
2. **Surprising without context** — a future reader will look at the code and wonder "why on earth did they do it this way?"
3. **The result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons
If a decision is easy to reverse, skip it — you'll just reverse it. If it's not surprising, nobody will wonder why. If there was no real alternative, there's nothing to record beyond "we did the obvious thing."
### What qualifies
- **Architectural shape.** "We're using a monorepo." "The write model is event-sourced, the read model is projected into Postgres."
- **Integration patterns between contexts.** "Ordering and Billing communicate via domain events, not synchronous HTTP."
- **Technology choices that carry lock-in.** Database, message bus, auth provider, deployment target. Not every library — just the ones that would take a quarter to swap out.
- **Boundary and scope decisions.** "Customer data is owned by the Customer context; other contexts reference it by ID only." The explicit no-s are as valuable as the yes-s.
- **Deliberate deviations from the obvious path.** "We're using manual SQL instead of an ORM because X." Anything where a reasonable reader would assume the opposite. These stop the next engineer from "fixing" something that was deliberate.
- **Constraints not visible in the code.** "We can't use AWS because of compliance requirements." "Response times must be under 200ms because of the partner API contract."
- **Rejected alternatives when the rejection is non-obvious.** If you considered GraphQL and picked REST for subtle reasons, record it — otherwise someone will suggest GraphQL again in six months.
@@ -0,0 +1,77 @@
# CONTEXT.md Format
## Structure
```md
# {Context Name}
{One or two sentence description of what this context is and why it exists.}
## Language
**Order**:
{A concise description of the term}
_Avoid_: Purchase, transaction
**Invoice**:
A request for payment sent to a customer after delivery.
_Avoid_: Bill, payment request
**Customer**:
A person or organization that places orders.
_Avoid_: Client, buyer, account
## Relationships
- An **Order** produces one or more **Invoices**
- An **Invoice** belongs to exactly one **Customer**
## Example dialogue
> **Dev:** "When a **Customer** places an **Order**, do we create the **Invoice** immediately?"
> **Domain expert:** "No — an **Invoice** is only generated once a **Fulfillment** is confirmed."
## Flagged ambiguities
- "account" was used to mean both **Customer** and **User** — resolved: these are distinct concepts.
```
## Rules
- **Be opinionated.** When multiple words exist for the same concept, pick the best one and list the others as aliases to avoid.
- **Flag conflicts explicitly.** If a term is used ambiguously, call it out in "Flagged ambiguities" with a clear resolution.
- **Keep definitions tight.** One sentence max. Define what it IS, not what it does.
- **Show relationships.** Use bold term names and express cardinality where obvious.
- **Only include terms specific to this project's context.** General programming concepts (timeouts, error types, utility patterns) don't belong even if the project uses them extensively. Before adding a term, ask: is this a concept unique to this context, or a general programming concept? Only the former belongs.
- **Group terms under subheadings** when natural clusters emerge. If all terms belong to a single cohesive area, a flat list is fine.
- **Write an example dialogue.** A conversation between a dev and a domain expert that demonstrates how the terms interact naturally and clarifies boundaries between related concepts.
## Single vs multi-context repos
**Single context (most repos):** One `CONTEXT.md` at the repo root.
**Multiple contexts:** A `CONTEXT-MAP.md` at the repo root lists the contexts, where they live, and how they relate to each other:
```md
# Context Map
## Contexts
- [Ordering](./src/ordering/CONTEXT.md) — receives and tracks customer orders
- [Billing](./src/billing/CONTEXT.md) — generates invoices and processes payments
- [Fulfillment](./src/fulfillment/CONTEXT.md) — manages warehouse picking and shipping
## Relationships
- **Ordering → Fulfillment**: Ordering emits `OrderPlaced` events; Fulfillment consumes them to start picking
- **Fulfillment → Billing**: Fulfillment emits `ShipmentDispatched` events; Billing consumes them to generate invoices
- **Ordering ↔ Billing**: Shared types for `CustomerId` and `Money`
```
The skill infers which structure applies:
- If `CONTEXT-MAP.md` exists, read it to find contexts
- If only a root `CONTEXT.md` exists, single context
- If neither exists, create a root `CONTEXT.md` lazily when the first term is resolved
When multiple contexts exist, infer which one the current topic relates to. If unclear, ask.
+88
View File
@@ -0,0 +1,88 @@
---
name: grill-with-docs
description: Grilling session that challenges your plan against the existing domain model, sharpens terminology, and updates documentation (CONTEXT.md, ADRs) inline as decisions crystallise. Use when user wants to stress-test a plan against their project's language and documented decisions.
---
<what-to-do>
Interview me relentlessly about every aspect of this plan until we reach a shared understanding. Walk down each branch of the design tree, resolving dependencies between decisions one-by-one. For each question, provide your recommended answer.
Ask the questions one at a time, waiting for feedback on each question before continuing.
If a question can be answered by exploring the codebase, explore the codebase instead.
</what-to-do>
<supporting-info>
## Domain awareness
During codebase exploration, also look for existing documentation:
### File structure
Most repos have a single context:
```
/
├── CONTEXT.md
├── docs/
│ └── adr/
│ ├── 0001-event-sourced-orders.md
│ └── 0002-postgres-for-write-model.md
└── src/
```
If a `CONTEXT-MAP.md` exists at the root, the repo has multiple contexts. The map points to where each one lives:
```
/
├── CONTEXT-MAP.md
├── docs/
│ └── adr/ ← system-wide decisions
├── src/
│ ├── ordering/
│ │ ├── CONTEXT.md
│ │ └── docs/adr/ ← context-specific decisions
│ └── billing/
│ ├── CONTEXT.md
│ └── docs/adr/
```
Create files lazily — only when you have something to write. If no `CONTEXT.md` exists, create one when the first term is resolved. If no `docs/adr/` exists, create it when the first ADR is needed.
## During the session
### Challenge against the glossary
When the user uses a term that conflicts with the existing language in `CONTEXT.md`, call it out immediately. "Your glossary defines 'cancellation' as X, but you seem to mean Y — which is it?"
### Sharpen fuzzy language
When the user uses vague or overloaded terms, propose a precise canonical term. "You're saying 'account' — do you mean the Customer or the User? Those are different things."
### Discuss concrete scenarios
When domain relationships are being discussed, stress-test them with specific scenarios. Invent scenarios that probe edge cases and force the user to be precise about the boundaries between concepts.
### Cross-reference with code
When the user states how something works, check whether the code agrees. If you find a contradiction, surface it: "Your code cancels entire Orders, but you just said partial cancellation is possible — which is right?"
### Update CONTEXT.md inline
When a term is resolved, update `CONTEXT.md` right there. Don't batch these up — capture them as they happen. Use the format in [CONTEXT-FORMAT.md](./CONTEXT-FORMAT.md).
`CONTEXT.md` should be totally devoid of implementation details. Do not treat `CONTEXT.md` as a spec, a scratch pad, or a repository for implementation decisions. It is a glossary and nothing else.
### Offer ADRs sparingly
Only offer to create an ADR when all three are true:
1. **Hard to reverse** — the cost of changing your mind later is meaningful
2. **Surprising without context** — a future reader will wonder "why did they do it this way?"
3. **The result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons
If any of the three is missing, skip the ADR. Use the format in [ADR-FORMAT.md](./ADR-FORMAT.md).
</supporting-info>
@@ -1,6 +1,6 @@
---
name: sm-flow
description: OpenSpec-first 的结构化工程开发协议层 harness。编排 OpenSpec 的完整生命周期,通过阶段、门控、人类对齐和长期记忆,约束 agent 以正确的顺序、条件和标准使用 OpenSpec。用户想把粗略想法、issue、PRD 或已有 research 推进为准确 OpenSpec change,并通过 OpenSpec apply 实现、验证、归档时使用。
description: OpenSpec-first 工程流程 harness。仅在用户显式调用 /sm-flow、/sm-flow explore、/sm-flow apply、/sm-flow archive,或明确要求使用 sm-flow 流程时使用;不要根据需求类型自动触发。
---
# SM Flow
@@ -9,6 +9,15 @@ SM Flow 是一个**协议层 harness**——编排 OpenSpec 的完整生命周
sm-flow 会自动维护 `devflow/` 目录作为项目长期记忆。用户不需要手动管理它,sm-flow 会在流程中自动读取和回填。
## 触发规则
只在用户显式调用时使用 sm-flow:
- 用户输入 `/sm-flow`、`/sm-flow explore`、`/sm-flow apply`、`/sm-flow archive`。
- 用户用自然语言明确要求"使用 sm-flow"、"走 sm-flow 流程"或等价表达。
不要根据需求类型自动触发 sm-flow。即使任务涉及 OpenSpec、跨模块、接口契约、需求澄清或 devflow 归档,只要用户没有显式要求 sm-flow,就按普通工程任务处理。
## 四层架构
```
@@ -30,10 +39,10 @@ sm-flow → 编排层(harness):阶段、门控、产物约束、人
1. **OpenSpec 是唯一执行真理源**。apply 阶段必须读取 Committed OpenSpec 文件作为执行依据;对话中的描述不等于产物。Draft OpenSpec 是讨论对象,不是执行许可。
2. **不得跳过 context**。生成 OpenSpec 前,必须先读取相关 devflow 上下文(glossary、ADR、历史项目)。
3. **不得跳过 grill**。即使需求看起来很清楚,至少解决三个高价值澄清或验证问题。
3. **不得跳过 grill**。必须按 `references/scales.md` 的当前分档要求完成澄清或验证。
4. **不得跳过 commit**。进入 apply 前,Draft OpenSpec 必须通过 commit 检查成为 Committed OpenSpec。
5. **冲突必须先分类再处理**。OpenSpec 不准(规格遗漏)→ 修正 OpenSpec;代码偏离(实现偏差)→ 修正代码;不确定或涉及设计方向 → 暂停并等待用户确认。
6. **子 skill 必须显式调用**。每个阶段指定的子 skill 必须显式调用;如果子 skill 不存在,流程失败,不得静默跳过或降级执行。
6. **能力来源必须显式声明**。每个阶段先声明使用外部子 skill / OpenSpec CLI / sm-flow 内置协议;外部能力不可用时可使用 `references/fallbacks.md` 的内置协议,但必须标注为 fallback。若外部能力和内置协议都不可用,流程失败。
每个阶段的过程约束(question pool、one-at-a-time、cross-artifact 对齐、冲突回写等)和质量约束(可观测产出要求)见 `references/phase-contracts.md` 中对应阶段的退出条件和 checkpoint。
@@ -48,11 +57,26 @@ sm-flow → 编排层(harness):阶段、门控、产物约束、人
用户也可以用自然语言指定从某个阶段继续,例如"ops-message-support 的 grill 已经做完了,继续"。harness 识别意图后,自动补做最小前置检查,然后从指定阶段继续。
## 可见 Checkpoint
内部阶段不是用户 API。对用户汇报进度时,默认只暴露 4 个 checkpoint:
| Checkpoint | 覆盖内部阶段 | 用户可见含义 |
|---|---|---|
| Discover | clarify + context + propose + grill | 澄清目标、读取 devflow、形成轻量 proposal、解决关键问题 |
| Commit | specify + audit + commit | 补全 OpenSpec、做架构/产物对齐、生成 Committed OpenSpec |
| Apply | apply | 基于 Committed OpenSpec 实现和验证 |
| Archive | archive | 回填 devflow、汇报验收、询问是否归档 OpenSpec |
除非用户要求看细节,进度汇报、暂停点和恢复提示应使用 checkpoint 名称,而不是逐个暴露 9 个内部阶段。内部阶段仍按顺序执行,并以 `references/phase-contracts.md` 为准。
## 首次加载
执行前只读取当前任务需要的 reference 文件:
- 需要执行阶段时,先读取 `references/phase-contracts.md`;如果当前阶段涉及接口影响分级、分档、启动规则、快速模式或完成标准,再补读 `references/operating-rules.md`。
- 需要执行阶段时,先读取 `references/phase-contracts.md`;如果当前阶段涉及接口影响分级、分档、启动规则、快速模式或完成标准,再补读 `references/operating-rules.md`;如果外部 OpenSpec 能力或子 skill 不可用,再补读 `references/fallbacks.md`。
- 判断或执行 `micro / standard / complex` 分档时,读取 `references/scales.md`;其它文件不得重复定义分档细节。
- 当 checkpoint / gate / fallback / Draft / Committed 等术语含义不清,或需要统一对用户说明时,读取 `references/glossary.md`。
- 创建或更新 PRD、ADR、验收报告、词汇表、复合知识文档时,读取 `references/templates.md`。
- archive 阶段或需要从 OpenSpec 提取产物时,读取 `references/archive-rules.md`。
@@ -2,6 +2,49 @@
archive 阶段的目标是把 OpenSpec 产物、实现结果和过程日志转化为持久、可读、可复用的项目记忆。sm-flow 在 clarify → apply 期间只维护 `decisions.md` 作为过程日志,archive 阶段从中提取完整 devflow 档案。
## Archive 强制执行顺序
Archive 阶段必须按以下顺序执行,不得跳过或重排:
### Step 1: 创建 devflow 档案(必需)
- [ ] 创建 `devflow/projects/YYYY-MM-DD-{slug}/brief.md`
(从 proposal.md 提取:背景、目标、范围、非目标)
- [ ] 按 `references/scales.md` 的当前分档决定是否创建 `devflow/projects/YYYY-MM-DD-{slug}/evidence.md`
(创建时从 decisions.md 提取 evidence-driven 记录)
- [ ] 创建 `devflow/projects/YYYY-MM-DD-{slug}/decisions.md`
(整理为最终版:关键决策、权衡、风险)
- [ ] 创建 `devflow/projects/YYYY-MM-DD-{slug}/acceptance.md`
(记录:静态验证、脚本验证、浏览器/人工验证、未验证)
### Step 2: 更新索引(必需)
- [ ] 在 `devflow/index.md` 末尾追加或更新一行:
`| YYYY-MM-DD | slug | 领域 | 关键词 | openspec/changes/xxx | {status} |`
### Step 3: 标记 OpenSpec(必需)
- [ ] 创建 `openspec/changes/{slug}/.archive-ready` 文件
### Step 4: 向用户汇报(必需)
- [ ] 列出创建的 devflow 档案文件路径(验证文件实际存在于磁盘)
- [ ] 汇报验证情况(按静态验证、脚本验证、浏览器/人工验证、未验证分类)
- [ ] 列出剩余风险或后续事项
- [ ] 询问:**是否现在归档 OpenSpec?**
### Step 5: 用户确认后执行 OpenSpec Archive(可选)
- [ ] 调用 `openspec-archive-change`
- [ ] 记录 archive 结果
**自检**:在执行 Step 4 前,检查 Step 1-3 是否都完成。
---
## 目录规则
项目档案路径:
@@ -10,10 +53,10 @@ archive 阶段的目标是把 OpenSpec 产物、实现结果和过程日志转
devflow/projects/YYYY-MM-DD-{slug}/
```
archive 阶段创建以下文件:
archive 阶段按 `references/scales.md` 的当前分档创建以下文件:
- `brief.md`:从 proposal.md 提取背景、目标、范围、非目标。
- `evidence.md`:从 decisions.md 中的 evidence-driven 记录提取。
- `evidence.md`:从 decisions.md 中的 evidence-driven 记录提取;是否独立创建按 `references/scales.md` 执行。
- `decisions.md`:保持为最终版,整理格式。
- `acceptance.md`:从实现结果和验证结果提取。
@@ -34,11 +77,7 @@ archive 阶段创建以下文件:
## 产物分档
| 分档 | 适用场景 | 必须文件 | 扩展文件 |
| --- | --- | --- | --- |
| `micro` | 小改动、低风险、需求明确 | `brief.md`、`decisions.md`、`acceptance.md` | 证据少时并入 `brief.md` |
| `standard` | 默认模式 | `brief.md`、`evidence.md`、`decisions.md`、`acceptance.md` | 按需 ADR/compound |
| `complex` | 高风险、跨模块、需求不清、多人协作 | standard 全部文件 | 按需 `prd.md`、`research.md`、`design.md`、`tasks.md`、`alignment.md` |
分档的适用场景和必须文件见 `references/scales.md`。本文件只定义 archive 阶段的创建顺序、提取映射和索引规则。
## 提取映射
@@ -0,0 +1,49 @@
# 内置执行协议
本文件只在外部 OpenSpec CLI 或子 skill 不可用时使用。fallback 不是跳过阶段,而是由 sm-flow 用文件方式完成同等最小产物。每次使用 fallback 都必须写入 `decisions.md` 或 `acceptance.md`,说明能力来源、缺失能力、影响和剩余风险。
## 通用规则
- 优先使用外部能力;只有不可用、不可发现或无法在当前环境调用时才使用内置协议。
- 不得因为使用 fallback 跳过 context、grill、commit、apply 授权或 archive 确认。
- fallback 产物仍写入 `openspec/changes/{slug}/` 和 `devflow/projects/YYYY-MM-DD-{slug}/`。
- 如果内置协议也无法满足阶段退出条件,暂停并向用户说明阻塞项。
## grill 内置协议
- 建立 question pool,至少覆盖术语、边界、验收;涉及参考实现或项目基础设施时加入技术实现问题。
- 将问题标记为 `evidence-driven` 或 `user-interview`。
- 先查证 evidence-driven 问题并汇报结论,再逐个询问 user-interview 问题。
- 按 `references/scales.md` 的当前分档满足 grill 要求。
- 将 question pool、证据结论、用户原话和确认状态写入 `decisions.md`;影响实现的结论回写 `proposal.md`。
## openspec 提案内置协议
- 在 `openspec/changes/{slug}/` 创建或更新:
- `proposal.md`:问题、方案、范围、非目标、上下文约束、风险。
- 设计产物:实现设计、接口影响、关键决策、架构风险;形式按 `references/scales.md` 的当前分档要求执行。
- `specs/*/spec.md` 或等价 functional spec:描述用户可观察行为和验收场景。
- `tasks.md`:按可执行切片拆分任务,并给每项写可验证验收标准。
- 运行 cross-artifact 对齐检查:proposal → 设计产物 → specs → tasks。
- 如果发现 gap,先修正 OpenSpec,再进入 commit。
## audit 内置协议
- 用 5 句话以内说明模块链路、数据所有权、跨模块依赖、架构风险和是否需要回写 OpenSpec。
- 如果风险影响实现,修正设计产物或 `tasks.md`。
- 将结论写入 `decisions.md`。
## openspec apply 内置协议
- 只依据 Committed OpenSpec 的 specs/tasks 实现;devflow 只作上下文参考。
- 开始前检查 `.committed` 文件;缺失则返回 commit。
- 如触发 pre-apply checkpoint,先阅读参考实现、grep 项目基础设施模式,并把技术栈清单写入 `decisions.md`。
- 按 tasks 的纵向切片实现、验证并更新任务状态。
- 发现冲突时按三类处理:OpenSpec 不准则修 OpenSpec,代码偏离则修代码,不确定则暂停等用户确认。
## openspec archive 内置协议
- 不删除或移动 OpenSpec change;只标记归档准备状态。
- 完成 devflow 回填、更新 `devflow/index.md`、创建 `.archive-ready`。
- 向用户汇报已创建文件、验证分类、剩余风险,并询问是否需要真实 OpenSpec archive。
- 如果外部 archive 能力仍不可用,在 `acceptance.md` 标记 `accepted-unarchived`。
@@ -0,0 +1,21 @@
# 术语表
本文件统一 sm-flow 协议中的核心词。优先使用这些词,避免同一概念多种说法。
| 术语 | 含义 | 使用边界 |
| --- | --- | --- |
| sm-flow | 协议层 harness | 编排 OpenSpec 生命周期,不替代 OpenSpec |
| OpenSpec | 当前变更的执行真理源 | apply 只能依据 Committed OpenSpec |
| devflow | 长期记忆和上下文层 | 提供术语、历史决策、验收记录,不直接指挥实现 |
| checkpoint | 用户可见检查点 | 默认只暴露 Discover / Commit / Apply / Archive |
| gate | 硬门控 | 不满足就不能进入下一关键动作,如 commit gate |
| Draft OpenSpec | 讨论和审计对象 | propose/specify 期间产生,不能直接 apply |
| Committed OpenSpec | 已通过 commit gate 的 OpenSpec | apply 的唯一执行依据 |
| fallback | 内置执行协议 | 外部 OpenSpec CLI 或子 skill 不可用时使用,必须标注 |
| decisions.md | 过程日志 | clarify 到 apply 期间记录问题、证据、决策、冲突和回写 |
| .committed | commit gate 标记文件 | 存在才可进入合规 apply |
| .archive-ready | archive 准备标记文件 | 表示 devflow 已回填,等待用户确认是否 archive |
| Discover | 用户可见 checkpoint | 覆盖 clarify + context + propose + grill |
| Commit | 用户可见 checkpoint | 覆盖 specify + audit + commit |
| Apply | 用户可见 checkpoint | 覆盖 apply |
| Archive | 用户可见 checkpoint | 覆盖 archive |
@@ -27,13 +27,14 @@
- `/sm-flow apply [change]`:只执行,检查 commit gate → apply。
- `/sm-flow explore`:带上下文的探索模式,不走标准阶段链。
- `/sm-flow archive [change]`:收尾,回填 devflow + 归档确认。
- 明确要求"使用 sm-flow"或"走 sm-flow 流程":按显式调用处理。
- 自然语言指定阶段继续:识别意图后,自动补做最小前置检查,然后从指定阶段继续。
2. 判断启动模式:
- 完整模式:用户提供粗略想法或初始 PRD。
- Research 模式:用户已有 research,需要转成或修正 OpenSpec。
- PRD 文件模式:用户提供已有 PRD 路径。
- 恢复模式:用户希望从某个阶段继续(补做最小前置检查)。
- 快速模式:小改动,合并 gate(见下文)。
- 快速模式:小改动,合并 gate;具体分档规则见 `references/scales.md`。
3. 如果缺少 `devflow/`,初始化:
- `devflow/projects/`
- `devflow/glossary/CONTEXT.md`
@@ -42,7 +43,25 @@
5. 检查 OpenSpec 和子 skill 是否可用:
- OpenSpec 能力:`openspec-propose`、`openspec-apply-change`、`openspec-archive-change`。
- 辅助能力:`to-prd`、`grill-with-docs`、`diagnose`、`tdd`、`zoom-out`。
6. 如果 OpenSpec 不可用,不要直接绕过;使用内置执行协议(见 `references/fallbacks.md`),并在 apply 前向用户说明。
6. 如果 OpenSpec 或子 skill 不可用,不要静默跳过;使用内置执行协议(见 `references/fallbacks.md`),并在当前 checkpoint 说明 fallback 来源、影响和剩余风险。
## 进度汇报
用户可见进度默认折叠为 4 个 checkpoint:
| Checkpoint | 内部阶段 |
| --- | --- |
| Discover | clarify + context + propose + grill |
| Commit | specify + audit + commit |
| Apply | apply |
| Archive | archive |
汇报规则:
- 面向用户时优先使用 checkpoint 名称,不逐个汇报 9 个内部阶段。
- 内部阶段只在 checkpoint 摘要中作为证据列出,例如"Discover 已完成:读取了 devflow、生成 proposal、解决 2 个问题"。
- 只有发生阻塞、冲突、fallback、用户要求继续某个内部阶段,或需要解释恢复位置时,才暴露内部阶段名。
- 当前分档的汇报压缩规则见 `references/scales.md`;无论分档如何,都不要把内部阶段名当作用户操作入口。
## 项目标识规则
@@ -63,7 +82,7 @@ Devflow 是 sm-flow 自动维护的项目长期记忆层,不复制 OpenSpec
**最终档案**(archive 阶段从 decisions.md + OpenSpec 产物提取):
- `brief.md`:背景、目标、范围、非目标、分档、关联 OpenSpec change。
- `evidence.md`:代码/文档证据、历史决策、evidence-driven 结论和汇报状态。
- `evidence.md`:代码/文档证据、历史决策、evidence-driven 结论和汇报状态;分档要求见 `references/scales.md` 和 `references/archive-rules.md`。
- `acceptance.md`:实现结果、验证命令、未验证项、归档状态、后续事项。
**按需产物**(archive 阶段按需创建):
@@ -75,28 +94,17 @@ Devflow 是 sm-flow 自动维护的项目长期记忆层,不复制 OpenSpec
- `alignment.md` / `clarifications.md`:仅在 gap 或澄清很多时使用。
- `adr/*.md` 和 `compound/*.md`:仅在满足 ADR / compound knowledge 规则时使用。
**规模分档**:
- `micro`:小且低风险,gate 合并(见快速模式),最终档案同 standard。
- `standard`:默认模式。
- `complex`:高风险、跨模块、需求不清或多人协作时,在 standard 基础上按需增加扩展产物。
**规模分档**:`micro / standard / complex` 的唯一规则源是 `references/scales.md`。
## 快速模式
快速模式适用于小而低风险的变更。它合并 gate 而不仅仅是压缩产物:
```
standard 流程:clarify → context → propose checkpoint → grill → specify → audit checkpoint → commit
micro 流程:clarify+context 合并 checkpoint → propose+specify 合并 checkpoint → grill(最少 1 个问题) → commit(简化检查)
```
micro 的定位:**gate 变少但保留最关键的**(grill 最小澄清 + commit gate)。
快速模式适用于 `references/scales.md` 定义的 micro 变更。它合并 gate 而不仅仅是压缩产物;具体覆盖规则见 `references/scales.md`。
无论什么模式,以下内容必须保留:
- context 最小上下文收集:至少检查 glossary 和相关 ADR。
- grill 最小澄清:至少一个术语问题、一个边界问题、一个验收问题;evidence-driven 结论仍需汇报。
- commit gate:确认没有未解决用户问题、接口影响已记录、OpenSpec tasks/specs 可执行。
- grill 最小澄清:按 `references/scales.md` 当前分档要求执行;evidence-driven 结论仍需汇报。
- commit gate:确认没有未解决用户问题、接口影响已记录、OpenSpec tasks/specs 可执行;完整性检查按 `references/scales.md` 当前分档要求执行。
- apply 仍由 OpenSpec tasks/specs 驱动执行。
- archive 轻量回填:记录验收结果、OpenSpec 链接和归档状态。
@@ -104,8 +112,9 @@ micro 的定位:**gate 变少但保留最关键的**(grill 最小澄清 + co
只有同时满足以下条件,流程才算完成:
- OpenSpec proposal/design/specs/tasks 已生成或更新到可执行状态。
- 用户可见的 Discover、Commit、Apply、Archive checkpoint 已完成,或未完成项已明确标记为暂停/不适用。
- OpenSpec proposal、设计产物、specs、tasks 已按当前分档生成或更新到可执行状态。
- 实现或规划工作已完成,且执行依据来自 OpenSpec。
- 已运行验证,或已记录未运行验证的原因。
- `devflow/projects/YYYY-MM-DD-{slug}/` 包含 brief.md、evidence.md、decisions.md、acceptance.md。
- `devflow/projects/YYYY-MM-DD-{slug}/` 包含 `references/scales.md` 和 `references/archive-rules.md` 要求的当前分档档案。
- 用户知道剩余风险与下一步,并已被询问是否归档 OpenSpec change。
@@ -4,6 +4,18 @@
执行顺序:clarify → context → propose → grill → specify → audit → commit → apply → archive。
## 目录
- clarify — 入口澄清
- context — 上下文收集
- propose — 轻量 propose
- grill — 人类对齐澄清
- specify — 细化 + 对齐
- audit — 架构审计
- commit — Commit OpenSpec
- apply — OpenSpec 执行
- archive — 回填 + 归档
## clarify — 入口澄清
**进入条件**:用户提供粗略想法、初始 PRD、已有 research、issue,或要求启动 SM Flow。
@@ -13,7 +25,7 @@
- 如果用户已有 research,先识别它是否已经包含用户价值、技术方案、验收标准和任务拆分。
- 如果输入过于模糊,最多追加三轮聚焦问题。
- 当答案会改变 OpenSpec proposal/specs/tasks 时,优先一次只问一个问题。
- 如果需要判断 `micro / standard / complex` 分档,补读 `references/operating-rules.md`。
- 如果需要判断 `micro / standard / complex` 分档,补读 `references/scales.md`。
**退出条件**:
- 问题可以用 1-2 句话说清楚。
@@ -36,7 +48,7 @@
- 读取 `devflow/glossary/CONTEXT.md`,提取相关术语和业务规则。
- 搜索 `devflow/projects/` 中相关 PRD、design、tasks、acceptance 和 ADR。
- 搜索 `devflow/compound/` 中可复用 learning、trick、decision、explore。
- 记录哪些上下文会影响 OpenSpec proposal/design/specs/tasks。
- 记录哪些上下文会影响 OpenSpec proposal、设计产物、specs 或 tasks。
- 如果发现旧根目录 `CONTEXT.md` 与 `devflow/glossary/CONTEXT.md` 冲突,暂停并向用户汇报。
**退出条件**:
@@ -70,18 +82,22 @@
**Human checkpoint**:
- 向用户简要说明 proposal 范围、关键假设、主要风险、devflow 上下文如何影响方案。
- 询问是否继续进入 grill 澄清阶段;用户明确要求"全自动执行"时可跳过等待。
- 作为 Discover checkpoint 的中间状态汇报;询问是否继续完成 Discover 的人类澄清部分。用户明确要求"全自动执行"时可跳过等待。
## grill — 人类对齐澄清
**进入条件**:propose 已有轻量 proposal.md。
**显式子 skill**:`grill-with-docs`。进入本阶段必须调用 `.agents/skills/grill-with-docs/SKILL.md`。
**能力来源**:优先使用 `grill-with-docs`;不可用时使用 `references/fallbacks.md#grill-内置协议`,并在 `decisions.md` 标注 fallback。
**动作**:
- 优先使用 `grill-with-docs`。
- 进入 grill 时先建立一个 question pool,并记录到 `decisions.md`:
- 默认至少覆盖术语、边界、验收三个维度。
- **技术实现维度**(新增):当 proposal 提到参考实现、或涉及项目现有基础设施时,增加技术澄清问题:
- 参考实现的具体文件路径是什么?
- 项目现有的 [请求结构/MQ/缓存/加密/工具类] 标准是什么?
- 有哪些技术点需要先调研或新建?
- 如果变更涉及多模块、接口、权限、下游消费者、响应结构或生命周期规则,先把这些维度补进问题池。
- 逐项标记每个问题的模式:
- `evidence-driven`:问题能通过代码、文档、测试、OpenSpec 或既有 ADR 证明;代理先查证,再向用户汇报证据、结论和是否需要确认。
@@ -97,7 +113,7 @@
**退出条件**:
- question pool 已建立并覆盖当前 change 所需维度。
- 至少解决三个高价值澄清或验证问题,并记录每个问题属于 `evidence-driven` 还是 `user-interview`。
- 已满足 `references/scales.md` 中当前分档的 grill 要求。每个问题都必须记录属于 `evidence-driven` 还是 `user-interview`。
- 所有 evidence-driven 结论已向用户汇报。
- 所有 user-interview 决策已获得用户确认。
- 没有未解决或代理代确认的 user-interview 问题。
@@ -113,20 +129,20 @@
**Human checkpoint**:
- 汇报已解决和未解决的问题、proposal 变更、术语和 ADR 更新。
- 询问是否继续进入 specify 细化阶段。
- 汇报 Discover checkpoint 完成情况,并询问是否继续进入 Commit checkpoint。
## specify — 细化 + 对齐
**进入条件**:grill 已退出,需求已通过澄清稳定下来。
**显式子 skill**:`openspec-propose`(基于已稳定的 proposal 补全完整 OpenSpec);`to-prd`(按需生成 PRD)。进入本阶段必须先声明调用方式。
**能力来源**:优先使用 `openspec-propose`(基于已稳定的 proposal 补全完整 OpenSpec);按需使用 `to-prd`。进入本阶段必须先声明调用方式;外部能力不可用时使用 `references/fallbacks.md#openspec-提案-内置协议`,并在 `decisions.md` 标注 fallback。
**动作**:
- 基于已稳定的 proposal.md 补全 design.md、specs/、tasks.md:
- 优先调用 `openspec-propose`,输入中明确说明"proposal.md 已存在,本次只需补全 design/specs/tasks"。
- 如果不可用,执行 `references/fallbacks.md#openspec-提案-降级`。
- 基于已稳定的 proposal.md 补全设计产物、specs/、tasks.md:
- 优先调用 `openspec-propose`,输入中明确说明"proposal.md 已存在,本次只需按当前分档补全设计产物/specs/tasks"。
- 如果不可用,执行 `references/fallbacks.md#openspec-提案-内置协议`。
- 如果没有结构化 PRD,按需按 `to-prd` 协议生成 `brief.md`;复杂需求、对外协作或用户明确要求时再生成 `prd.md`。
- `micro` 模式默认不创建独立 PRD,除非用户要求或需求复杂度升级。
- 独立 PRD 是否需要按 `references/scales.md` 的当前分档和用户要求判断。
- 用 grill 阶段的 decisions.md 记录增强 OpenSpec 产物:确保 design/specs/tasks 反映所有已确认的决策。
- **显式 cross-artifact 对齐检查**——在 checkpoint 中输出对齐检查表:
- `brief/prd` 中的目标、范围、非目标和验收预期 → `proposal` 是否覆盖。
@@ -143,14 +159,14 @@
- 如果发现不一致,优先修正 OpenSpec,而不是只修改 devflow 文档。
**退出条件**:
- `design.md`、`specs/`、`tasks.md` 存在且与 proposal 对齐。
- OpenSpec 细化产物存在且与 proposal 对齐;产物形态按 `references/scales.md` 的当前分档要求执行。
- `brief.md` 已覆盖背景、目标、范围和非目标;复杂需求存在独立 `prd.md` 或用户明确不需要 PRD。
- cross-artifact 对齐检查表已生成(4 行,每行标记已对齐/存在 gap),没有未处理 gap。
- 涉及接口变更时,已记录接口影响等级和产物要求;不确定项已标记。
- 所有已知冲突已修正或等待用户决策。
**输出**:
- 完整的 Draft OpenSpec:proposal.md + design.md + specs/ + tasks.md。
- Draft OpenSpec:按 `references/scales.md` 的当前分档要求生成 proposal、设计、specs 和 tasks。
- `brief.md`,以及按需创建的 `prd.md`。
- cross-artifact 对齐检查表(写入 checkpoint 或 decisions.md)。
- 必要的 OpenSpec 修正。
@@ -159,7 +175,7 @@
**进入条件**:specify 已退出,完整 OpenSpec 产物已存在。
**显式子 skill**:`zoom-out`。进入本阶段必须调用 `.agents/skills/zoom-out/SKILL.md`。
**能力来源**:优先使用 `zoom-out`;不可用时使用 `references/fallbacks.md#audit-内置协议`,并在 `decisions.md` 标注 fallback。
**动作**:
- 画出输入 → 处理 → 输出的模块链路。
@@ -171,20 +187,20 @@
**退出条件**:
- 架构风险已被接受,或流程返回 grill/specify 修正 OpenSpec。
- OpenSpec design/tasks 已反映会影响实现的架构审计结论。
- OpenSpec 设计产物/tasks 已反映会影响实现的架构审计结论。
**输出**:
- 架构审计记录,写入 `decisions.md`;复杂架构审计可拆出 `design.md`。
- 必要的 OpenSpec design/tasks 修正。
- 必要的 OpenSpec 设计产物/tasks 修正。
**Human checkpoint**:
- 用不超过五句话向用户说明架构风险、OpenSpec 修正点和实现计划。
- 询问是否进入 commit。
- 作为 Commit checkpoint 的中间状态汇报;询问是否继续完成 commit gate。
## commit — Commit OpenSpec
**进入条件**:
- grill 已解决术语、边界、验收三个维度的高价值问题。
- grill 已满足 `references/scales.md` 中当前分档要求。
- 所有 `user-interview` 问题都已获得用户显式确认。
- audit 已经完成,或快速模式下已记录跳过原因;快速模式定义见 `references/operating-rules.md#快速模式`。
- Draft OpenSpec 已回写所有会影响实现的澄清、接口影响和架构审计结论。
@@ -194,7 +210,7 @@
- 检查 design 是否记录上下文约束、关键技术决策、架构风险和接口影响。
- 检查 specs 是否表达外部可观察行为,并覆盖验收口径。
- 检查 tasks 是否是可执行的纵向切片,而不是泛泛描述。
- 复核 cross-artifact 对齐:`brief/prd → proposal → design → specs → tasks` 是否闭环,没有把字段、范围项、验收行为或实现切片丢在上游产物里。
- 复核 cross-artifact 对齐:`brief/prd → proposal → 设计产物 → specs → tasks` 是否闭环,没有把字段、范围项、验收行为或实现切片丢在上游产物里。
- 检查 `decisions.md` 中所有影响实现的发现,是否已回写到 proposal、design、specs 或 tasks。
- 接口影响分级定义见 `references/operating-rules.md#接口影响分级`。
- 检查接口影响是否已按 L1/L4 判级;L3/L4 是否有独立接口文档或等价独立章节。
@@ -203,7 +219,16 @@
**退出条件**:
- Draft OpenSpec 已达到可执行状态,并记录为 Committed OpenSpec。
- apply 所需的 proposal、design、specs 和 tasks 均存在且一致;commit checkpoint 必须验证文件实际存在于磁盘,如果任一文件不存在,commit 失败,返回 specify 补写。
- **文件完整性检查**(按 `references/scales.md` 的当前分档要求执行):
- [ ] proposal 存在,且足以说明问题、建议方案、范围和非目标。
- [ ] 设计产物存在,形式符合当前分档要求。
- [ ] specs 存在,且表达用户可观察行为。
- [ ] tasks 存在,且任务可执行、验收标准可验证。
- **一致性检查**(必须通过):
- [ ] proposal 中的核心概念在设计产物中有对应设计
- [ ] 设计产物中的关键决策在 tasks 中有对应实现任务
- [ ] tasks 的验收标准可验证(不是"正确实现""完成功能"这类模糊描述)
- **标记文件**:检查通过后,创建 `openspec/changes/{slug}/.committed` 文件标记为 Committed OpenSpec
- 所有 preflight 风险已消除或明确记录为已接受。
**输出**:
@@ -212,36 +237,83 @@
**Human checkpoint**:
- 用不超过五句话说明 Committed OpenSpec 的范围、接口影响、剩余风险和执行计划。
- 询问是否进入 apply;除非用户在启动时明确要求"全自动执行",必须等待用户明确说出进入 apply、开始实现、执行修改或等价授权。
- 汇报 Commit checkpoint 完成情况,并询问是否进入 Apply checkpoint;除非用户在启动时明确要求"全自动执行",必须等待用户明确说出进入 apply、开始实现、执行修改或等价授权。
- 不得把 grill 的单个决策确认当作本 checkpoint 的授权。
## apply — OpenSpec 执行
**进入条件**:
- `openspec/changes/{slug}/` 中 proposal/design/specs/tasks 已通过 commit,成为 Committed OpenSpec。
- `openspec/changes/{slug}/` 中 proposal、设计产物、specs、tasks 已通过 commit,成为 Committed OpenSpec。
- **前置门控检查**(硬约束):
- 检查 `openspec/changes/{slug}/.committed` 文件是否存在
- 如不存在,执行以下流程:
1. 汇报:Draft OpenSpec 未通过 commit 检查
2. 列出缺失的 checkpoint 项(文件完整性、一致性检查)
3. 询问用户:是否补做 commit 检查;如用户要求不补做,则中止 apply 或标记为 `emergency-bypass`,且本次流程不得视为合规 sm-flow apply
- commit 后已获得用户明确的 apply 授权,除非用户在启动时要求"全自动执行"。
- devflow 与 OpenSpec 没有未解决冲突。
- 没有未解决的 user-interview 问题、未判级接口影响、未汇报 evidence-driven 结论或未接受架构风险。
**显式子 skill**:`openspec-apply-change`;遇到 bug/不确定行为时显式调用 `diagnose`;需要测试驱动时显式调用 `tdd`。进入本阶段必须调用指定子 skill,不得静默跳过。
**能力来源**:优先使用 `openspec-apply-change`;不可用时使用 `references/fallbacks.md#openspec-apply-内置协议`,并在 `decisions.md` 标注 fallback。遇到 bug/不确定行为时优先使用 `diagnose`;需要测试驱动时优先使用 `tdd`。不可用时执行对应最小协议并记录原因,不得静默跳过。
**动作**:
### Pre-apply Checkpoint
**触发条件**:当 OpenSpec 涉及以下任一情况时必须执行
- design 或 tasks 中提到"参考 XXX 实现"
- 需要调用项目现有基础设施(MQ/统一请求结构/工具类等)
- 技术栈不熟悉或第一次在该项目实现类似功能
**执行步骤**:
1. **阅读所有参考实现**
- 从 OpenSpec design 或 tasks 中定位参考实现文件
- 如果路径不明确,通过 Grep 搜索关键类名或模式
- 理解关键逻辑,提取可复用代码片段和模式
2. **Grep 关键技术栈**
- 请求/响应结构模式(如 `RequestMsg`、`ResponseMsg`、DTO 规范)
- 消息队列模式(如 `@KafkaListener`、`@YkMsg`、发送模板)
- 统一工具类(如 `XxxUtil`、`XxxHelper`、加密/验签工具)
- 异常处理和日志记录标准
3. **形成技术栈清单并写入 decisions.md**
- 项目使用的请求/响应结构标准
- MQ 消息定义和发送标准
- Consumer 标准位置和写法
- 加密/验签/工具类的标准用法
- 识别需要新建的工具类或基础设施
**输出要求**:
- 技术栈清单已写入 `decisions.md` 的 "Pre-apply Research" 章节。
- 已列出所有参考实现的文件路径。
- 已识别需要新建的工具类/基础设施。
**按风险执行**:执行深度按 `references/scales.md` 的当前分档和实现风险决定;退出判断以清单是否足以指导实现为准。
### 实现过程
- 优先调用 `openspec-apply-change`。
- 执行依据是 OpenSpec specs/tasks;devflow 只能作为上下文参考。
- 按 OpenSpec tasks 的纵向切片实现。
- **分步实现**:建议按 Controller → Service → MQ/异步组件 → Consumer/下游 顺序,每完成一层验证后再继续。
- 进入实现前先汇报本阶段的 capability 来源、当前 task 进度和本轮要推进的切片;否则 apply 不算真正开始。
- **首模块完成后对齐检查**:完成第一个接口/模块后,对比 OpenSpec design/tasks,标记"已完成/TODO";核心功能(加密/验签/核心业务逻辑)不允许空实现或纯 TODO 注释。
- 当用户质疑、用户要求修改、代码检查、测试失败或运行行为与 OpenSpec 冲突时,做三类判断:
- OpenSpec 不准(规格遗漏、边界未覆盖、验收口径缺失)→ 暂停 apply,修正 OpenSpec 后重新提交。
- 代码偏离(实现没按 OpenSpec 做)→ 修正代码,不改 OpenSpec。
- 不确定根因、涉及设计方向、用户改变目标或范围 → 暂停并等待用户确认。
- 判断结果、证据、用户确认和 OpenSpec 回写状态必须记录到 `decisions.md`。
- **快速失败**:连续返工 ≥ 2 次时,暂停并重新执行 pre-apply checkpoint 或向用户汇报。
- 当用户要求、行为复杂或回归风险高时使用 TDD。
- 当测试失败、行为意外或原因不确定时使用 diagnose。
- 如果 diagnose 发现根因是 OpenSpec 不准确,先修正 OpenSpec,再继续 apply。
- 修改文件前遵守仓库指令,例如 `AGENTS.md`。
**退出条件**:
- 已完成 pre-apply checkpoint(如触发条件满足),技术栈清单已写入 `decisions.md`。
- OpenSpec tasks 已完成,或剩余 tasks 已明确记录。
- 核心功能已实现或明确标注"待联调",无纯 TODO 占位。
- 所有实现期冲突已分类并处理;没有未确认的规格遗漏、设计冲突或用户变更。
- 已运行验证,或记录了未验证原因。
- 已列出已知限制。
@@ -255,13 +327,13 @@
**进入条件**:实现或规划工作已经达到可交接状态。
**显式子 skill**:`openspec-archive-change` 在用户确认 archive 后调用;archive 回填由 `sm-flow` 执行。必须调用子 skill,不得静默跳过。
**能力来源**:`openspec-archive-change` 在用户确认 archive 后优先调用;不可用时使用 `references/fallbacks.md#openspec-archive-内置协议`,并在 `acceptance.md` 标注 fallback。archive 回填由 `sm-flow` 执行。
**动作**:
- 遵循 `references/archive-rules.md`。
- 从 `decisions.md`(过程日志)+ OpenSpec 产物提炼完整 devflow 档案:
- `brief.md`:从 proposal.md 提取背景、目标、范围、非目标。
- `evidence.md`:从 decisions.md 中的 evidence-driven 记录提取。
- `evidence.md`:按 `references/scales.md` 和 `references/archive-rules.md` 的当前分档要求处理。
- `decisions.md`:保持为最终版,整理格式。
- `acceptance.md`:从实现结果和验证结果提取。
- 只在复杂场景按需拆出 PRD/research/design/tasks/alignment。
@@ -271,7 +343,7 @@
- 询问用户是否要 archive OpenSpec change;不要默认执行归档。
**退出条件**:
- `devflow/projects/YYYY-MM-DD-{slug}/` 包含 brief.md、evidence.md、decisions.md、acceptance.md;archive checkpoint 必须列出所有已创建的文件路径,验证文件实际存在于磁盘。
- `devflow/projects/YYYY-MM-DD-{slug}/` 包含 `references/scales.md` 和 `references/archive-rules.md` 要求的当前分档档案;archive checkpoint 必须列出所有已创建的文件路径,验证文件实际存在于磁盘。
- `devflow/index.md` 已包含或更新本项目条目。
- 用户已被询问是否 archive OpenSpec change。
@@ -0,0 +1,42 @@
# 分档规则
本文件是 `micro / standard / complex` 的唯一规则源。其它文件只引用本文件,不重复定义分档细节。
## standard 基准
standard 是默认分档,适用于普通功能、明确但有一定实现范围的变更。
- 用户可见 checkpoint:Discover → Commit → Apply → Archive。
- OpenSpec 产物:`proposal.md`、独立 `design.md`、`specs/`、`tasks.md`。
- grill:解决术语、边界、验收三个维度的高价值问题。
- commit gate:检查 proposal、design、specs、tasks 的完整性和一致性。
- devflow 档案:`brief.md`、`evidence.md`、`decisions.md`、`acceptance.md`。
## micro 覆盖
micro 适用于小改动、低风险、需求明确的变更。micro 是 standard 的减法,不是跳过流程。
- checkpoint 可合并:Discover + Commit 可在无阻塞时合并汇报。
- micro 内部流程压缩为:clarify+context 合并 checkpoint → 轻量 propose → grill → specify+commit 合并 checkpoint。
- context 保留最小收集:至少检查 glossary 和相关 ADR。
- grill 保留最小澄清:至少解决一个高价值问题,并记录术语、边界、验收三类是否明确;不明确项必须补问或标记风险。
- OpenSpec 仍需要 `proposal.md`、`specs/`、`tasks.md`。
- `design.md` 可不独立创建;允许在 `proposal.md` 或 `tasks.md` 中写等价设计小节。
- `specs/` 和 `tasks.md` 可轻量,但必须表达可观察行为和可执行任务。
- commit gate 仍必须通过,并创建 `.committed`。
- devflow 档案至少包含 `brief.md`、`decisions.md`、`acceptance.md`;证据少时可并入 `brief.md` 或 `decisions.md`。
- apply 仍只能依据 Committed OpenSpec。
- archive 仍要轻量回填 devflow,并询问是否归档 OpenSpec。
micro 不适用于接口影响不清、跨团队消费者、迁移/回滚、复杂状态机、长期架构决策或需求边界不清的变更;遇到这些情况应升级为 standard 或 complex。
## complex 增量
complex 适用于高风险、跨模块、需求不清、多人协作或长期架构影响明显的变更。complex 是 standard 的加法。
- 需要更完整的 Discover:增加需求澄清、证据查证、范围确认和风险接受。
- checkpoint 内可补充关键内部阶段结果,但不要把内部阶段名当作用户操作入口。
- 按需创建 `prd.md`、`research.md`、`alignment.md`、接口文档、ADR 或 compound knowledge。
- 接口影响、迁移、灰度、回滚、兼容性和消费者边界必须显式记录。
- audit 需要覆盖模块链路、数据所有权、生命周期、耦合风险和 ADR 冲突。
- archive 在 standard 档案基础上按需提炼长期 design、research、tasks、ADR 和 compound knowledge。
@@ -124,7 +124,7 @@
- 触发来源:用户质疑 / 用户变更 / 代码发现 / 测试失败 / 运行行为
- 冲突对象:proposal / design / specs / tasks / ADR / 代码行为
- 分类:实现偏差 / 规格遗漏 / 设计冲突 / 用户变更
- 分类:OpenSpec 不准 / 代码偏离 / 不确定
## 证据
@@ -136,7 +136,7 @@
- 决策:
- 是否需要用户确认:是 / 否
- OpenSpec 回写:不需要 / 已回写 / 待回写
- OpenSpec 回写:不需要 / 已回写 / 待回写 / 等待用户确认
- 代码处理:
- 验证方式:
```
@@ -353,8 +353,8 @@ specify 阶段的 checkpoint 必须包含此检查表。每项标记"已对齐"
| 上游 → 下游 | 检查内容 | 状态 |
|---|---|---|
| brief/prd → proposal | 目标、范围、非目标、验收预期是否进入 proposal | 已对齐 / 存在 gap |
| proposal → design | 范围、约束、关键承诺是否进入 design | 已对齐 / 存在 gap |
| design → specs/tasks | 影响实现的约束、接口影响、架构结论是否进入 specs 或 tasks | 已对齐 / 存在 gap |
| proposal → 设计产物 | 范围、约束、关键承诺是否进入 design.md 或等价设计小节 | 已对齐 / 存在 gap |
| 设计产物 → specs/tasks | 影响实现的约束、接口影响、架构结论是否进入 specs 或 tasks | 已对齐 / 存在 gap |
| specs → tasks | 可观察行为是否被 tasks 覆盖为可执行切片 | 已对齐 / 存在 gap |
### Gap 详情(如有)
+109
View File
@@ -0,0 +1,109 @@
---
name: tdd
description: Test-driven development with red-green-refactor loop. Use when user wants to build features or fix bugs using TDD, mentions "red-green-refactor", wants integration tests, or asks for test-first development.
---
# Test-Driven Development
## Philosophy
**Core principle**: Tests should verify behavior through public interfaces, not implementation details. Code can change entirely; tests shouldn't.
**Good tests** are integration-style: they exercise real code paths through public APIs. They describe _what_ the system does, not _how_ it does it. A good test reads like a specification - "user can checkout with valid cart" tells you exactly what capability exists. These tests survive refactors because they don't care about internal structure.
**Bad tests** are coupled to implementation. They mock internal collaborators, test private methods, or verify through external means (like querying a database directly instead of using the interface). The warning sign: your test breaks when you refactor, but behavior hasn't changed. If you rename an internal function and tests fail, those tests were testing implementation, not behavior.
See [tests.md](tests.md) for examples and [mocking.md](mocking.md) for mocking guidelines.
## Anti-Pattern: Horizontal Slices
**DO NOT write all tests first, then all implementation.** This is "horizontal slicing" - treating RED as "write all tests" and GREEN as "write all code."
This produces **crap tests**:
- Tests written in bulk test _imagined_ behavior, not _actual_ behavior
- You end up testing the _shape_ of things (data structures, function signatures) rather than user-facing behavior
- Tests become insensitive to real changes - they pass when behavior breaks, fail when behavior is fine
- You outrun your headlights, committing to test structure before understanding the implementation
**Correct approach**: Vertical slices via tracer bullets. One test → one implementation → repeat. Each test responds to what you learned from the previous cycle. Because you just wrote the code, you know exactly what behavior matters and how to verify it.
```
WRONG (horizontal):
RED: test1, test2, test3, test4, test5
GREEN: impl1, impl2, impl3, impl4, impl5
RIGHT (vertical):
RED→GREEN: test1→impl1
RED→GREEN: test2→impl2
RED→GREEN: test3→impl3
...
```
## Workflow
### 1. Planning
When exploring the codebase, use the project's domain glossary so that test names and interface vocabulary match the project's language, and respect ADRs in the area you're touching.
Before writing any code:
- [ ] Confirm with user what interface changes are needed
- [ ] Confirm with user which behaviors to test (prioritize)
- [ ] Identify opportunities for [deep modules](deep-modules.md) (small interface, deep implementation)
- [ ] Design interfaces for [testability](interface-design.md)
- [ ] List the behaviors to test (not implementation steps)
- [ ] Get user approval on the plan
Ask: "What should the public interface look like? Which behaviors are most important to test?"
**You can't test everything.** Confirm with the user exactly which behaviors matter most. Focus testing effort on critical paths and complex logic, not every possible edge case.
### 2. Tracer Bullet
Write ONE test that confirms ONE thing about the system:
```
RED: Write test for first behavior → test fails
GREEN: Write minimal code to pass → test passes
```
This is your tracer bullet - proves the path works end-to-end.
### 3. Incremental Loop
For each remaining behavior:
```
RED: Write next test → fails
GREEN: Minimal code to pass → passes
```
Rules:
- One test at a time
- Only enough code to pass current test
- Don't anticipate future tests
- Keep tests focused on observable behavior
### 4. Refactor
After all tests pass, look for [refactor candidates](refactoring.md):
- [ ] Extract duplication
- [ ] Deepen modules (move complexity behind simple interfaces)
- [ ] Apply SOLID principles where natural
- [ ] Consider what new code reveals about existing code
- [ ] Run tests after each refactor step
**Never refactor while RED.** Get to GREEN first.
## Checklist Per Cycle
```
[ ] Test describes behavior, not implementation
[ ] Test uses public interface only
[ ] Test would survive internal refactor
[ ] Code is minimal for this test
[ ] No speculative features added
```
+33
View File
@@ -0,0 +1,33 @@
# Deep Modules
From "A Philosophy of Software Design":
**Deep module** = small interface + lots of implementation
```
┌─────────────────────┐
│ Small Interface │ ← Few methods, simple params
├─────────────────────┤
│ │
│ │
│ Deep Implementation│ ← Complex logic hidden
│ │
│ │
└─────────────────────┘
```
**Shallow module** = large interface + little implementation (avoid)
```
┌─────────────────────────────────┐
│ Large Interface │ ← Many methods, complex params
├─────────────────────────────────┤
│ Thin Implementation │ ← Just passes through
└─────────────────────────────────┘
```
When designing interfaces, ask:
- Can I reduce the number of methods?
- Can I simplify the parameters?
- Can I hide more complexity inside?
+31
View File
@@ -0,0 +1,31 @@
# Interface Design for Testability
Good interfaces make testing natural:
1. **Accept dependencies, don't create them**
```typescript
// Testable
function processOrder(order, paymentGateway) {}
// Hard to test
function processOrder(order) {
const gateway = new StripeGateway();
}
```
2. **Return results, don't produce side effects**
```typescript
// Testable
function calculateDiscount(cart): Discount {}
// Hard to test
function applyDiscount(cart): void {
cart.total -= discount;
}
```
3. **Small surface area**
- Fewer methods = fewer tests needed
- Fewer params = simpler test setup
+59
View File
@@ -0,0 +1,59 @@
# When to Mock
Mock at **system boundaries** only:
- External APIs (payment, email, etc.)
- Databases (sometimes - prefer test DB)
- Time/randomness
- File system (sometimes)
Don't mock:
- Your own classes/modules
- Internal collaborators
- Anything you control
## Designing for Mockability
At system boundaries, design interfaces that are easy to mock:
**1. Use dependency injection**
Pass external dependencies in rather than creating them internally:
```typescript
// Easy to mock
function processPayment(order, paymentClient) {
return paymentClient.charge(order.total);
}
// Hard to mock
function processPayment(order) {
const client = new StripeClient(process.env.STRIPE_KEY);
return client.charge(order.total);
}
```
**2. Prefer SDK-style interfaces over generic fetchers**
Create specific functions for each external operation instead of one generic function with conditional logic:
```typescript
// GOOD: Each function is independently mockable
const api = {
getUser: (id) => fetch(`/users/${id}`),
getOrders: (userId) => fetch(`/users/${userId}/orders`),
createOrder: (data) => fetch('/orders', { method: 'POST', body: data }),
};
// BAD: Mocking requires conditional logic inside the mock
const api = {
fetch: (endpoint, options) => fetch(endpoint, options),
};
```
The SDK approach means:
- Each mock returns one specific shape
- No conditional logic in test setup
- Easier to see which endpoints a test exercises
- Type safety per endpoint
+10
View File
@@ -0,0 +1,10 @@
# Refactor Candidates
After TDD cycle, look for:
- **Duplication** → Extract function/class
- **Long methods** → Break into private helpers (keep tests on public interface)
- **Shallow modules** → Combine or deepen
- **Feature envy** → Move logic to where data lives
- **Primitive obsession** → Introduce value objects
- **Existing code** the new code reveals as problematic
+61
View File
@@ -0,0 +1,61 @@
# Good and Bad Tests
## Good Tests
**Integration-style**: Test through real interfaces, not mocks of internal parts.
```typescript
// GOOD: Tests observable behavior
test("user can checkout with valid cart", async () => {
const cart = createCart();
cart.add(product);
const result = await checkout(cart, paymentMethod);
expect(result.status).toBe("confirmed");
});
```
Characteristics:
- Tests behavior users/callers care about
- Uses public API only
- Survives internal refactors
- Describes WHAT, not HOW
- One logical assertion per test
## Bad Tests
**Implementation-detail tests**: Coupled to internal structure.
```typescript
// BAD: Tests implementation details
test("checkout calls paymentService.process", async () => {
const mockPayment = jest.mock(paymentService);
await checkout(cart, payment);
expect(mockPayment.process).toHaveBeenCalledWith(cart.total);
});
```
Red flags:
- Mocking internal collaborators
- Testing private methods
- Asserting on call counts/order
- Test breaks when refactoring without behavior change
- Test name describes HOW not WHAT
- Verifying through external means instead of interface
```typescript
// BAD: Bypasses interface to verify
test("createUser saves to database", async () => {
await createUser({ name: "Alice" });
const row = await db.query("SELECT * FROM users WHERE name = ?", ["Alice"]);
expect(row).toBeDefined();
});
// GOOD: Verifies through interface
test("createUser makes user retrievable", async () => {
const user = await createUser({ name: "Alice" });
const retrieved = await getUser(user.id);
expect(retrieved.name).toBe("Alice");
});
```
+76
View File
@@ -0,0 +1,76 @@
---
name: to-prd
description: Turn the current conversation context into a PRD and publish it to the project issue tracker. Use when user wants to create a PRD from the current context.
---
This skill takes the current conversation context and codebase understanding and produces a PRD. Do NOT interview the user — just synthesize what you already know.
The issue tracker and triage label vocabulary should have been provided to you — run `/setup-matt-pocock-skills` if not.
## Process
1. Explore the repo to understand the current state of the codebase, if you haven't already. Use the project's domain glossary vocabulary throughout the PRD, and respect any ADRs in the area you're touching.
2. Sketch out the major modules you will need to build or modify to complete the implementation. Actively look for opportunities to extract deep modules that can be tested in isolation.
A deep module (as opposed to a shallow module) is one which encapsulates a lot of functionality in a simple, testable interface which rarely changes.
Check with the user that these modules match their expectations. Check with the user which modules they want tests written for.
3. Write the PRD using the template below, then publish it to the project issue tracker. Apply the `ready-for-agent` triage label - no need for additional triage.
<prd-template>
## Problem Statement
The problem that the user is facing, from the user's perspective.
## Solution
The solution to the problem, from the user's perspective.
## User Stories
A LONG, numbered list of user stories. Each user story should be in the format of:
1. As an <actor>, I want a <feature>, so that <benefit>
<user-story-example>
1. As a mobile bank customer, I want to see balance on my accounts, so that I can make better informed decisions about my spending
</user-story-example>
This list of user stories should be extremely extensive and cover all aspects of the feature.
## Implementation Decisions
A list of implementation decisions that were made. This can include:
- The modules that will be built/modified
- The interfaces of those modules that will be modified
- Technical clarifications from the developer
- Architectural decisions
- Schema changes
- API contracts
- Specific interactions
Do NOT include specific file paths or code snippets. They may end up being outdated very quickly.
Exception: if a prototype produced a snippet that encodes a decision more precisely than prose can (state machine, reducer, schema, type shape), inline it within the relevant decision and note briefly that it came from a prototype. Trim to the decision-rich parts — not a working demo, just the important bits.
## Testing Decisions
A list of testing decisions that were made. Include:
- A description of what makes a good test (only test external behavior, not implementation details)
- Which modules will be tested
- Prior art for the tests (i.e. similar types of tests in the codebase)
## Out of Scope
A description of the things that are out of scope for this PRD.
## Further Notes
Any further notes about the feature.
</prd-template>
+7
View File
@@ -0,0 +1,7 @@
---
name: zoom-out
description: Tell the agent to zoom out and give broader context or a higher-level perspective. Use when you're unfamiliar with a section of code or need to understand how it fits into the bigger picture.
disable-model-invocation: true
---
I don't know this area of code well. Go up a layer of abstraction. Give me a map of all the relevant modules and callers, using the project's domain glossary vocabulary.
+107
View File
@@ -0,0 +1,107 @@
---
name: sm-flow
description: OpenSpec-first 工程流程 harness。仅在用户显式调用 /sm-flow、/sm-flow explore、/sm-flow apply、/sm-flow archive,或明确要求使用 sm-flow 流程时使用;不要根据需求类型自动触发。
---
# SM Flow
SM Flow 是一个**协议层 harness**——编排 OpenSpec 的完整生命周期。它通过阶段、门控、人类对齐和长期记忆,约束 agent 以正确的顺序、条件和标准使用 OpenSpec。
sm-flow 会自动维护 `devflow/` 目录作为项目长期记忆。用户不需要手动管理它,sm-flow 会在流程中自动读取和回填。
## 触发规则
只在用户显式调用时使用 sm-flow:
- 用户输入 `/sm-flow`、`/sm-flow explore`、`/sm-flow apply`、`/sm-flow archive`。
- 用户用自然语言明确要求"使用 sm-flow"、"走 sm-flow 流程"或等价表达。
不要根据需求类型自动触发 sm-flow。即使任务涉及 OpenSpec、跨模块、接口契约、需求澄清或 devflow 归档,只要用户没有显式要求 sm-flow,就按普通工程任务处理。
## 四层架构
```
sm-flow → 编排层(harness):阶段、门控、产物约束、人类对齐
OpenSpec → 执行引擎:propose/apply/archive 的能力提供方
devflow/ → 记忆层:为编排层提供上下文,接收执行结果的回填
code → 实现结果:apply 的产出
```
- OpenSpec 是唯一执行真理源:apply 阶段只能基于 OpenSpec 执行,不能绕过 OpenSpec 直接写代码。
- devflow 是上下文真理源:术语、历史决策、验收记录来自 devflow,用于增强 OpenSpec,不替代 OpenSpec。
- 如果 devflow 和 OpenSpec 冲突,先汇报冲突、让用户确认、修正 OpenSpec,再继续执行。
- propose 阶段产出的 OpenSpec 默认为 **Draft OpenSpec**:它是澄清和审计对象,不是 apply 的执行许可。
- 只有通过 commit 检查后的 OpenSpec 才是 **Committed OpenSpec**;apply 只能执行 Committed OpenSpec。
## 核心规则
以下 6 条是硬约束,违反即流程失败。其余约束按阶段定义在 `references/phase-contracts.md`。
1. **OpenSpec 是唯一执行真理源**。apply 阶段必须读取 Committed OpenSpec 文件作为执行依据;对话中的描述不等于产物。Draft OpenSpec 是讨论对象,不是执行许可。
2. **不得跳过 context**。生成 OpenSpec 前,必须先读取相关 devflow 上下文(glossary、ADR、历史项目)。
3. **不得跳过 grill**。必须按 `references/scales.md` 的当前分档要求完成澄清或验证。
4. **不得跳过 commit**。进入 apply 前,Draft OpenSpec 必须通过 commit 检查成为 Committed OpenSpec。
5. **冲突必须先分类再处理**。OpenSpec 不准(规格遗漏)→ 修正 OpenSpec;代码偏离(实现偏差)→ 修正代码;不确定或涉及设计方向 → 暂停并等待用户确认。
6. **能力来源必须显式声明**。每个阶段先声明使用外部子 skill / OpenSpec CLI / sm-flow 内置协议;外部能力不可用时可使用 `references/fallbacks.md` 的内置协议,但必须标注为 fallback。若外部能力和内置协议都不可用,流程失败。
每个阶段的过程约束(question pool、one-at-a-time、cross-artifact 对齐、冲突回写等)和质量约束(可观测产出要求)见 `references/phase-contracts.md` 中对应阶段的退出条件和 checkpoint。
## 用户命令
| 命令 | 用户意图 | harness 内部行为 |
|---|---|---|
| `/sm-flow` | 完整流程 | clarify → context → propose → grill → specify → audit → commit → apply → archive |
| `/sm-flow explore` | 先想想 | 带上下文的探索模式 |
| `/sm-flow apply` | 只执行 | 检查 commit gate → apply |
| `/sm-flow archive` | 收尾 | 回填 devflow + 归档确认 |
用户也可以用自然语言指定从某个阶段继续,例如"ops-message-support 的 grill 已经做完了,继续"。harness 识别意图后,自动补做最小前置检查,然后从指定阶段继续。
## 可见 Checkpoint
内部阶段不是用户 API。对用户汇报进度时,默认只暴露 4 个 checkpoint:
| Checkpoint | 覆盖内部阶段 | 用户可见含义 |
|---|---|---|
| Discover | clarify + context + propose + grill | 澄清目标、读取 devflow、形成轻量 proposal、解决关键问题 |
| Commit | specify + audit + commit | 补全 OpenSpec、做架构/产物对齐、生成 Committed OpenSpec |
| Apply | apply | 基于 Committed OpenSpec 实现和验证 |
| Archive | archive | 回填 devflow、汇报验收、询问是否归档 OpenSpec |
除非用户要求看细节,进度汇报、暂停点和恢复提示应使用 checkpoint 名称,而不是逐个暴露 9 个内部阶段。内部阶段仍按顺序执行,并以 `references/phase-contracts.md` 为准。
## 首次加载
执行前只读取当前任务需要的 reference 文件:
- 需要执行阶段时,先读取 `references/phase-contracts.md`;如果当前阶段涉及接口影响分级、分档、启动规则、快速模式或完成标准,再补读 `references/operating-rules.md`;如果外部 OpenSpec 能力或子 skill 不可用,再补读 `references/fallbacks.md`。
- 判断或执行 `micro / standard / complex` 分档时,读取 `references/scales.md`;其它文件不得重复定义分档细节。
- 当 checkpoint / gate / fallback / Draft / Committed 等术语含义不清,或需要统一对用户说明时,读取 `references/glossary.md`。
- 创建或更新 PRD、ADR、验收报告、词汇表、复合知识文档时,读取 `references/templates.md`。
- archive 阶段或需要从 OpenSpec 提取产物时,读取 `references/archive-rules.md`。
## 内部阶段
9 个内部阶段,按执行顺序:
1. clarify — 入口澄清:接收初始需求,澄清到可生成轻量 proposal。
2. context — 上下文收集:读取 devflow 的 glossary、ADR、历史项目、compound knowledge。
3. propose — 轻量 propose:只生成 proposal.md,不调用 openspec-propose。
4. grill — 人类对齐澄清:evidence-driven 查证 + user-interview one-at-a-time,回写 proposal。
5. specify — 细化 + 对齐:基于已稳定的 proposal 补全 design/specs/tasks,做 cross-artifact 对齐。
6. audit — 架构审计:审计结果如果影响实现,回写 OpenSpec design/tasks。
7. commit — Commit OpenSpec:检查 Draft OpenSpec 是否达到可执行状态,提交为 Committed OpenSpec。
8. apply — OpenSpec 执行:基于 Committed OpenSpec 实现代码。
9. archive — 回填 + 归档:从 OpenSpec 产物和 decisions.md 提炼长期档案,询问是否归档。
每个阶段的进入条件、动作、输出和退出标准见 `references/phase-contracts.md`。
关键阶段的完成判断也以 `references/phase-contracts.md` 为准;如果缺少显式 checkpoint 或能力来源声明,该阶段不得视为已完成。
## 快速模式
快速模式的具体约束见 `references/operating-rules.md`。
## 完成标准
流程完成标准见 `references/operating-rules.md`。
@@ -0,0 +1,167 @@
# 归档规则
archive 阶段的目标是把 OpenSpec 产物、实现结果和过程日志转化为持久、可读、可复用的项目记忆。sm-flow 在 clarify → apply 期间只维护 `decisions.md` 作为过程日志,archive 阶段从中提取完整 devflow 档案。
## Archive 强制执行顺序
Archive 阶段必须按以下顺序执行,不得跳过或重排:
### Step 1: 创建 devflow 档案(必需)
- [ ] 创建 `devflow/projects/YYYY-MM-DD-{slug}/brief.md`
(从 proposal.md 提取:背景、目标、范围、非目标)
- [ ] 按 `references/scales.md` 的当前分档决定是否创建 `devflow/projects/YYYY-MM-DD-{slug}/evidence.md`
(创建时从 decisions.md 提取 evidence-driven 记录)
- [ ] 创建 `devflow/projects/YYYY-MM-DD-{slug}/decisions.md`
(整理为最终版:关键决策、权衡、风险)
- [ ] 创建 `devflow/projects/YYYY-MM-DD-{slug}/acceptance.md`
(记录:静态验证、脚本验证、浏览器/人工验证、未验证)
### Step 2: 更新索引(必需)
- [ ] 在 `devflow/index.md` 末尾追加或更新一行:
`| YYYY-MM-DD | slug | 领域 | 关键词 | openspec/changes/xxx | {status} |`
### Step 3: 标记 OpenSpec(必需)
- [ ] 创建 `openspec/changes/{slug}/.archive-ready` 文件
### Step 4: 向用户汇报(必需)
- [ ] 列出创建的 devflow 档案文件路径(验证文件实际存在于磁盘)
- [ ] 汇报验证情况(按静态验证、脚本验证、浏览器/人工验证、未验证分类)
- [ ] 列出剩余风险或后续事项
- [ ] 询问:**是否现在归档 OpenSpec?**
### Step 5: 用户确认后执行 OpenSpec Archive(可选)
- [ ] 调用 `openspec-archive-change`
- [ ] 记录 archive 结果
**自检**:在执行 Step 4 前,检查 Step 1-3 是否都完成。
---
## 目录规则
项目档案路径:
```text
devflow/projects/YYYY-MM-DD-{slug}/
```
archive 阶段按 `references/scales.md` 的当前分档创建以下文件:
- `brief.md`:从 proposal.md 提取背景、目标、范围、非目标。
- `evidence.md`:从 decisions.md 中的 evidence-driven 记录提取;是否独立创建按 `references/scales.md` 执行。
- `decisions.md`:保持为最终版,整理格式。
- `acceptance.md`:从实现结果和验证结果提取。
同时维护仓库级索引:
- `devflow/index.md`
按需创建以下扩展文件:
- `prd.md`
- `research.md`
- `design.md`
- `tasks.md`
- `alignment.md`
- `adr/*.md`
不要逐字复制完整 OpenSpec 文件,也不要重复 OpenSpec 的 proposal/design/tasks。应提炼 OpenSpec 如何指导执行:背景、证据、用户决策、任务状态、假设、验证结果、风险,以及执行中对 OpenSpec 的修正。
## 产物分档
分档的适用场景和必须文件见 `references/scales.md`。本文件只定义 archive 阶段的创建顺序、提取映射和索引规则。
## 提取映射
| 来源 | 提取内容 | 写入位置 |
| --- | --- | --- |
| `decisions.md`(过程日志) | question pool、evidence-driven 汇报状态、user-interview 确认状态、关键取舍 | `decisions.md`(整理格式为最终版) |
| `decisions.md`(过程日志) | evidence-driven 结论、代码/文档证据 | `evidence.md` |
| `proposal.md` | 为什么做、做什么、范围、非目标 | `brief.md` |
| `design.md` | 技术方案、关键决策、风险;只提炼长期有用内容 | `evidence.md` / 按需 `design.md` |
| `specs/**/*.md` | requirement 标题和 scenario 意图 | `brief.md` 或 `acceptance.md` 的验收追踪 |
| `tasks.md` | checkbox 状态、剩余工作、执行切片 | `acceptance.md`;复杂项目可拆 `tasks.md` |
| 测试/构建输出 | 验证命令、结果、验证类型 | `acceptance.md` |
| diagnose 记录 | 根因、修复、回归验证 | `acceptance.md` |
| 词汇表更新 | 术语和业务规则 | `devflow/glossary/CONTEXT.md` |
| 可复用经验 | 持久工程知识 | `devflow/compound/YYYY-MM-DD-{type}-{slug}.md` |
| 项目索引 | 日期、slug、领域、关键词、关联 OpenSpec、状态 | `devflow/index.md` |
## 索引维护规则
`devflow/index.md` 是 context 阶段的默认入口,archive 阶段回填时必须维护。
最小字段:
| 日期 | slug | 领域 | 关键词 | 关联 OpenSpec | 状态 |
| --- | --- | --- | --- | --- | --- |
规则:
- 每个 `devflow/projects/YYYY-MM-DD-{slug}/` 默认对应一行索引。
- archive 阶段新建或更新项目档案时,必须新增或更新对应行。
- 如果项目仍在进行,状态写 `active`;已验收但未 archive 写 `accepted-unarchived`;已 archive 写 `archived`;暂停写 `paused`。
- 关键词只放能帮助 context 阶段定位的术语,不复制 brief 内容。
- 如果无法准确判断领域或状态,写 `unknown`,并在 `acceptance.md` 记录待补。
## 验收记录规则
必须真实记录验证情况,并按类型分类:
- **静态验证**:语法检查、grep/rg 检查、结构检查、类型检查等不运行完整功能的验证。
- **脚本验证**:生成脚本、测试命令、构建命令、自动化检查等可重复命令。
- **浏览器/人工验证**:需要用户或代理在界面中点击、观察、确认的行为验证。
- **未验证**:未运行的验证必须记录原因、风险和建议补验步骤。
记录要求:
- 如果验证通过,记录命令/步骤和覆盖范围。
- 如果验证失败,记录失败摘要和是否阻塞验收。
- 如果需要人工验证,列出明确步骤,不要用"手动测试一下"这种模糊描述。
## ADR 规则
同时满足以下条件时创建 ADR:
1. 决策难以逆转。
2. 缺少上下文会让未来维护者困惑。
3. 决策来自真实权衡,而不是简单偏好。
项目内 ADR 存放于:
```text
devflow/projects/YYYY-MM-DD-{slug}/adr/
```
跨项目可复用决策或经验存放于:
```text
devflow/compound/YYYY-MM-DD-decision-{slug}.md
```
## 归档确认
OpenSpec archive 是显式 human-in-the-loop 动作。archive 前必须确认 devflow 已经回填 OpenSpec 的关键执行信息:
- archive 阶段可以建议 archive,但必须先询问用户。
- 在用户确认前,不要执行 archive。
- 如果用户暂不归档,在 acceptance 中记录原因或状态。
- 如果用户确认归档,执行后记录 archive 结果和剩余档案位置。
## 归档交接
archive 阶段结束时告诉用户:
- 创建或更新了哪些档案文件。
- `devflow/index.md` 是否已更新。
- 运行了哪些验证,并按静态验证、脚本验证、浏览器/人工验证、未验证分类。
- 还剩哪些风险或后续事项。
- 明确询问:是否现在 archive OpenSpec change?
@@ -0,0 +1,49 @@
# 内置执行协议
本文件只在外部 OpenSpec CLI 或子 skill 不可用时使用。fallback 不是跳过阶段,而是由 sm-flow 用文件方式完成同等最小产物。每次使用 fallback 都必须写入 `decisions.md` 或 `acceptance.md`,说明能力来源、缺失能力、影响和剩余风险。
## 通用规则
- 优先使用外部能力;只有不可用、不可发现或无法在当前环境调用时才使用内置协议。
- 不得因为使用 fallback 跳过 context、grill、commit、apply 授权或 archive 确认。
- fallback 产物仍写入 `openspec/changes/{slug}/` 和 `devflow/projects/YYYY-MM-DD-{slug}/`。
- 如果内置协议也无法满足阶段退出条件,暂停并向用户说明阻塞项。
## grill 内置协议
- 建立 question pool,至少覆盖术语、边界、验收;涉及参考实现或项目基础设施时加入技术实现问题。
- 将问题标记为 `evidence-driven` 或 `user-interview`。
- 先查证 evidence-driven 问题并汇报结论,再逐个询问 user-interview 问题。
- 按 `references/scales.md` 的当前分档满足 grill 要求。
- 将 question pool、证据结论、用户原话和确认状态写入 `decisions.md`;影响实现的结论回写 `proposal.md`。
## openspec 提案内置协议
- 在 `openspec/changes/{slug}/` 创建或更新:
- `proposal.md`:问题、方案、范围、非目标、上下文约束、风险。
- 设计产物:实现设计、接口影响、关键决策、架构风险;形式按 `references/scales.md` 的当前分档要求执行。
- `specs/*/spec.md` 或等价 functional spec:描述用户可观察行为和验收场景。
- `tasks.md`:按可执行切片拆分任务,并给每项写可验证验收标准。
- 运行 cross-artifact 对齐检查:proposal → 设计产物 → specs → tasks。
- 如果发现 gap,先修正 OpenSpec,再进入 commit。
## audit 内置协议
- 用 5 句话以内说明模块链路、数据所有权、跨模块依赖、架构风险和是否需要回写 OpenSpec。
- 如果风险影响实现,修正设计产物或 `tasks.md`。
- 将结论写入 `decisions.md`。
## openspec apply 内置协议
- 只依据 Committed OpenSpec 的 specs/tasks 实现;devflow 只作上下文参考。
- 开始前检查 `.committed` 文件;缺失则返回 commit。
- 如触发 pre-apply checkpoint,先阅读参考实现、grep 项目基础设施模式,并把技术栈清单写入 `decisions.md`。
- 按 tasks 的纵向切片实现、验证并更新任务状态。
- 发现冲突时按三类处理:OpenSpec 不准则修 OpenSpec,代码偏离则修代码,不确定则暂停等用户确认。
## openspec archive 内置协议
- 不删除或移动 OpenSpec change;只标记归档准备状态。
- 完成 devflow 回填、更新 `devflow/index.md`、创建 `.archive-ready`。
- 向用户汇报已创建文件、验证分类、剩余风险,并询问是否需要真实 OpenSpec archive。
- 如果外部 archive 能力仍不可用,在 `acceptance.md` 标记 `accepted-unarchived`。
@@ -0,0 +1,21 @@
# 术语表
本文件统一 sm-flow 协议中的核心词。优先使用这些词,避免同一概念多种说法。
| 术语 | 含义 | 使用边界 |
| --- | --- | --- |
| sm-flow | 协议层 harness | 编排 OpenSpec 生命周期,不替代 OpenSpec |
| OpenSpec | 当前变更的执行真理源 | apply 只能依据 Committed OpenSpec |
| devflow | 长期记忆和上下文层 | 提供术语、历史决策、验收记录,不直接指挥实现 |
| checkpoint | 用户可见检查点 | 默认只暴露 Discover / Commit / Apply / Archive |
| gate | 硬门控 | 不满足就不能进入下一关键动作,如 commit gate |
| Draft OpenSpec | 讨论和审计对象 | propose/specify 期间产生,不能直接 apply |
| Committed OpenSpec | 已通过 commit gate 的 OpenSpec | apply 的唯一执行依据 |
| fallback | 内置执行协议 | 外部 OpenSpec CLI 或子 skill 不可用时使用,必须标注 |
| decisions.md | 过程日志 | clarify 到 apply 期间记录问题、证据、决策、冲突和回写 |
| .committed | commit gate 标记文件 | 存在才可进入合规 apply |
| .archive-ready | archive 准备标记文件 | 表示 devflow 已回填,等待用户确认是否 archive |
| Discover | 用户可见 checkpoint | 覆盖 clarify + context + propose + grill |
| Commit | 用户可见 checkpoint | 覆盖 specify + audit + commit |
| Apply | 用户可见 checkpoint | 覆盖 apply |
| Archive | 用户可见 checkpoint | 覆盖 archive |
@@ -0,0 +1,120 @@
# 运行规则
本文件承载稳定但不必放在顶层 `SKILL.md` 的运行规则。
## 接口影响分级
接口影响分级判断的是"记录在哪里、是否需要独立文档",不是判断"是否需要关注"。凡涉及字段、DTO、service 方法、API、事件、回调、数据库契约、命令契约、跨模块调用语义或内部决策逻辑变化,都必须先做分级。
| 级别 | 判断条件 | 产物要求 |
| --- | --- | --- |
| L1 内部实现 | 不改变任何调用方可观察的接口、字段、状态、错误码、数据范围、排序、过滤、权限结果、状态流转、副作用或文档承诺 | 不需要接口影响文档,只在 OpenSpec tasks 或 acceptance 记录验证 |
| L2 内部接口 | 改 DTO、service 方法、内部事件、内部 RPC 或内部判断逻辑,且所有消费者都在同一实现范围内 | 必须记录接口影响范围,可内联到 OpenSpec design/specs/tasks 或 devflow evidence/decisions |
| L3 协作接口 | 影响其他模块、其他服务、前端、外部系统、跨团队消费者、数据库契约、消息事件、回调或 SDK | 必须产出独立接口文档或等价独立章节 |
| L4 破坏性接口 | 删除字段、改字段语义、改状态机、改错误码、破坏兼容、旧调用方可能失败,或需要迁移、灰度、回滚 | 独立接口文档 + 迁移/回滚说明;必要时创建 ADR |
判断策略:
- 如果只是修复 bug,让接口回到原 OpenSpec 或原文档承诺,通常是 L1/L2。
- 如果判断逻辑改变了返回数据、错误码、状态、权限结果、排序/过滤、幂等性、时序或副作用,至少按 L3 检查。
- 如果旧调用方不改代码会失败、少数据、多数据、状态不同或错误码不同,按 L4 处理。
- 如果无法确定调用方边界或兼容性,默认提高一级并作为 `user-interview` 问题等待确认。
## 启动检查
1. 识别用户命令意图:
- `/sm-flow`(无参数):完整流程,从 clarify 开始。
- `/sm-flow apply [change]`:只执行,检查 commit gate → apply。
- `/sm-flow explore`:带上下文的探索模式,不走标准阶段链。
- `/sm-flow archive [change]`:收尾,回填 devflow + 归档确认。
- 明确要求"使用 sm-flow"或"走 sm-flow 流程":按显式调用处理。
- 自然语言指定阶段继续:识别意图后,自动补做最小前置检查,然后从指定阶段继续。
2. 判断启动模式:
- 完整模式:用户提供粗略想法或初始 PRD。
- Research 模式:用户已有 research,需要转成或修正 OpenSpec。
- PRD 文件模式:用户提供已有 PRD 路径。
- 恢复模式:用户希望从某个阶段继续(补做最小前置检查)。
- 快速模式:小改动,合并 gate;具体分档规则见 `references/scales.md`。
3. 如果缺少 `devflow/`,初始化:
- `devflow/projects/`
- `devflow/glossary/CONTEXT.md`
- `devflow/compound/`
4. 如果根目录存在旧 `CONTEXT.md`,且 `devflow/glossary/CONTEXT.md` 不存在或为空,询问用户是迁移还是合并。
5. 检查 OpenSpec 和子 skill 是否可用:
- OpenSpec 能力:`openspec-propose`、`openspec-apply-change`、`openspec-archive-change`。
- 辅助能力:`to-prd`、`grill-with-docs`、`diagnose`、`tdd`、`zoom-out`。
6. 如果 OpenSpec 或子 skill 不可用,不要静默跳过;使用内置执行协议(见 `references/fallbacks.md`),并在当前 checkpoint 说明 fallback 来源、影响和剩余风险。
## 进度汇报
用户可见进度默认折叠为 4 个 checkpoint:
| Checkpoint | 内部阶段 |
| --- | --- |
| Discover | clarify + context + propose + grill |
| Commit | specify + audit + commit |
| Apply | apply |
| Archive | archive |
汇报规则:
- 面向用户时优先使用 checkpoint 名称,不逐个汇报 9 个内部阶段。
- 内部阶段只在 checkpoint 摘要中作为证据列出,例如"Discover 已完成:读取了 devflow、生成 proposal、解决 2 个问题"。
- 只有发生阻塞、冲突、fallback、用户要求继续某个内部阶段,或需要解释恢复位置时,才暴露内部阶段名。
- 当前分档的汇报压缩规则见 `references/scales.md`;无论分档如何,都不要把内部阶段名当作用户操作入口。
## 项目标识规则
- 整个流程使用同一个 slug。
- 优先使用 OpenSpec change name。
- 如果还没有,则从功能标题生成 kebab-case slug。
- 项目档案目录格式:`devflow/projects/YYYY-MM-DD-{slug}/`。
- 如果目录已存在,默认恢复该项目;除非用户明确要求新开一轮。
## Devflow 产物分层
Devflow 是 sm-flow 自动维护的项目长期记忆层,不复制 OpenSpec 的执行产物。
**过程日志**(clarify → apply 期间维护):
- `decisions.md`:question pool、evidence-driven 汇报状态、user-interview 确认状态、关键取舍、风险接受、OpenSpec 回写记录、冲突分类记录。
**最终档案**(archive 阶段从 decisions.md + OpenSpec 产物提取):
- `brief.md`:背景、目标、范围、非目标、分档、关联 OpenSpec change。
- `evidence.md`:代码/文档证据、历史决策、evidence-driven 结论和汇报状态;分档要求见 `references/scales.md` 和 `references/archive-rules.md`。
- `acceptance.md`:实现结果、验证命令、未验证项、归档状态、后续事项。
**按需产物**(archive 阶段按需创建):
- `prd.md`:需求复杂、用户明确要求、或需要对外协作。
- `research.md`:存在真实调研、代码考古、竞品/API 对比或复杂方案比较。
- `design.md`:不适合放进 OpenSpec design 的长期背景或架构审计摘要。
- `tasks.md`:跨会话的人类追踪;执行任务仍属于 OpenSpec。
- `alignment.md` / `clarifications.md`:仅在 gap 或澄清很多时使用。
- `adr/*.md` 和 `compound/*.md`:仅在满足 ADR / compound knowledge 规则时使用。
**规模分档**:`micro / standard / complex` 的唯一规则源是 `references/scales.md`。
## 快速模式
快速模式适用于 `references/scales.md` 定义的 micro 变更。它合并 gate 而不仅仅是压缩产物;具体覆盖规则见 `references/scales.md`。
无论什么模式,以下内容必须保留:
- context 最小上下文收集:至少检查 glossary 和相关 ADR。
- grill 最小澄清:按 `references/scales.md` 当前分档要求执行;evidence-driven 结论仍需汇报。
- commit gate:确认没有未解决用户问题、接口影响已记录、OpenSpec tasks/specs 可执行;完整性检查按 `references/scales.md` 当前分档要求执行。
- apply 仍由 OpenSpec tasks/specs 驱动执行。
- archive 轻量回填:记录验收结果、OpenSpec 链接和归档状态。
## 完成标准
只有同时满足以下条件,流程才算完成:
- 用户可见的 Discover、Commit、Apply、Archive checkpoint 已完成,或未完成项已明确标记为暂停/不适用。
- OpenSpec proposal、设计产物、specs、tasks 已按当前分档生成或更新到可执行状态。
- 实现或规划工作已完成,且执行依据来自 OpenSpec。
- 已运行验证,或已记录未运行验证的原因。
- `devflow/projects/YYYY-MM-DD-{slug}/` 包含 `references/scales.md` 和 `references/archive-rules.md` 要求的当前分档档案。
- 用户知道剩余风险与下一步,并已被询问是否归档 OpenSpec change。
@@ -0,0 +1,352 @@
# 阶段契约
本文件是 SM Flow 的逐阶段执行准则。核心原则:**sm-flow 编排 OpenSpec,OpenSpec 指挥执行,执行结果回填 devflow**。
执行顺序:clarify → context → propose → grill → specify → audit → commit → apply → archive。
## 目录
- clarify — 入口澄清
- context — 上下文收集
- propose — 轻量 propose
- grill — 人类对齐澄清
- specify — 细化 + 对齐
- audit — 架构审计
- commit — Commit OpenSpec
- apply — OpenSpec 执行
- archive — 回填 + 归档
## clarify — 入口澄清
**进入条件**:用户提供粗略想法、初始 PRD、已有 research、issue,或要求启动 SM Flow。
**动作**:
- 收集问题、期望结果、目标用户、涉及代码区域、约束条件和可能的非目标。
- 如果用户已有 research,先识别它是否已经包含用户价值、技术方案、验收标准和任务拆分。
- 如果输入过于模糊,最多追加三轮聚焦问题。
- 当答案会改变 OpenSpec proposal/specs/tasks 时,优先一次只问一个问题。
- 如果需要判断 `micro / standard / complex` 分档,补读 `references/scales.md`。
**退出条件**:
- 问题可以用 1-2 句话说清楚。
- 期望结果可以用 1-2 句话说清楚。
- 已列出已知影响代码或模块;如果未知,也明确标记。
- 可以生成 OpenSpec change slug。
**输出**:
- 入口摘要。
- 初步 slug。
- devflow 规模分档:`micro` / `standard` / `complex`。
## context — 上下文收集
**进入条件**:clarify 已经有足够信息定位领域、项目或变更方向。
**动作**:
- 优先读取 `devflow/index.md`,按日期、slug、领域、关键词和关联 OpenSpec 定位候选项目。
- 如果 `devflow/index.md` 不存在,先从 `devflow/projects/` 现有目录初始化轻量索引,再继续本次上下文收集。
- 读取 `devflow/glossary/CONTEXT.md`,提取相关术语和业务规则。
- 搜索 `devflow/projects/` 中相关 PRD、design、tasks、acceptance 和 ADR。
- 搜索 `devflow/compound/` 中可复用 learning、trick、decision、explore。
- 记录哪些上下文会影响 OpenSpec proposal、设计产物、specs 或 tasks。
- 如果发现旧根目录 `CONTEXT.md` 与 `devflow/glossary/CONTEXT.md` 冲突,暂停并向用户汇报。
**退出条件**:
- 已形成"OpenSpec 输入上下文摘要"。
- 已记录 `devflow/index.md` 的使用状态:已命中 / 已初始化 / 无相关条目。
- 已列出相关 ADR 和不能违反的历史决策。
- 已列出需要写入或修正 OpenSpec 的上下文点。
**输出**:
- 上下文摘要,写入 `decisions.md`(过程日志)。会影响实现的上下文必须标记为"需进入 OpenSpec"。
## propose — 轻量 propose
**进入条件**:clarify + context 已经足够生成轻量 proposal。
**执行者**:sm-flow 内置协议。**不调用 openspec-propose**(完整 OpenSpec 产物留待 specify 阶段生成)。
**动作**:
- 创建或识别 `openspec/changes/{slug}/`。
- 写入 `proposal.md`,包含:问题、建议方案、范围、非目标、来自 devflow 的上下文约束、风险。
- **不生成 design.md、specs/、tasks.md**——这些留待 grill 澄清需求后在 specify 阶段补全。
- 用 context 阶段的 devflow 上下文增强 proposal。
- 在承诺方案方向前,先检查相关仓库代码。
**退出条件**:
- `openspec/changes/{slug}/proposal.md` 存在。
- 关键假设已显式记录。
**输出**:
- Draft OpenSpec proposal.md(轻量版)。
**Human checkpoint**:
- 向用户简要说明 proposal 范围、关键假设、主要风险、devflow 上下文如何影响方案。
- 作为 Discover checkpoint 的中间状态汇报;询问是否继续完成 Discover 的人类澄清部分。用户明确要求"全自动执行"时可跳过等待。
## grill — 人类对齐澄清
**进入条件**:propose 已有轻量 proposal.md。
**能力来源**:优先使用 `grill-with-docs`;不可用时使用 `references/fallbacks.md#grill-内置协议`,并在 `decisions.md` 标注 fallback。
**动作**:
- 优先使用 `grill-with-docs`。
- 进入 grill 时先建立一个 question pool,并记录到 `decisions.md`:
- 默认至少覆盖术语、边界、验收三个维度。
- **技术实现维度**(新增):当 proposal 提到参考实现、或涉及项目现有基础设施时,增加技术澄清问题:
- 参考实现的具体文件路径是什么?
- 项目现有的 [请求结构/MQ/缓存/加密/工具类] 标准是什么?
- 有哪些技术点需要先调研或新建?
- 如果变更涉及多模块、接口、权限、下游消费者、响应结构或生命周期规则,先把这些维度补进问题池。
- 逐项标记每个问题的模式:
- `evidence-driven`:问题能通过代码、文档、测试、OpenSpec 或既有 ADR 证明;代理先查证,再向用户汇报证据、结论和是否需要确认。
- `user-interview`:问题涉及产品偏好、范围边界、验收口径、风险接受度或价值取舍;必须问用户并等待确认。
- evidence-driven 和 user-interview 的推进节奏:先批量查证 evidence-driven 并一次性汇报结论,再逐个处理 user-interview 问题。不要把所有问题攒到最后一起问。
- 一次只问一个 `user-interview` 问题。
- 每个 `user-interview` 问题必须等待用户显式回答,并在 decisions.md 中记录:问题原文、用户原话、确认状态(已确认/未确认)。未确认的问题不能从 question pool 移除。
- 单个 `user-interview` 的确认只能解除该问题本身的阻塞,不能被解释为进入 apply 或修改执行目标文件的授权。
- 对接口影响等级、消费者边界或兼容性存在不确定时,必须作为 `user-interview` 问题等待用户确认。
- 如果澄清结果影响实现,必须回写 proposal.md。
- 术语一旦确认,更新 `devflow/glossary/CONTEXT.md`。
- 对难以逆转、依赖上下文、源自真实权衡的决策创建 ADR。
**退出条件**:
- question pool 已建立并覆盖当前 change 所需维度。
- 已满足 `references/scales.md` 中当前分档的 grill 要求。每个问题都必须记录属于 `evidence-driven` 还是 `user-interview`。
- 所有 evidence-driven 结论已向用户汇报。
- 所有 user-interview 决策已获得用户确认。
- 没有未解决或代理代确认的 user-interview 问题。
- 没有未判级或未确认的接口影响问题。
- 影响实现的结论已回写 proposal.md。
- 单个 grill 决策确认不等于 apply 授权;grill 完成后必须停在 commit,等待用户明确要求进入 apply。
- question pool、evidence-driven 结论、user-interview 确认必须写入 `decisions.md` 文件,不能只记录在对话中。
**输出**:
- 更新后的 proposal.md。
- 澄清记录:写入 `decisions.md`。包含 question pool、evidence-driven 汇报状态、user-interview 确认状态。
- 更新后的词汇表和 ADR。
**Human checkpoint**:
- 汇报已解决和未解决的问题、proposal 变更、术语和 ADR 更新。
- 汇报 Discover checkpoint 完成情况,并询问是否继续进入 Commit checkpoint。
## specify — 细化 + 对齐
**进入条件**:grill 已退出,需求已通过澄清稳定下来。
**能力来源**:优先使用 `openspec-propose`(基于已稳定的 proposal 补全完整 OpenSpec);按需使用 `to-prd`。进入本阶段必须先声明调用方式;外部能力不可用时使用 `references/fallbacks.md#openspec-提案-内置协议`,并在 `decisions.md` 标注 fallback。
**动作**:
- 基于已稳定的 proposal.md 补全设计产物、specs/、tasks.md:
- 优先调用 `openspec-propose`,输入中明确说明"proposal.md 已存在,本次只需按当前分档补全设计产物/specs/tasks"。
- 如果不可用,执行 `references/fallbacks.md#openspec-提案-内置协议`。
- 如果没有结构化 PRD,按需按 `to-prd` 协议生成 `brief.md`;复杂需求、对外协作或用户明确要求时再生成 `prd.md`。
- 独立 PRD 是否需要按 `references/scales.md` 的当前分档和用户要求判断。
- 用 grill 阶段的 decisions.md 记录增强 OpenSpec 产物:确保 design/specs/tasks 反映所有已确认的决策。
- **显式 cross-artifact 对齐检查**——在 checkpoint 中输出对齐检查表:
- `brief/prd` 中的目标、范围、非目标和验收预期 → `proposal` 是否覆盖。
- `proposal` 中的范围、约束和关键承诺 → `design` 是否覆盖。
- `design` 中影响实现的约束、接口影响和架构结论 → `specs` 或 `tasks` 是否覆盖。
- `specs` 中的可观察行为 → `tasks` 是否覆盖为可执行切片。
- 每项标记:已对齐 / 存在 gap。
- 检查是否涉及接口影响:
- 接口影响分级定义见 `references/operating-rules.md#接口影响分级`。
- 是否改变字段、DTO、service 方法、API、事件、回调、数据库契约、命令契约或跨模块调用语义。
- 接口内部判断逻辑是否改变调用方可观察行为。
- 按 L1/L2/L3/L4 记录接口影响等级;不确定时标记为 `user-interview` 问题。
- 如果存在 gap,在进入下一阶段前修复 OpenSpec。
- 如果发现不一致,优先修正 OpenSpec,而不是只修改 devflow 文档。
**退出条件**:
- OpenSpec 细化产物存在且与 proposal 对齐;产物形态按 `references/scales.md` 的当前分档要求执行。
- `brief.md` 已覆盖背景、目标、范围和非目标;复杂需求存在独立 `prd.md` 或用户明确不需要 PRD。
- cross-artifact 对齐检查表已生成(4 行,每行标记已对齐/存在 gap),没有未处理 gap。
- 涉及接口变更时,已记录接口影响等级和产物要求;不确定项已标记。
- 所有已知冲突已修正或等待用户决策。
**输出**:
- Draft OpenSpec:按 `references/scales.md` 的当前分档要求生成 proposal、设计、specs 和 tasks。
- `brief.md`,以及按需创建的 `prd.md`。
- cross-artifact 对齐检查表(写入 checkpoint 或 decisions.md)。
- 必要的 OpenSpec 修正。
## audit — 架构审计
**进入条件**:specify 已退出,完整 OpenSpec 产物已存在。
**能力来源**:优先使用 `zoom-out`;不可用时使用 `references/fallbacks.md#audit-内置协议`,并在 `decisions.md` 标注 fallback。
**动作**:
- 画出输入 → 处理 → 输出的模块链路。
- 识别跨模块依赖、数据所有权、生命周期和耦合风险。
- 检查是否与既有架构、ADR、OpenSpec design 冲突。
- 用不超过五句话写出架构风险评估。
- 如果审计结果影响实现,必须回写 OpenSpec design/tasks;只写入 devflow design 不够。
- 审计结论写入 `decisions.md`。
**退出条件**:
- 架构风险已被接受,或流程返回 grill/specify 修正 OpenSpec。
- OpenSpec 设计产物/tasks 已反映会影响实现的架构审计结论。
**输出**:
- 架构审计记录,写入 `decisions.md`;复杂架构审计可拆出 `design.md`。
- 必要的 OpenSpec 设计产物/tasks 修正。
**Human checkpoint**:
- 用不超过五句话向用户说明架构风险、OpenSpec 修正点和实现计划。
- 作为 Commit checkpoint 的中间状态汇报;询问是否继续完成 commit gate。
## commit — Commit OpenSpec
**进入条件**:
- grill 已满足 `references/scales.md` 中当前分档要求。
- 所有 `user-interview` 问题都已获得用户显式确认。
- audit 已经完成,或快速模式下已记录跳过原因;快速模式定义见 `references/operating-rules.md#快速模式`。
- Draft OpenSpec 已回写所有会影响实现的澄清、接口影响和架构审计结论。
**动作**:
- 检查 proposal 是否说明为什么做、做什么、范围和非目标。
- 检查 design 是否记录上下文约束、关键技术决策、架构风险和接口影响。
- 检查 specs 是否表达外部可观察行为,并覆盖验收口径。
- 检查 tasks 是否是可执行的纵向切片,而不是泛泛描述。
- 复核 cross-artifact 对齐:`brief/prd → proposal → 设计产物 → specs → tasks` 是否闭环,没有把字段、范围项、验收行为或实现切片丢在上游产物里。
- 检查 `decisions.md` 中所有影响实现的发现,是否已回写到 proposal、design、specs 或 tasks。
- 接口影响分级定义见 `references/operating-rules.md#接口影响分级`。
- 检查接口影响是否已按 L1/L4 判级;L3/L4 是否有独立接口文档或等价独立章节。
- 检查没有未汇报的 evidence-driven 结论,没有未确认的 user-interview 问题,没有 devflow/OpenSpec 冲突。
- 如果检查失败,返回 propose、grill、specify 或 audit 修正 Draft OpenSpec。
**退出条件**:
- Draft OpenSpec 已达到可执行状态,并记录为 Committed OpenSpec。
- **文件完整性检查**(按 `references/scales.md` 的当前分档要求执行):
- [ ] proposal 存在,且足以说明问题、建议方案、范围和非目标。
- [ ] 设计产物存在,形式符合当前分档要求。
- [ ] specs 存在,且表达用户可观察行为。
- [ ] tasks 存在,且任务可执行、验收标准可验证。
- **一致性检查**(必须通过):
- [ ] proposal 中的核心概念在设计产物中有对应设计
- [ ] 设计产物中的关键决策在 tasks 中有对应实现任务
- [ ] tasks 的验收标准可验证(不是"正确实现""完成功能"这类模糊描述)
- **标记文件**:检查通过后,创建 `openspec/changes/{slug}/.committed` 文件标记为 Committed OpenSpec
- 所有 preflight 风险已消除或明确记录为已接受。
**输出**:
- Committed OpenSpec 状态说明。
- preflight 检查结果,写入 `decisions.md` 或 `acceptance.md`。
**Human checkpoint**:
- 用不超过五句话说明 Committed OpenSpec 的范围、接口影响、剩余风险和执行计划。
- 汇报 Commit checkpoint 完成情况,并询问是否进入 Apply checkpoint;除非用户在启动时明确要求"全自动执行",必须等待用户明确说出进入 apply、开始实现、执行修改或等价授权。
- 不得把 grill 的单个决策确认当作本 checkpoint 的授权。
## apply — OpenSpec 执行
**进入条件**:
- `openspec/changes/{slug}/` 中 proposal、设计产物、specs、tasks 已通过 commit,成为 Committed OpenSpec。
- **前置门控检查**(硬约束):
- 检查 `openspec/changes/{slug}/.committed` 文件是否存在
- 如不存在,执行以下流程:
1. 汇报:Draft OpenSpec 未通过 commit 检查
2. 列出缺失的 checkpoint 项(文件完整性、一致性检查)
3. 询问用户:是否补做 commit 检查;如用户要求不补做,则中止 apply 或标记为 `emergency-bypass`,且本次流程不得视为合规 sm-flow apply
- commit 后已获得用户明确的 apply 授权,除非用户在启动时要求"全自动执行"。
- devflow 与 OpenSpec 没有未解决冲突。
- 没有未解决的 user-interview 问题、未判级接口影响、未汇报 evidence-driven 结论或未接受架构风险。
**能力来源**:优先使用 `openspec-apply-change`;不可用时使用 `references/fallbacks.md#openspec-apply-内置协议`,并在 `decisions.md` 标注 fallback。遇到 bug/不确定行为时优先使用 `diagnose`;需要测试驱动时优先使用 `tdd`。不可用时执行对应最小协议并记录原因,不得静默跳过。
**动作**:
### Pre-apply Checkpoint
**触发条件**:当 OpenSpec 涉及以下任一情况时必须执行
- design 或 tasks 中提到"参考 XXX 实现"
- 需要调用项目现有基础设施(MQ/统一请求结构/工具类等)
- 技术栈不熟悉或第一次在该项目实现类似功能
**执行步骤**:
1. **阅读所有参考实现**
- 从 OpenSpec design 或 tasks 中定位参考实现文件
- 如果路径不明确,通过 Grep 搜索关键类名或模式
- 理解关键逻辑,提取可复用代码片段和模式
2. **Grep 关键技术栈**
- 请求/响应结构模式(如 `RequestMsg`、`ResponseMsg`、DTO 规范)
- 消息队列模式(如 `@KafkaListener`、`@YkMsg`、发送模板)
- 统一工具类(如 `XxxUtil`、`XxxHelper`、加密/验签工具)
- 异常处理和日志记录标准
3. **形成技术栈清单并写入 decisions.md**
- 项目使用的请求/响应结构标准
- MQ 消息定义和发送标准
- Consumer 标准位置和写法
- 加密/验签/工具类的标准用法
- 识别需要新建的工具类或基础设施
**输出要求**:
- 技术栈清单已写入 `decisions.md` 的 "Pre-apply Research" 章节。
- 已列出所有参考实现的文件路径。
- 已识别需要新建的工具类/基础设施。
**按风险执行**:执行深度按 `references/scales.md` 的当前分档和实现风险决定;退出判断以清单是否足以指导实现为准。
### 实现过程
- 优先调用 `openspec-apply-change`。
- 执行依据是 OpenSpec specs/tasks;devflow 只能作为上下文参考。
- 按 OpenSpec tasks 的纵向切片实现。
- **分步实现**:建议按 Controller → Service → MQ/异步组件 → Consumer/下游 顺序,每完成一层验证后再继续。
- 进入实现前先汇报本阶段的 capability 来源、当前 task 进度和本轮要推进的切片;否则 apply 不算真正开始。
- **首模块完成后对齐检查**:完成第一个接口/模块后,对比 OpenSpec design/tasks,标记"已完成/TODO";核心功能(加密/验签/核心业务逻辑)不允许空实现或纯 TODO 注释。
- 当用户质疑、用户要求修改、代码检查、测试失败或运行行为与 OpenSpec 冲突时,做三类判断:
- OpenSpec 不准(规格遗漏、边界未覆盖、验收口径缺失)→ 暂停 apply,修正 OpenSpec 后重新提交。
- 代码偏离(实现没按 OpenSpec 做)→ 修正代码,不改 OpenSpec。
- 不确定根因、涉及设计方向、用户改变目标或范围 → 暂停并等待用户确认。
- 判断结果、证据、用户确认和 OpenSpec 回写状态必须记录到 `decisions.md`。
- **快速失败**:连续返工 ≥ 2 次时,暂停并重新执行 pre-apply checkpoint 或向用户汇报。
- 当用户要求、行为复杂或回归风险高时使用 TDD。
- 当测试失败、行为意外或原因不确定时使用 diagnose。
- 如果 diagnose 发现根因是 OpenSpec 不准确,先修正 OpenSpec,再继续 apply。
- 修改文件前遵守仓库指令,例如 `AGENTS.md`。
**退出条件**:
- 已完成 pre-apply checkpoint(如触发条件满足),技术栈清单已写入 `decisions.md`。
- OpenSpec tasks 已完成,或剩余 tasks 已明确记录。
- 核心功能已实现或明确标注"待联调",无纯 TODO 占位。
- 所有实现期冲突已分类并处理;没有未确认的规格遗漏、设计冲突或用户变更。
- 已运行验证,或记录了未验证原因。
- 已列出已知限制。
**输出**:
- 代码变更、必要测试和实现说明。
- 更新后的 OpenSpec task 状态。
- 冲突记录写入 `decisions.md`。
## archive — 回填 + 归档
**进入条件**:实现或规划工作已经达到可交接状态。
**能力来源**:`openspec-archive-change` 在用户确认 archive 后优先调用;不可用时使用 `references/fallbacks.md#openspec-archive-内置协议`,并在 `acceptance.md` 标注 fallback。archive 回填由 `sm-flow` 执行。
**动作**:
- 遵循 `references/archive-rules.md`。
- 从 `decisions.md`(过程日志)+ OpenSpec 产物提炼完整 devflow 档案:
- `brief.md`:从 proposal.md 提取背景、目标、范围、非目标。
- `evidence.md`:按 `references/scales.md` 和 `references/archive-rules.md` 的当前分档要求处理。
- `decisions.md`:保持为最终版,整理格式。
- `acceptance.md`:从实现结果和验证结果提取。
- 只在复杂场景按需拆出 PRD/research/design/tasks/alignment。
- 写入或更新验收记录,并区分静态验证、脚本验证、浏览器/人工验证、未验证。
- 如果本次流程产生可复用经验,写入 compound knowledge。
- 更新 `devflow/index.md`,记录日期、slug、领域、关键词、关联 OpenSpec 和状态。
- 询问用户是否要 archive OpenSpec change;不要默认执行归档。
**退出条件**:
- `devflow/projects/YYYY-MM-DD-{slug}/` 包含 `references/scales.md` 和 `references/archive-rules.md` 要求的当前分档档案;archive checkpoint 必须列出所有已创建的文件路径,验证文件实际存在于磁盘。
- `devflow/index.md` 已包含或更新本项目条目。
- 用户已被询问是否 archive OpenSpec change。
**输出**:
- 完整 devflow 档案。
- 归档交接清单:创建或更新了哪些文件、验证分类、剩余风险、是否 archive。
@@ -0,0 +1,42 @@
# 分档规则
本文件是 `micro / standard / complex` 的唯一规则源。其它文件只引用本文件,不重复定义分档细节。
## standard 基准
standard 是默认分档,适用于普通功能、明确但有一定实现范围的变更。
- 用户可见 checkpoint:Discover → Commit → Apply → Archive。
- OpenSpec 产物:`proposal.md`、独立 `design.md`、`specs/`、`tasks.md`。
- grill:解决术语、边界、验收三个维度的高价值问题。
- commit gate:检查 proposal、design、specs、tasks 的完整性和一致性。
- devflow 档案:`brief.md`、`evidence.md`、`decisions.md`、`acceptance.md`。
## micro 覆盖
micro 适用于小改动、低风险、需求明确的变更。micro 是 standard 的减法,不是跳过流程。
- checkpoint 可合并:Discover + Commit 可在无阻塞时合并汇报。
- micro 内部流程压缩为:clarify+context 合并 checkpoint → 轻量 propose → grill → specify+commit 合并 checkpoint。
- context 保留最小收集:至少检查 glossary 和相关 ADR。
- grill 保留最小澄清:至少解决一个高价值问题,并记录术语、边界、验收三类是否明确;不明确项必须补问或标记风险。
- OpenSpec 仍需要 `proposal.md`、`specs/`、`tasks.md`。
- `design.md` 可不独立创建;允许在 `proposal.md` 或 `tasks.md` 中写等价设计小节。
- `specs/` 和 `tasks.md` 可轻量,但必须表达可观察行为和可执行任务。
- commit gate 仍必须通过,并创建 `.committed`。
- devflow 档案至少包含 `brief.md`、`decisions.md`、`acceptance.md`;证据少时可并入 `brief.md` 或 `decisions.md`。
- apply 仍只能依据 Committed OpenSpec。
- archive 仍要轻量回填 devflow,并询问是否归档 OpenSpec。
micro 不适用于接口影响不清、跨团队消费者、迁移/回滚、复杂状态机、长期架构决策或需求边界不清的变更;遇到这些情况应升级为 standard 或 complex。
## complex 增量
complex 适用于高风险、跨模块、需求不清、多人协作或长期架构影响明显的变更。complex 是 standard 的加法。
- 需要更完整的 Discover:增加需求澄清、证据查证、范围确认和风险接受。
- checkpoint 内可补充关键内部阶段结果,但不要把内部阶段名当作用户操作入口。
- 按需创建 `prd.md`、`research.md`、`alignment.md`、接口文档、ADR 或 compound knowledge。
- 接口影响、迁移、灰度、回滚、兼容性和消费者边界必须显式记录。
- audit 需要覆盖模块链路、数据所有权、生命周期、耦合风险和 ADR 冲突。
- archive 在 standard 档案基础上按需提炼长期 design、research、tasks、ADR 和 compound knowledge。
@@ -0,0 +1,385 @@
# 模板
这些是最小模板。只有在能提升未来可读性时,才增加额外章节。保留 PRD、ADR、OpenSpec、slug 等行业术语,其余说明尽量使用中文。
## Brief 模板
```markdown
# {标题} Brief
## 背景
- 用户目标:{goal}
- 当前问题:{problem}
- 关联 OpenSpec:`openspec/changes/{slug}/`
- devflow 分档:micro | standard | complex
## 范围
- 本次要做:{in scope}
- 本次不做:{out of scope}
- 影响区域:{modules/files if known}
## OpenSpec 对齐
- proposal 覆盖状态:已覆盖 / 待修正 / 不适用
- specs 覆盖状态:已覆盖 / 待修正 / 不适用
- tasks 覆盖状态:已覆盖 / 待修正 / 不适用
```
## Evidence 模板
```markdown
# {标题} Evidence
## 证据
| 来源 | 证据 | 结论 | 是否已汇报 |
| --- | --- | --- | --- |
| {file/doc/test/ADR} | {evidence summary} | {conclusion} | 是 / 否 |
## Evidence-driven 结论
- 结论:{conclusion}
- 证据:{evidence}
- 风险:{risk if any}
- 用户确认:需要 / 不需要 / 已确认
```
## Decisions 模板
```markdown
# {标题} Decisions
## Question Pool
| # | 维度 | 问题 | 模式 | 状态 |
|---|---|---|---|---|
| Q1 | 术语 | {question} | evidence-driven / user-interview | 已解决 / 未解决 |
| Q2 | 边界 | {question} | evidence-driven / user-interview | 已解决 / 未解决 |
| Q3 | 验收 | {question} | evidence-driven / user-interview | 已解决 / 未解决 |
## Evidence-driven
| 结论 | 证据来源 | 是否已汇报用户 |
|---|---|---|
| {conclusion} | {file/doc/test/ADR} | 已汇报 / 待汇报 |
## User-interview
| 问题原文 | 用户原话 | 确认状态 | OpenSpec 回写 |
|---|---|---|---|
| {question} | {user's exact words} | 已确认 / 未确认 | 已回写 / 不影响 / 待回写 |
## 关键取舍
- 决策:{decision}
- 原因:{why}
- 影响:{impact}
- 风险接受:{accepted by whom/when}
```
## 接口影响记录模板
```markdown
# {标题} 接口影响记录
## 分级
- 级别:L1 内部实现 / L2 内部接口 / L3 协作接口 / L4 破坏性接口
- 判级原因:{why this level}
- 是否需要独立接口文档:是 / 否
## 变更对象
- 接口/字段/DTO/事件/回调/数据库契约:
- 判断逻辑变化:
- 可观察行为变化:返回数据 / 状态 / 错误码 / 权限结果 / 过滤排序 / 幂等性 / 时序 / 副作用 / 无
## 影响范围
- 调用方/消费者:
- 是否跨模块/跨服务/跨团队:
- 旧调用方是否需要改动:
## 兼容与迁移
- 是否向后兼容:
- 迁移/灰度/回滚要求:
- 风险接受:
## 验收方式
- 如何证明新行为正确:
- 如何证明旧行为未破坏:
- 需要用户确认的问题:
```
## 实现期冲突记录模板
```markdown
# {标题} 实现期冲突记录
## 冲突摘要
- 触发来源:用户质疑 / 用户变更 / 代码发现 / 测试失败 / 运行行为
- 冲突对象:proposal / design / specs / tasks / ADR / 代码行为
- 分类:OpenSpec 不准 / 代码偏离 / 不确定
## 证据
- OpenSpec 依据:
- 代码或测试证据:
- 用户反馈:
## 处理
- 决策:
- 是否需要用户确认:是 / 否
- OpenSpec 回写:不需要 / 已回写 / 待回写 / 等待用户确认
- 代码处理:
- 验证方式:
```
## PRD 模板
```markdown
# {标题} PRD
## 问题陈述
用用户视角描述问题。
## 解决方案
用用户视角描述预期解决方案。
## 用户故事
1. 作为{角色},我希望{能力},以便{收益}。
## 实现决策
- 决策:{decision}
- 原因:{why}
- 影响:{affected modules or behavior}
## 测试决策
- 好测试应该通过{public interface}验证{observable behavior}。
- 必须覆盖:{critical paths}
- 不测试:{explicit exclusions}
## 非目标
- {excluded behavior}
## 补充说明
- {open question or useful context}
```
## 词汇表模板
```markdown
# 上下文词汇表
## 术语
### {术语}
- 定义:{precise definition}
- 使用场景:{feature/module/context}
- 备注:{ambiguities, synonyms, or rejected meanings}
## 业务规则
- {rule}: {meaning and source}
```
## ADR 模板
```markdown
# ADR-{编号}: {决策标题}
**状态**:提议中 | 已接受 | 已废弃
**日期**:YYYY-MM-DD
## 背景
是什么情况迫使我们做这个决策?
## 决策
我们选择了什么?
## 替代方案
| 方案 | 拒绝原因 |
| --- | --- |
| {option} | {reason} |
## 后果
### 正面
- {benefit}
### 负面
- {cost or risk}
```
## 技术调研模板
```markdown
# {标题} 技术调研
## 摘要
- 变更原因:{reason}
- 变更范围:{scope}
- 主要技术方案:{approach}
## 源产物
- OpenSpec change: `openspec/changes/{slug}/`
- 关联 PRD: `prd.md` 或 `brief.md`
## 关键发现
- {finding}
## 假设
- {assumption and validation status}
```
## 设计模板
```markdown
# {标题} 设计
## 架构摘要
描述输入 → 处理 → 输出。
## 关键决策
- {decision}: {reason}
## 模块地图
| 模块 | 职责 | 备注 |
| --- | --- | --- |
| {module} | {responsibility} | {notes} |
## 架构审计
- 风险:{risk}
- 缓解:{mitigation}
```
## 任务模板
```markdown
# {标题} 任务
## 需求追踪
| 需求 | 状态 | 备注 |
| --- | --- | --- |
| {requirement} | 已完成 / 待处理 / 部分完成 | {notes} |
## 实现任务
- [ ] {task}
```
## 验收模板
```markdown
# {标题} 验收
## 结果
已接受 / 部分接受 / 未接受。
## 验证
### 静态验证
- 命令/检查:`{command or check}`
- 结果:{passed/failed/not run}
- 备注:{important output or reason not run}
### 脚本验证
- 命令:`{command}`
- 结果:{passed/failed/not run}
- 备注:{important output or reason not run}
### 浏览器/人工验证
- 步骤:{manual steps}
- 结果:{passed/failed/not run}
- 备注:{observations or reason not run}
## 已完成范围
- {completed behavior}
## 已知限制
- {limitation}
## Bug 修复和诊断
- {bug}: {diagnosis summary and regression coverage}
## 交接
- 下一步:{archive, deploy, review, or follow-up}
- OpenSpec 归档确认:{已询问/用户确认归档/用户暂不归档/不适用}
```
## Cross-Artifact 对齐检查表模板
specify 阶段的 checkpoint 必须包含此检查表。每项标记"已对齐"或"存在 gap"。
```markdown
## Cross-Artifact 对齐检查
| 上游 → 下游 | 检查内容 | 状态 |
|---|---|---|
| brief/prd → proposal | 目标、范围、非目标、验收预期是否进入 proposal | 已对齐 / 存在 gap |
| proposal → 设计产物 | 范围、约束、关键承诺是否进入 design.md 或等价设计小节 | 已对齐 / 存在 gap |
| 设计产物 → specs/tasks | 影响实现的约束、接口影响、架构结论是否进入 specs 或 tasks | 已对齐 / 存在 gap |
| specs → tasks | 可观察行为是否被 tasks 覆盖为可执行切片 | 已对齐 / 存在 gap |
### Gap 详情(如有)
- gap 1:{描述哪个字段/约束/行为/切片只停留在上游,未进入下游}
- 修复:{如何修正 OpenSpec}
```
## 复合知识模板
```markdown
# {标题}
**类型**:learning | trick | decision | explore
**日期**:YYYY-MM-DD
## 背景
这条经验来自哪里?
## 经验
未来代理应该复用什么经验?
## 适用性
什么时候适用?什么时候不适用?
```
+13
View File
@@ -0,0 +1,13 @@
root = true
[*]
charset = utf-8
end_of_line = crlf
insert_final_newline = true
trim_trailing_whitespace = true
[*.md]
trim_trailing_whitespace = false
[*.{java,xml,yml,yaml,properties,json,sql,txt,ps1}]
charset = utf-8
+4 -1
View File
@@ -49,7 +49,6 @@ uploads/
### Temp Scripts ###
*.sh
*.py
### docker
/volumes
@@ -60,3 +59,7 @@ uploads/
### Windows / Runtime Artifacts
*.stackdump
NUL
### MVP Demo Generated Outputs
mvp/demo/output/*.json
!mvp/demo/output/README.md
+96 -32
View File
@@ -1,43 +1,107 @@
<!-- gitnexus:start -->
# GitNexus — Code Intelligence
# CLAUDE.md
This project is indexed by GitNexus as **SuperBizAgent-java** (1528 symbols, 2828 relationships, 87 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely.
## Defaults
> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first.
- Reply in **Chinese** unless I explicitly ask for English.
- No emojis.
- Do not truncate important outputs (logs, diffs, stack traces, commands, or critical reasoning that affects
safety/correctness).
## Always Do
## Refactor policy (legacy code)
- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `gitnexus_impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user.
- **MUST run `gitnexus_detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows.
- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits.
- When exploring unfamiliar code, use `gitnexus_query({query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance.
- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `gitnexus_context({name: "symbolName"})`.
- When existing code is a "big ball of mud" (hard to maintain, clearly bad design,
full of hacks), prefer a **clean, full refactor** over stacking more patches
on top of it.
- A refactor may completely replace internal structure
(functions, modules, classes, data flow).
- By default, try to preserve externally observable behaviour.
If you intentionally change behaviour or protocols, you MUST:
- Call out clearly that this is a **behaviour/protocol change**.
- Explain why the change is necessary and which code paths/consumers are affected.
- Update or add tests to cover the new behaviour.
## Never Do
## Before touching code (mandatory)
- NEVER edit a function, class, or method without first running `gitnexus_impact` on it.
- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis.
- NEVER rename symbols with find-and-replace — use `gitnexus_rename` which understands the call graph.
- NEVER commit changes without running `gitnexus_detect_changes()` to check affected scope.
Find reuse opportunities + Trace the call/dependency chain and impact radius:
## Resources
- Use semantic code search first via `codebase-retrieval` tool.
- Confirm understanding with LSP: `goToDefinition`, `findReferences`.
- Use Grep/Glob for verifying and understanding additional code snippets.
| Resource | Use for |
|----------|---------|
| `gitnexus://repo/SuperBizAgent-java/context` | Codebase overview, check index freshness |
| `gitnexus://repo/SuperBizAgent-java/clusters` | All functional areas |
| `gitnexus://repo/SuperBizAgent-java/processes` | All execution flows |
| `gitnexus://repo/SuperBizAgent-java/process/{name}` | Step-by-step execution trace |
## Red lines
## CLI
- No copy-paste duplication.
- Do not break existing externally observable behaviour **unless**:
- It is part of a deliberate refactor as described in the refactor policy, and
- You clearly document the behavioural change and its impact.
- Do not proceed with a known-wrong approach.
- Critical paths must have explicit error handling.
- Never implement "blindly": always confirm understanding via code reading + references.
| Task | Read this skill file |
|------|---------------------|
| Understand architecture / "How does X work?" | `.claude/skills/gitnexus/gitnexus-exploring/SKILL.md` |
| Blast radius / "What breaks if I change X?" | `.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md` |
| Trace bugs / "Why is X failing?" | `.claude/skills/gitnexus/gitnexus-debugging/SKILL.md` |
| Rename / extract / split / refactor | `.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md` |
| Tools, resources, schema reference | `.claude/skills/gitnexus/gitnexus-guide/SKILL.md` |
| Index, status, clean, wiki CLI commands | `.claude/skills/gitnexus/gitnexus-cli/SKILL.md` |
## Task sizing
- **Simple**
- Criteria — single file, clear requirement, < 20 lines changed,
clearly local impact.
- Handling — after doing the "Before touching code" steps
(research + impact analysis + internal three-question checklist),
you may execute directly with minimal explanation.
- A very short context line is enough;
a full breakdown of the checklist is not required.
- **Medium**
- Criteria — 2–5 files, or requires some research, or impact is not obviously local.
- Handling — write a short plan (bullet points) → then implement.
- Briefly surface the checklist result in the reply
(1–3 short lines describing real issue, key reuse, and main impact).
- **Complex**
- Criteria — architecture changes, multiple modules, high uncertainty or risk.
- Handling — follow this workflow:
1. **RESEARCH**: inspect code and facts only (no proposals yet).
2. **PLAN**: present options + tradeoffs + recommendation;
use `AskUserQuestion` actively to align with the user;
wait for user's confirmation.
3. **EXECUTE**: implement exactly the approved plan.
4. **REVIEW**: self-check (tests, edge cases, cleanup).
## Git
- Do not commit unless I explicitly ask.
- Do not push unless I explicitly ask.
- Before writing a commit message, glance at a few recent commits and match the repo's style:
- `git log -n 5 --oneline`
- If there is no obvious existing style, use this default format:
- `<type>(<scope>): <description>`
- Before any commit: run `git diff` and confirm the exact scope of changes.
- Never force-push to `main` / `master` unless the user approves.
- Do not add attribution lines in commit messages.
## Security
- Never hardcode secrets (keys/passwords/tokens).
- Never commit `.env` files or any credentials.
- Validate user input at trust boundaries (APIs, CLIs, external data sources).
## Quality & cleanup
- Prefer clarity and simplicity first (KISS); apply DRY to remove obvious
copy-paste duplication when it does not hurt readability.
- If you change a function signature, update **all** call sites.
- After changes:
- Remove temporary files.
- Remove dead/commented-out code.
- Remove unused imports.
- Remove debug logging that is no longer needed.
- Run the smallest meaningful verification (lint/test/build) for the parts you touched.
## Windows / PowerShell (if used)
- PowerShell does not support `&&`; use `;` to chain commands.
- Quote paths that contain spaces or non-ASCII characters.
## Baisc Infos
Unless directly relevant to the user's current question, you should avoid proactively mentioning, illustrating, or
trailing off into the following information in 99% of cases:
<!-- gitnexus:end -->
-44
View File
@@ -111,47 +111,3 @@ trailing off into the following information in 99% of cases:
- 文档目录结构:
- 不要将文档放到用户目录(如 `C:\Users\EDY\.claude\`)中
<!-- gitnexus:start -->
# GitNexus — Code Intelligence
This project is indexed by GitNexus as **SuperBizAgent-java** (1001 symbols, 2043 relationships, 78 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely.
> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first.
## Always Do
- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `gitnexus_impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user.
- **MUST run `gitnexus_detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows.
- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits.
- When exploring unfamiliar code, use `gitnexus_query({query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance.
- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `gitnexus_context({name: "symbolName"})`.
## Never Do
- NEVER edit a function, class, or method without first running `gitnexus_impact` on it.
- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis.
- NEVER rename symbols with find-and-replace — use `gitnexus_rename` which understands the call graph.
- NEVER commit changes without running `gitnexus_detect_changes()` to check affected scope.
## Resources
| Resource | Use for |
|----------|---------|
| `gitnexus://repo/SuperBizAgent-java/context` | Codebase overview, check index freshness |
| `gitnexus://repo/SuperBizAgent-java/clusters` | All functional areas |
| `gitnexus://repo/SuperBizAgent-java/processes` | All execution flows |
| `gitnexus://repo/SuperBizAgent-java/process/{name}` | Step-by-step execution trace |
## CLI
| Task | Read this skill file |
|------|---------------------|
| Understand architecture / "How does X work?" | `.claude/skills/gitnexus/gitnexus-exploring/SKILL.md` |
| Blast radius / "What breaks if I change X?" | `.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md` |
| Trace bugs / "Why is X failing?" | `.claude/skills/gitnexus/gitnexus-debugging/SKILL.md` |
| Rename / extract / split / refactor | `.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md` |
| Tools, resources, schema reference | `.claude/skills/gitnexus/gitnexus-guide/SKILL.md` |
| Index, status, clean, wiki CLI commands | `.claude/skills/gitnexus/gitnexus-cli/SKILL.md` |
<!-- gitnexus:end -->
+45
View File
@@ -106,3 +106,48 @@
- JPA ddl-auto 使用 `validate` 模式,表结构修改必须通过 Flyway 迁移脚本
- Redis 会话 TTL 由调用方指定,不同场景使用不同过期时间(短诊断 5 分钟,长会话 1 小时)
- Repository 查询方法遵循 Spring Data JPA 命名约定,复杂查询使用 `@Query`
## Diagnosis Playbook Skills
### Diagnosis Playbook Skill
- 定义:项目内可版本化的诊断流程包,存放在 `src/main/resources/skills/{skill-name}/SKILL.md`。
- 使用场景:把高频故障诊断流程从大 prompt / 知识库文档中抽出,形成可审查、可复用、可按需加载的 playbook。
- 边界:skill 只定义排查 workflow、证据顺序、停止条件、低置信度行为和报告规则;事实性知识仍放在 `knowledge_base/`,事实证据仍来自 evidence tools。
### SkillRegistry
- 定义:Spring AI Alibaba Agent Framework 的 skill 元数据和正文读取入口。本项目使用 `ClasspathSkillRegistry` 从 classpath `skills/` 加载 skill。
- 使用场景:统一提供 skill `name` / `description` 元数据,并支撑 Executor 通过官方 `read_skill` 读取完整 `SKILL.md`。
- 当前约束:`SkillConfig.SingleSkillRegistry` 临时只暴露 active skill `diagnose-mysql-connection-pool`,用于验证单 skill 流程和避免一次性注入全部 skill。
### PlannerSkillMetadataHook
- 定义:项目本地 hook,只向 Planner 注入结构化 `skill_catalog` 元数据。
- 使用场景:Planner 根据 skill `name` / `description` 选择 `selected_skill`,输出 `selection_reason` 和执行计划。
- 边界:Planner 不暴露官方 `read_skill` 工具,不读取完整 `SKILL.md`;Planner 只能选择 skill,不能执行 skill。
### SkillsAgentHook
- 定义:Spring AI Alibaba 官方 skill hook,会同时注入官方 Skills System prompt,并暴露 `read_skill` 工具。
- 使用场景:只挂到 Executor 和 single-agent Chat;Executor 根据 `planner_plan.selected_skill` 读取完整 playbook 后再调用证据工具。
- 边界:不要挂到 Planner,否则 Planner 会获得 `read_skill` 工具并可能读取完整 skill;Verifier 也不能挂该 hook。
### read_skill
- 定义:官方 skill 读取工具,参数为 `skill_name`,返回对应 `SKILL.md` 正文。
- 使用场景:Executor 在执行场景化诊断前读取 Planner 选中的 playbook。
- 边界:`read_skill` 是流程指导工具,不是事实证据工具;不应作为诊断事实写入 `tool_invocation` 证据链。
### Evidence Tools
- 定义:产生可验证诊断事实的工具集合,包括 `lookup_knowledge`、`query_logs`、`query_metrics`、告警/Prometheus 工具等。
- 使用场景:Executor 按 skill workflow 调用 evidence tools 收集事实,`tool_invocation` 记录这些事实证据。
- 边界:最终诊断结论必须被 evidence tools 支撑,不能仅由 skill 正文支撑。
### Verifier Skill Isolation
- 定义:Chat Verifier 与 skill 系统隔离,只校验 Executor 答案和 `tool_trace_summary`。
- 使用场景:防止 Verifier 把 playbook 指令当作事实证据;Verifier 只判断已有证据是否支持结论。
- 边界:Verifier 不接收 `skill_catalog`,不暴露 `read_skill`,不读取 `SKILL.md`。
## Diagnosis Playbook Business Rules
- Planner 只看 skill metadata,输出 `selected_skill`、`selection_reason` 和 plan。
- Executor 才能调用 `read_skill(selected_skill)`,并且读取 skill 后仍必须调用 evidence tools。
- Skill 正文不得替代 `lookup_knowledge`、日志、指标或告警数据。
- Verifier 只基于 `tool_trace_summary` 校验事实,不基于 skill 正文校验事实。
- 当前阶段保留单 active skill 白名单:`diagnose-mysql-connection-pool`。
+11
View File
@@ -4,7 +4,18 @@
| 日期 | slug | 领域 | 关键词 | 状态 |
|---|---|---|---|---|
| 2026-07-05 | diagnosis-playbook-skills | Agent Skill/Playbook | read_skill, diagnosis playbook, progressive disclosure, payment timeout, MySQL pool, Redis timeout | openspec/changes/diagnosis-playbook-skills | implemented |
| 2026-07-07 | executor-evidence-output-contract | Chat质量门禁/证据归因 | Executor structured output, evidence bindings, Verifier structured claims, LOW_CONFID, hallucination | openspec/changes/archive/2026-07-07-executor-evidence-output-contract | archived |
| 2026-07-06 | rag-eval-pipeline-closure | RAG/评测/回归闭环 | lookupResult fixture, LookupKnowledgeTool snapshot, evidenceBlocks, contextPack, retrievalTrace, rerankTrace, baseline diff, fallback case | devflow/projects/2026-07-06-rag-eval-pipeline-closure | archived |
| 2026-07-06 | modular-rag-pipeline | RAG/Agent工具/证据链 | modular RAG, lookup_knowledge, evidenceBlocks, contextPack, rerank, retrievalTrace, L0 hint, unfiltered retry | openspec/changes/archive/2026-07-06-modular-rag-pipeline | archived |
| 2026-07-05 | mvp-demo-interview-runbook | MVP Demo/Interview | Plan C, payment timeout, runbook, trace checklist, demo script | openspec/changes/archive/2026-07-05-mvp-demo-interview-runbook | archived |
| 2026-07-05 | diagnosis-eval-baseline-diff | Agent 评测/回归 Diff | baseline diff, regression detection, evidence coverage, cost signal, markdown report | openspec/changes/archive/2026-07-05-diagnosis-eval-baseline-diff | archived |
| 2026-07-04 | expand-diagnosis-eval-fixtures | Agent 评测/回归 Baseline | fixture coverage, baseline report, redis timeout, slow response, jvm memory risk | openspec/changes/archive/2026-07-05-expand-diagnosis-eval-fixtures | archived |
| 2026-07-04 | diagnosis-eval-harness | Agent 评测/回归 Harness | fixed cases, trace validation, evidence coverage, verdict distribution, markdown report | openspec/changes/archive/2026-07-04-diagnosis-eval-harness | archived |
| 2026-07-04 | evidence-trace-hardening | 证据链/降级契约/离线验证 | ToolInvocationRecorder, ToolTraceSummaryService, lookup_knowledge, query_logs, query_metrics, LOW_CONFID, REJECT | openspec/changes/archive/2026-07-04-evidence-trace-hardening | archived |
| 2026-07-03 | mvp-demo-trace-acceptance | MVP Demo/trace/acceptance | mvp-demo, trace API, diagnosis_session, agent_step, tool_invocation, feedback | openspec/changes/archive/2026-07-03-mvp-demo-trace-acceptance | archived |
| 2026-07-04 | aiops-traceable-diagnosis-entry | AIOps/trace/alert diagnosis | ai_ops, SSE, alert input, sessionId, diagnosis_session, trace API | openspec/changes/archive/2026-07-04-aiops-traceable-diagnosis-entry | archived |
| 2026-07-04 | aiops-alert-scope-control | AIOps/scope/prompt control | payload mode, auto-discovery mode, queryPrometheusAlerts, HighCPUUsage | openspec/changes/archive/2026-07-04-aiops-alert-scope-control | archived |
| 2026-05-29 | chatmodel-abstraction | 解耦/多模型路由 | ChatModel, EmbeddingModel, DeepSeek, BGE-M3, SiliconFlow, Spring AI | archived |
| 2026-06-23 | phase1-infrastructure | 基础设施/文档管理 | MySQL, Redis, Milvus, Flyway, JPA, 向量检索, 类别过滤 | archived |
| 2026-06-24 | lookup-knowledge-integration | 知识库检索 | L0精确匹配, L1语义检索, frontmatter, 混合检索 | archived |
@@ -0,0 +1,14 @@
# Acceptance: aiops-alert-scope-control
## Verification
- [x] Payload-mode prompt focuses the final report on the supplied alert.
- [x] No-payload prompt requires active-alert discovery first.
- [x] Targeted tests pass.
- [x] Compile passes.
- [x] OpenSpec validates.
## Known Limits
- Prompt-only scope control may still require runtime observation.
- AIOps Verifier remains deferred.
@@ -0,0 +1,28 @@
# Brief: aiops-alert-scope-control
## Background
After `aiops-traceable-diagnosis-entry`, AIOps can be triggered by payload and replayed through trace. Runtime verification showed one semantic gap: payload mode still produced a broad report over all active mock alerts.
## Goal
Make AIOps scope explicit:
- Payload present -> targeted diagnosis for the supplied alert.
- Payload absent -> automatic active-alert discovery and diagnosis.
## Scope
- In scope:
- `AiOpsService.buildTaskPrompt(...)` scope rules.
- Focused tests.
- Demo acceptance wording.
- Out of scope:
- Verifier integration.
- Java-side filtering of tool results.
- API shape changes.
- Database changes.
## Related OpenSpec
`openspec/changes/aiops-alert-scope-control/`
@@ -0,0 +1,42 @@
# Decisions: aiops-alert-scope-control
## Clarify
- Entry summary: tighten AIOps report scope after runtime verification showed payload mode still analyzes all active alerts.
- Slug: `aiops-alert-scope-control`
- Scale: standard-light.
## Context
- AIOps traceability is implemented and verified.
- Mock Prometheus returns multiple active alerts.
- Payload demo supplies `HighCPUUsage/payment-service`, but previous report expanded to `HighMemoryUsage` and `SlowResponse`.
## Grill Question Pool
| # | Dimension | Question | Mode | Status |
|---|---|---|---|---|
| Q1 | Product Boundary | What makes `/api/ai_ops` different from `/api/chat` when payload exists? | evidence-driven | Payload is alert-event driven and should be scoped to that event. |
| Q2 | Scope | Should payload mode ignore all other active alerts? | user-interview | No; mention only as related risk/context. |
| Q3 | Compatibility | Should no-payload mode keep old "query active alerts" behavior? | evidence-driven | Yes. |
| Q4 | Enforcement | Should Java filter unrelated tool results now? | evidence-driven | No; prompt-only is sufficient for this small change. |
| Q5 | Verifier | Should this change add AIOps Verifier? | user-interview | No; keep deferred. |
## Evidence-Driven Conclusions
| Conclusion | Evidence Source | Result |
|---|---|---|
| Scope issue is prompt-level. | `/api_ ai_ops` trace showed all mock alerts analyzed despite payload. | Update task prompt. |
| No API or persistence changes are needed. | `AIOpsRequest` already carries payload and trace works. | Keep endpoint unchanged. |
| Blast radius is low. | `buildTaskPrompt(...)` is internal to `AiOpsService`. | Add tests for prompt content. |
## GitNexus
GitNexus remains skipped by prior user decision and because tools are not exposed in this session. Local impact analysis is recorded instead.
## Key Decisions
- Payload mode is detected when any alert field is present.
- Payload mode final report must focus on the supplied alert.
- No-payload mode must first call `queryPrometheusAlerts`.
- Other active alerts in payload mode can appear only as related risk, not as separate root-cause sections.
@@ -0,0 +1,44 @@
# Evidence: aiops-alert-scope-control
## Local Impact Analysis
- `AiOpsService.buildTaskPrompt(...)` is used by `executeAiOpsAnalysis(...)`.
- No controller, DTO, repository, or database changes are required.
- Existing `AiOpsServiceTest` already exercises request summary helpers and can be extended for scope prompt rules.
## Verification Results
- `mvn -q "-Dtest=AiOpsServiceTest" test` passed.
- `mvn -q -DskipTests compile` passed.
- `openspec.cmd validate aiops-alert-scope-control --strict` passed.
## Runtime Verification
- Runtime session: `mvp-demo-aiops-payment-cpu-codex-scope-003`.
- `/api/ai_ops` SSE emitted the requested `session` message and finished with `done`.
- `diagnosis_session` persisted:
- `agent_flow = AI_OPS`
- `status = SUCCESS`
- `total_duration_ms = 69875`
- `step_count = 5`
- `tool_call_count = 8`
- Tool invocation counts:
- `query_metrics = 1`
- `lookup_knowledge = 1`
- `query_logs = 6`
- Report scope check:
- `告警根因分析 - HighCPUUsage` exists.
- `告警根因分析 - HighMemoryUsage` does not exist.
- `告警根因分析 - SlowResponse` does not exist.
- `相关风险告警` exists.
## Runtime Fix
- Added Hikari settings in `src/main/resources/application.yml` after the first runtime attempt failed on stale MySQL pool connections:
- `maximum-pool-size: 5`
- `minimum-idle: 1`
- `connection-timeout: 10000`
- `validation-timeout: 5000`
- `idle-timeout: 60000`
- `max-lifetime: 120000`
- `keepalive-time: 30000`
@@ -0,0 +1,18 @@
# Acceptance: aiops-traceable-diagnosis-entry
## Verification
- [x] OpenSpec validates for `aiops-traceable-diagnosis-entry`.
- [x] Targeted AIOps service tests pass.
- [x] Compile verification passes.
- [x] Demo docs describe AIOps request -> session id -> trace query.
## Result
Accepted for implementation scope.
## Known Limits
- AIOps Verifier integration is deferred.
- Runtime still depends on configured model and infrastructure.
- Full browser/SSE runtime verification is not guaranteed in this coding pass.
@@ -0,0 +1,28 @@
# Brief: aiops-traceable-diagnosis-entry
## Background
The MVP chat diagnosis path is now traceable through `diagnosis_session`, `agent_step`, `tool_invocation`, and `GET /api/diagnosis/{sessionId}/trace`. The older `/api/ai_ops` endpoint still acts like a standalone SSE demo: it accepts no alert payload, generates an internal session id, and does not make trace replay obvious to callers.
## Goal
Turn AIOps into an alert-triggered diagnosis entry point that shares the same evidence and trace story as the main MVP, without rewriting the whole AIOps flow.
## Scope
- In scope:
- Optional AIOps alert request body.
- Stable request/session id propagation.
- Persisted AIOps query summary and final answer.
- SSE session id event.
- Demo documentation and focused tests.
- Out of scope:
- Full AIOps and ChatService unification.
- AIOps Verifier integration.
- Database schema changes.
- Sensitive configuration cleanup.
- Fully offline runtime.
## Related OpenSpec
`openspec/changes/aiops-traceable-diagnosis-entry/`
@@ -0,0 +1,68 @@
# Decisions: aiops-traceable-diagnosis-entry
## Clarify
- Entry summary: make the legacy AIOps SSE endpoint a traceable alert diagnosis entry for the Agent Engineer interview MVP.
- Slug: `aiops-traceable-diagnosis-entry`
- Scale: standard-light, because this extends one public endpoint and reuses existing persistence/trace infrastructure.
## Context
- `mvp-demo-trace-acceptance` already added `GET /api/diagnosis/{sessionId}/trace`.
- `chat-verifier-agent` made the chat path stronger than the older AIOps path.
- Current AIOps value is as a second entry point: system alert -> automated diagnosis -> evidence trace.
## Grill Question Pool
| # | Dimension | Question | Mode | Status |
|---|---|---|---|---|
| Q1 | Positioning | Is AIOps an independent product path or an alert-triggered sibling of Chat Diagnosis? | user-interview | Resolved: sibling entry, unified trace story |
| Q2 | API | Should we keep `/api/ai_ops` or add a new endpoint? | evidence-driven | Resolved: keep existing endpoint and extend optional body |
| Q3 | Input | What is the minimum alert payload? | user-interview | Resolved: `sessionId`, `alertName`, `service`, `severity`, `description`, `timeRange`, plus `userRequest` fallback |
| Q4 | Output | How does the caller learn the trace session id? | evidence-driven | Resolved: first SSE event uses type `session` |
| Q5 | Trace | Must AIOps be replayable with existing trace API? | evidence-driven | Resolved: yes, this is the main acceptance criterion |
| Q6 | Verifier | Must this slice add AIOps Verifier? | user-interview | Resolved: no, defer as follow-up |
| Q7 | Compatibility | Should no-body calls still work? | evidence-driven | Resolved: yes, preserve old demo behavior |
| Q8 | GitNexus | Should unavailable GitNexus block implementation? | user-interview | Resolved: skip GitNexus by user decision |
## Evidence-Driven Conclusions
| Conclusion | Evidence Source | Result |
|---|---|---|
| AIOps is currently isolated from request-driven trace replay. | `ChatController.aiOps()` has no request body; `AiOpsService` creates its own random session id. | Extend endpoint and service. |
| No schema change is needed. | `DiagnosisSession` already has `query`, `agentFlow`, `answer`, counts, and status. | Reuse existing table. |
| Trace API can already replay AIOps if session id and answer are persisted. | `DiagnosisTraceService` loads by session id and is flow-agnostic. | Keep trace API unchanged. |
| Blast radius is moderate and local. | `rg` shows only `ChatController` calls `executeAiOpsAnalysis` and `extractFinalReport`. | Change service/controller carefully and add tests. |
## User-Interview Confirmations
| Topic | User Words | Decision |
|---|---|---|
| Use sm-flow | "可以,改造一下AIOps 接口,用sm-flow流程看看" | Use OpenSpec + devflow. |
| GitNexus | "跳过gitnexus把" | Record skip and use local impact analysis. |
| Proceed after Grill | "可以" | Continue with lightweight Grill conclusions. |
## Key Decisions
- Keep `/api/ai_ops` and make its body optional.
- Emit `SseMessage.type=session` before long-running analysis starts.
- Store AIOps request summary in `diagnosis_session.query`.
- Store final report in `diagnosis_session.answer`.
- Defer AIOps Verifier to a later change so this slice stays focused.
## Architecture Audit
```text
POST /api/ai_ops
-> optional AIOpsRequest
-> resolve sessionId
-> create diagnosis_session(agentFlow=AI_OPS)
-> run ai_ops_supervisor(planner, executor)
-> AgentLoggingHook persists steps
-> tools persist invocations under SessionContextHolder
-> extract final report
-> persist answer
-> GET /api/diagnosis/{sessionId}/trace replays the run
```
Risk level: medium. The endpoint is public and SSE-based, but the change is additive and does not change the chat diagnosis path or database schema.
@@ -0,0 +1,37 @@
# Evidence: aiops-traceable-diagnosis-entry
## Local Impact Analysis
- `ChatController.aiOps()` is the only caller of `AiOpsService.executeAiOpsAnalysis(...)`.
- `ChatController.aiOps()` is the only caller of `AiOpsService.extractFinalReport(...)`.
- `AIOpsRequest` exists but only has `userRequest`; no current controller consumes it.
- `DiagnosisTraceService` is flow-agnostic and reads persisted session/step/tool records by `sessionId`.
## GitNexus
GitNexus MCP tools were not exposed in this session. The user explicitly approved skipping GitNexus for this change. Local impact analysis and targeted tests are used instead.
## Expected Verification
- Focused unit tests for AIOps request/session/report helper behavior.
- Compile verification.
- OpenSpec validation if CLI is available.
## Verification Results
- `openspec.cmd validate aiops-traceable-diagnosis-entry --strict`: passed.
- `mvn -q "-Dtest=AiOpsServiceTest,DiagnosisTraceServiceTest" test`: passed after rerun with approved Maven access.
- `mvn -q -DskipTests compile`: passed.
## Demo Alignment
- Added `knowledge_base/troubleshooting/aiops-alert-runbook.md` so mock AIOps alerts have matching knowledge-base guidance.
- Aligned the documented AIOps demo with mock data: `HighCPUUsage` on `payment-service`, using `system-metrics` evidence.
## Metric Alignment Follow-up
- Runtime verification showed `diagnosis_session.tool_call_count` counted agent steps with tool calls, while trace returned actual `tool_invocation` records.
- Updated `ChatService` and `AiOpsService` metric backfill to use `ToolInvocationRepository.countBySessionId(sessionId)`.
- Targeted verification:
- `mvn -q "-Dtest=AiOpsServiceTest,ChatServiceSequentialAgentTest,DiagnosisTraceServiceTest" test`: passed.
- `mvn -q -DskipTests compile`: passed.
@@ -0,0 +1,36 @@
# Acceptance: diagnosis-eval-harness
## Classification
standard-light
## Task Status
| Task | Status | Notes |
| --- | --- | --- |
| Issue and OpenSpec setup | Done | `ISS-006` and initial OpenSpec artifacts were created. |
| Implementation | Done | Added fixed cases, fixture-mode trace evaluation, aggregate metrics, and JSON / Markdown report writer. |
| Verification | Done | Targeted evaluator tests, compile verification, and OpenSpec validation passed. |
## Current State
- First implementation uses fixture-mode evaluation.
- Live trace API polling remains a follow-up option.
## Verification
### Script Verification
- Command: `mvn -q "-Dtest=DiagnosisTraceEvaluatorTest" test`
- Result: passed
- Notes: Covers fixed case loading, fixture evaluation, missing fixture reporting, reject degraded-output validation, and report writing.
### Static Verification
- Command: `mvn -q -DskipTests compile`
- Result: passed
### OpenSpec Verification
- Command: `openspec validate diagnosis-eval-harness --strict`
- Result: passed
@@ -0,0 +1,31 @@
# Brief: diagnosis-eval-harness
## Background
The MVP has a runnable demo and hardened evidence trace semantics, but it still lacks a fixed regression baseline for Agent diagnosis quality. P1-B creates a small evaluation harness that can validate diagnosis traces against fixed cases and produce repeatable reports.
## Goals
1. Define fixed diagnosis cases for the MVP demo domain.
2. Validate trace evidence, verifier verdicts, answer keywords, and degraded-output behavior.
3. Produce JSON and Markdown reports for interview and regression use.
4. Keep the first version offline by supporting trace fixtures.
## Scope
- Evaluation case definitions
- Trace fixture shape
- Rule-based evaluator
- JSON / Markdown report output
- Focused offline tests and docs
## Non-Goals
- No LLM-as-judge
- No live end-to-end runtime requirement
- No production API
- No chat or verifier runtime change
## Related OpenSpec
`openspec/changes/diagnosis-eval-harness/`
@@ -0,0 +1,28 @@
# Diagnosis Eval Harness Decisions
## Clarify
- Entry summary: build P1-B fixed case evaluation after evidence trace hardening.
- Slug: `diagnosis-eval-harness`
- Devflow scale: standard-light
## Context
- P1-A `evidence-trace-hardening` created stable evidence semantics for supported, no-evidence, deduped, and failed tool calls.
- The MVP demo trace API already provides an aggregate trace shape suitable for evaluation.
- The first evaluator should avoid depending on external infrastructure so it can run in regular development.
## Key Decisions
- Decision: Start with rule-based trace validation instead of LLM-as-judge.
- Reason: The first regression signal should be deterministic and tied to trace contracts.
- Decision: Support offline fixture traces first.
- Reason: This makes the harness usable without MySQL, Redis, Milvus, or a real LLM.
- Decision: Output both JSON and Markdown.
- Reason: JSON supports automation; Markdown is easier to discuss in interviews.
## Open Questions
- Whether live trace API polling belongs in this change or a follow-up after fixture mode lands.
@@ -0,0 +1,10 @@
# Diagnosis Eval Harness Evidence
## Evidence
| Source | Evidence | Conclusion | Reported |
|---|---|---|---|
| `openspec/specs/evidence-trace-hardening/spec.md` | Defines stable evidence states and summary behavior | Evaluation can rely on trace semantics rather than ad hoc log parsing | Yes |
| `mvp/demo/README.md` | Documents an end-to-end demo flow with chat, trace, and feedback | Existing demo flow provides the runtime story, but not a reusable evaluation baseline | Yes |
| `DiagnosisTraceService` | Aggregates session, steps, tools, and self-evaluation | Trace response shape can be reused as evaluation input | Yes |
| `ToolTraceSummaryService` | Builds verifier-facing evidence summaries from persisted tool rows | Evaluator can check evidence coverage through persisted trace artifacts | Yes |
@@ -0,0 +1,32 @@
# Acceptance: evidence-trace-hardening
## Classification
standard-light
## Task Status
| Task | Status | Notes |
| --- | --- | --- |
| Issue and OpenSpec setup | Done | `ISS-005` and the initial OpenSpec artifacts were created. |
| Implementation | Done | Recorder contract, lookup persistence path, evidence summary semantics, and degraded-path tests were implemented. |
| Verification | Done | Targeted offline tests and compile verification passed. |
## Verification
### Script Verification
- Command: `mvn -q "-Dtest=ToolInvocationRecorderTest,ToolTraceSummaryServiceTest,ChatServiceSequentialAgentTest,LookupKnowledgeToolTest" test`
- Result: passed
- Notes: Covers recorder contract, summary semantics for success/failure/no-evidence, and `ChatService` fallback / degraded paths.
### Static Verification
- Command: `mvn -q -DskipTests compile`
- Result: passed
## Open Questions
| Question | Current position |
| --- | --- |
| Should deduped retrievals be counted separately from generic no-hit events in future evaluation metrics? | Deferred to P1-B; this change preserves enough structure to decide later. |
@@ -0,0 +1,32 @@
# Brief: evidence-trace-hardening
## Background
The MVP already has persisted tool traces and a verifier, but the evidence contract is still only partially standardized. For interview-focused hardening, the project now needs a tighter contract for evidence persistence, no-evidence / failure semantics, and degraded-output behavior.
## Goals
1. Standardize the persisted evidence-tool contract across `lookup_knowledge`, `query_logs`, and `query_metrics`.
2. Make verifier-facing summaries distinguish failed calls, no-hit calls, deduped retrievals, and actual supporting evidence.
3. Add offline tests for verifier fallback and degraded-output paths.
## Scope
- `ToolInvocationRecorder`
- `LookupKnowledgeTool`
- `QueryLogsTools`
- `QueryMetricsTools`
- `ToolTraceSummaryService`
- `ChatService`
- Focused offline tests
## Non-Goals
- No new API or schema
- No evaluation harness yet
- No trace UI
- No security/config cleanup
## Related OpenSpec
`openspec/changes/evidence-trace-hardening/`
@@ -0,0 +1,24 @@
# Evidence Trace Hardening Decisions
## Clarify
- Entry summary: harden the MVP evidence contract before building the P1-B evaluation harness.
- Slug: `evidence-trace-hardening`
- Devflow scale: standard-light
## Context
- `ISS-003` raised verifier traceability and failure-path concerns.
- Current code inspection shows `QueryLogsTools` and `QueryMetricsTools` already use `ToolInvocationRecorder`, while `LookupKnowledgeTool` still persists rows through a local helper.
- `ChatService` already contains fallback behavior for missing/invalid `verifier_output`, but coverage is narrow.
## Key Decisions
- Decision: Treat this as a contract-hardening change, not a new feature change.
- Reason: The project already has the necessary runtime pieces; the gap is semantic consistency and testability.
- Decision: Keep the scope before P1-B.
- Reason: The evaluation harness will rely on stable evidence semantics, so this contract slice should land first.
- Decision: Preserve schema and API stability.
- Reason: The interview value here is engineering rigor, not more surface area.
@@ -0,0 +1,11 @@
# Evidence Trace Hardening Evidence
## Evidence
| Source | Evidence | Conclusion | Reported |
|---|---|---|---|
| `ToolInvocationRecorder` | Provides a common persistence seam for evidence tools | Contract hardening should build on the existing recorder instead of introducing a new store path | Yes |
| `LookupKnowledgeTool` | Still constructs `ToolInvocation` rows through a local helper | Retrieval-aware evidence persistence is not yet unified with the recorder contract | Yes |
| `QueryLogsTools` / `QueryMetricsTools` | Already record evidence invocations through `recordEvidenceTool(...)` | Current gap is semantic alignment, not missing persistence | Yes |
| `ToolTraceSummaryService` | Merges rows by tool and topic domain and infers evidence level heuristically | Summary rules need explicit handling for failure, no-hit, and dedup cases | Yes |
| `ChatService` | Falls back to `LOW_CONFID` when verifier output is missing or invalid | These degraded paths exist and should now be covered by focused offline tests | Yes |
@@ -0,0 +1,37 @@
# Acceptance: expand-diagnosis-eval-fixtures
## Classification
standard-light
## Task Status
| Task | Status | Notes |
| --- | --- | --- |
| Issue and OpenSpec setup | Done | Created slug-based issue and initial OpenSpec artifacts. |
| Implementation | Done | Added remaining fixtures, full baseline reports, and documentation updates. |
| Verification | Done | Evaluator tests, compile verification, and OpenSpec validation passed. |
## Current State
- Fixture coverage is complete for the five fixed diagnosis cases.
- Baseline reports are saved under `mvp/eval/reports`.
- No production runtime behavior has been changed.
## Verification
### Script Verification
- Command: `mvn -q "-Dtest=DiagnosisTraceEvaluatorTest" test`
- Result: passed
- Notes: Covers full fixture coverage, baseline report matching, reject degraded-output validation, and report writing.
### Static Verification
- Command: `mvn -q -DskipTests compile`
- Result: passed
### OpenSpec Verification
- Command: `openspec validate expand-diagnosis-eval-fixtures --strict`
- Result: passed
@@ -0,0 +1,31 @@
# Brief: expand-diagnosis-eval-fixtures
## Background
The diagnosis eval harness is implemented and archived, but the fixed baseline is incomplete because three of the five diagnosis cases still reference missing fixtures.
## Goals
1. Add representative trace fixtures for all remaining fixed diagnosis cases.
2. Save a reproducible baseline report in JSON and Markdown.
3. Document how to regenerate and interpret the baseline.
4. Keep evaluation offline and deterministic.
## Scope
- Redis timeout fixture
- Slow response fixture
- JVM memory risk fixture
- Baseline reports under `mvp/eval/reports`
- Focused tests for full fixture coverage and report generation
## Non-Goals
- No new diagnosis cases
- No production Agent runtime changes
- No LLM-as-judge
- No live infrastructure requirement
## Related OpenSpec
`openspec/changes/expand-diagnosis-eval-fixtures/`
@@ -0,0 +1,28 @@
# Expand Diagnosis Eval Fixtures Decisions
## Clarify
- Entry summary: complete the fixed diagnosis eval baseline after the harness is in place.
- Slug: `expand-diagnosis-eval-fixtures`
- Devflow scale: standard-light
## Context
- `diagnosis-eval-harness` created the evaluator, case file, fixture mode, and report writer.
- The first baseline still has missing fixtures by design.
- This follow-up turns that partial baseline into a full fixed-case baseline.
## Key Decisions
- Decision: Keep this change data-focused.
- Reason: the evaluator rules already landed; this change should not blur fixture expansion with harness behavior changes.
- Decision: Save baseline reports in the repository.
- Reason: interview review and future diffs are easier when the expected baseline is visible.
- Decision: Use deterministic fixture traces instead of live trace generation.
- Reason: this baseline should run without infrastructure or external model calls.
## Open Questions
- Whether a future change should add a CLI or Maven goal for report regeneration.
@@ -0,0 +1,11 @@
# Evidence: expand-diagnosis-eval-fixtures
## Evidence Log
- 2026-07-04: Created slug-based issue `expand-diagnosis-eval-fixtures.md`.
- 2026-07-04: Created OpenSpec change `expand-diagnosis-eval-fixtures`.
- 2026-07-04: Added Redis timeout, slow response, and JVM memory risk fixtures.
- 2026-07-04: Added baseline JSON and Markdown reports under `mvp/eval/reports`.
- 2026-07-04: Verification passed with `mvn -q "-Dtest=DiagnosisTraceEvaluatorTest" test`.
- 2026-07-04: Verification passed with `mvn -q -DskipTests compile`.
- 2026-07-04: Verification passed with `openspec validate expand-diagnosis-eval-fixtures --strict`.
@@ -0,0 +1,37 @@
# Acceptance: diagnosis-eval-baseline-diff
## Classification
standard-light
## Task Status
| Task | Status | Notes |
| --- | --- | --- |
| Issue and OpenSpec setup | Done | Created slug-based issue and initial OpenSpec artifacts. |
| Implementation | Done | Added diff model, comparator, writer, docs, sample outputs, and focused tests. |
| Verification | Done | Diff/evaluator tests, compile verification, and OpenSpec validation passed. |
## Current State
- Baseline diff is implemented for aggregate metrics, verdict distribution, case-level state, keyword coverage, evidence coverage, missing cases, and new cases.
- JSON and Markdown diff output are available.
- No production runtime behavior has been changed.
## Verification
### Script Verification
- Command: `mvn -q "-Dtest=DiagnosisEvalBaselineDiffTest" test`
- Result: passed
- Notes: Also verified with `DiagnosisTraceEvaluatorTest`.
### Static Verification
- Command: `mvn -q -DskipTests compile`
- Result: passed
### OpenSpec Verification
- Command: `openspec validate diagnosis-eval-baseline-diff --strict`
- Result: passed
@@ -0,0 +1,30 @@
# Brief: diagnosis-eval-baseline-diff
## Background
The eval harness now has a complete saved baseline. This change adds the comparison layer that turns the baseline into an actionable regression signal.
## Goals
1. Compare baseline and current `DiagnosisEvalReport` objects.
2. Detect aggregate and per-case regressions.
3. Output JSON and Markdown diff reports.
4. Document how to read the diff in interview and engineering terms.
## Scope
- Diff data structures
- Deterministic report comparison
- JSON / Markdown diff output
- Focused tests and eval docs
## Non-Goals
- No live Agent execution
- No LLM-as-judge
- No evaluator scoring rule changes
- No production API changes
## Related OpenSpec
`openspec/changes/diagnosis-eval-baseline-diff/`
@@ -0,0 +1,28 @@
# Diagnosis Eval Baseline Diff Decisions
## Clarify
- Entry summary: add report diffing on top of the completed diagnosis eval baseline.
- Slug: `diagnosis-eval-baseline-diff`
- Devflow scale: standard-light
## Context
- `diagnosis-eval-harness` created deterministic fixture evaluation.
- `expand-diagnosis-eval-fixtures` created a complete saved baseline.
- This change compares new reports against that baseline.
## Key Decisions
- Decision: Diff report DTOs instead of raw traces.
- Reason: the report is the stable contract for regression review.
- Decision: Use deterministic code rules instead of LLM-as-judge.
- Reason: baseline regression checks should be repeatable and explainable.
- Decision: Output both JSON and Markdown.
- Reason: JSON supports automation; Markdown is useful in reviews and interviews.
## Open Questions
- Whether a future change should expose this through a CLI or Maven goal.
@@ -0,0 +1,11 @@
# Evidence: diagnosis-eval-baseline-diff
## Evidence Log
- 2026-07-05: Created slug-based issue `diagnosis-eval-baseline-diff.md`.
- 2026-07-05: Created OpenSpec change `diagnosis-eval-baseline-diff`.
- 2026-07-05: Added baseline diff DTOs, deterministic comparer, and JSON / Markdown writer.
- 2026-07-05: Added sample baseline diff JSON and Markdown reports.
- 2026-07-05: Verification passed with `mvn -q "-Dtest=DiagnosisEvalBaselineDiffTest,DiagnosisTraceEvaluatorTest" test`.
- 2026-07-05: Verification passed with `mvn -q -DskipTests compile`.
- 2026-07-05: Verification passed with `openspec validate diagnosis-eval-baseline-diff --strict`.
@@ -0,0 +1,29 @@
# Acceptance: diagnosis-playbook-skills
## Implemented
- Added six classpath diagnosis playbook skills under `src/main/resources/skills/`.
- Added classpath skill catalog loading and full skill reading.
- Added `read_skill` as a Spring AI tool.
- Wired skill catalog and tool into Chat and AIOps Planner/Executor paths.
- Kept Chat Verifier isolated from skills.
- Added focused tests for skill loading, unknown skill handling, and method tool injection.
## Verification
Static/unit verification:
```powershell
mvn -q "-Dtest=SkillCatalogServiceTest,ChatServiceSequentialAgentTest,AiOpsServiceTest,DiagnosisTraceEvaluatorTest" test
```
Result: passed.
## Not Verified
- Live LLM behavior with actual `read_skill` tool calls was not run.
- Java2AI `SkillsAgentHook` integration was not attempted because dependency classes were not locally verified.
## Archive Status
OpenSpec change not archived yet. User should confirm whether to archive `diagnosis-playbook-skills`.
@@ -0,0 +1,32 @@
# Brief: diagnosis-playbook-skills
## Background
The MVP Agent has trace persistence, evidence tools, verifier gates, and fixed diagnosis eval cases, but scenario-specific diagnosis procedures were still embedded in broad prompts and knowledge-base documents.
## Goal
Introduce versionable diagnosis playbook skills that agents can discover from a compact catalog and read on demand through a `read_skill` tool.
## Scope
- Six initial diagnosis playbooks: payment timeout, MySQL connection pool, Redis timeout, slow response, JVM memory risk, and AIOps alert.
- Classpath skill catalog loader.
- `read_skill` Spring AI tool.
- Prompt catalog injection for Chat and AIOps Planner/Executor paths.
- Focused tests.
## Non-goals
- No external API or database schema changes.
- No replacement of `lookup_knowledge`.
- No Verifier skill loading.
- No direct dependency on Java2AI `SkillsAgentHook` until local package names are verified.
## Scale
standard
## OpenSpec
`openspec/changes/diagnosis-playbook-skills/`
@@ -0,0 +1,23 @@
# Decisions: diagnosis-playbook-skills
## Key Decisions
- Use "diagnosis playbook skills" as the canonical term: skill is the runtime loading unit, playbook is the diagnosis workflow content.
- Keep factual knowledge in `knowledge_base/`; skills contain workflow, evidence order, stop conditions, and report rules.
- Implement a project-local progressive disclosure mechanism first because local Maven cache did not confirm Java2AI skill hook package names.
- Keep Verifier unchanged so it only validates existing tool evidence.
- Do not persist `read_skill` as evidence in `tool_invocation`; diagnostic facts must still come from evidence tools.
## Interface Impact
L2 internal interface:
- New `SkillCatalogService`.
- New `ReadSkillTool`.
- Chat/AIOps internal method tools include `read_skill`.
- No HTTP, DTO, database, or external response contract changes.
## Verification
- `mvn -q "-Dtest=SkillCatalogServiceTest,ChatServiceSequentialAgentTest,AiOpsServiceTest,DiagnosisTraceEvaluatorTest" test`
- Result: passed.
@@ -0,0 +1,25 @@
# Acceptance: mvp-demo-interview-runbook
## Classification
standard-light
## Task Status
| Task | Status | Notes |
| --- | --- | --- |
| Issue and OpenSpec setup | Done | Created slug-based issue and OpenSpec artifacts. |
| Implementation | Done | Added request payload, runnable script, output directory docs, interview walkthrough, and trace checklist. |
| Verification | Done | OpenSpec validation passed. |
## Current State
- No backend runtime behavior has been changed.
- Demo is packaged under `mvp/demo` for interview use.
## Verification
### OpenSpec Verification
- Command: `openspec validate mvp-demo-interview-runbook --strict`
- Result: passed
@@ -0,0 +1,29 @@
# Brief: mvp-demo-interview-runbook
## Background
Plan C is the interview-facing demo package. The project has the engineering pieces, but needs a single place to run and explain the MVP flow.
## Goals
1. Provide a fixed payment-timeout request payload.
2. Provide a PowerShell script that runs chat, trace, and feedback.
3. Save demo responses under `mvp/demo/output`.
4. Add interview walkthrough and trace checklist.
## Scope
- Demo docs and scripts only
- Existing local APIs only
- Existing `mvp-demo` profile only
## Non-Goals
- No backend code changes
- No eval extension
- No secret cleanup
- No full offline runtime
## Related OpenSpec
`openspec/changes/mvp-demo-interview-runbook/`
@@ -0,0 +1,27 @@
# MVP Demo Interview Runbook Decisions
## Clarify
- Entry summary: package existing MVP capabilities into a repeatable interview demo.
- Slug: `mvp-demo-interview-runbook`
- Devflow scale: standard-light
## Context
- Evidence trace and eval baseline work are already done.
- The next useful step is not more eval tooling, but a runnable demo path.
## Key Decisions
- Decision: Keep this change documentation/script-only.
- Reason: Plan C is about demo packaging, not new runtime capability.
- Decision: Use a stable session id.
- Reason: it makes trace lookup and saved output predictable.
- Decision: Save outputs to `mvp/demo/output`.
- Reason: generated artifacts should be easy to review without mixing into source fixtures.
## Open Questions
- Whether a later change should add a truly offline stubbed demo mode.
@@ -0,0 +1,9 @@
# Evidence: mvp-demo-interview-runbook
## Evidence Log
- 2026-07-05: Created Plan C demo packaging issue and OpenSpec change.
- 2026-07-05: Added fixed payment-timeout request payload.
- 2026-07-05: Added PowerShell demo script for chat, trace, and feedback.
- 2026-07-05: Added interview walkthrough and trace inspection checklist.
- 2026-07-05: Verification passed with `openspec validate mvp-demo-interview-runbook --strict`.
@@ -0,0 +1,101 @@
# Modular RAG Pipeline — Acceptance
## 验收状态
状态:通过,OpenSpec 已归档。
任务完成:
- OpenSpec tasks:31/31 完成。
- Review 后新增去重边界修复和回归测试。
- OpenSpec archive:`openspec/changes/archive/2026-07-06-modular-rag-pipeline`。
## 静态验证
```powershell
openspec validate modular-rag-pipeline --strict
```
结果:
```text
Change 'modular-rag-pipeline' is valid
```
```powershell
git diff --check
```
结果:
```text
PASS
```
说明:仅出现 Windows LF/CRLF warning,无 whitespace error。
## 脚本验证
```powershell
mvn -q -DskipTests compile
```
结果:PASS。
```powershell
mvn -q "-Dtest=LookupKnowledgeToolTest,ToolInvocationRecorderTest" test
```
结果:PASS。
覆盖:
- filtered L1 成功不 retry。
- filtered L1 低质量触发 raw unfiltered retry。
- filtered L1 无 evidence 触发 raw unfiltered retry。
- L0 hint 不作为 standalone fact evidence。
- 无 L0 hint 时直接 unfiltered vector search。
- rerank 使用 hint match 并记录 trace。
- context pack 保留 source/title/breadcrumb/hit reasons。
- evidence blocks 按 source 去重。
- session dedup 不再返回可消费 evidence/context。
- recorder 记录 retrieval trace、rerank trace、context pack summary 和 evidence summaries。
```powershell
$env:MILVUS_TOKEN = <application.yml 中的 milvus.token>; mvn -q test
```
结果:PASS。
说明:
- `MilvusConnectionTest` 需要 `MILVUS_TOKEN` 环境变量,直接读 `System.getenv`,不会自动读 `application.yml`。
- 注入该环境变量后完整测试通过。
## 实现验收
已验证行为:
- `LookupKnowledgeTool` 已变为 pipeline orchestrator。
- `LookupResult` 新契约包含 `evidenceBlocks`、`contextPack`、`retrievalTrace`、`rerankTrace`。
- 旧 `primary/supplement` 字段和 DTO 已删除。
- `ToolInvocationRecorder` 不再依赖 `result.getPrimary()`。
- filtered retrieval 失败时会记录 `filtered_vector_no_evidence` 或 `filtered_vector_low_quality`。
- no-evidence 情况返回 `found=false` 且保留 retrieval trace。
- session dedup 情况返回 `found=false` 且 evidence/context 为空。
## 未验证项
人工 Demo 未执行:
- 还没有通过真实 Chat/AIOps 会话观察 Agent 是否稳定按 `contextPack.packedText` 和 `evidenceBlocks` 引用证据。
风险:
- 工具 JSON 契约是 L4 breaking change,prompt 已更新,但真实对话行为仍建议做一次端到端 demo。
## 后续建议
- 增加一组 RAG eval cases,固定 query、期望 evidence source、期望 fallback path。
- 将 `MilvusConnectionTest` 改成 Spring 配置驱动或 integration profile,避免配置源混用。
- 后续可在评测数据足够后再考虑 model-based rerank 或 hybrid retrieval。
@@ -0,0 +1,54 @@
# Modular RAG Pipeline — Brief
## 背景
`lookup_knowledge` 已经能返回知识库证据,但实现集中在 `LookupKnowledgeTool` 内部:L0 查询分析、L1 向量召回、相关性归一化、证据组装、会话去重和 trace 入库耦合在一起。
旧返回契约 `primary/supplement` 也延续了“L0 是主结果、L1 是补充”的语义,和当前设计目标不一致。新的目标是让 L0 只作为 query understanding / filter / rerank / trace 信号,让 L1 向量检索成为事实证据来源。
## 目标
- 将 `lookup_knowledge` 改造成模块化 RAG pipeline。
- 保留显式 Agent tool 边界,不改工具名和 query 参数。
- L0 只提供领域、关键词、实体、category filter 和 trace hint。
- L1 filtered vector retrieval 失败或低质量时,降级为 raw query unfiltered L1 retry。
- 输出 evidence-first contract:`evidenceBlocks`、`contextPack`、`retrievalTrace`、`rerankTrace`。
- 保持 `tool_invocation` 表结构稳定,把新 trace 写入 `retrieval_details` JSON。
## 范围
已完成:
- 新增 pipeline DTO:`KnowledgeQuery`、`RetrievedEvidenceCandidate`、`ContextPack`、`RetrievalTrace`、`RerankTrace`、`EvidencePostprocessResult`。
- 新增 pipeline service:`KnowledgeQueryTransformer`、`KnowledgeDocumentRetriever`、`KnowledgeEvidencePostProcessor`、`KnowledgeContextPacker`、`LookupResultAssembler`。
- 重构 `LookupKnowledgeTool` 为薄 orchestration 层。
- 迁移 `LookupResult`,删除 `primary/supplement` 字段和 `PrimaryResult` / `SupplementResult` 类。
- 更新 `ToolInvocationRecorder`,记录 query transform、retrieval trace、context pack summary、rerank trace、fallback reason 和 evidence summaries。
- 更新 executor prompt 和 RAG 架构文档。
- 补充 lookup、recorder、fallback、rerank、context pack、session dedup 测试。
非目标:
- 不引入 implicit Advisor。
- 不引入 cross-encoder、BM25、RRF、Elasticsearch、OpenSearch。
- 不改文档上传、chunk、embedding 写入、Milvus schema。
- 不改变 Agent 何时调用 `lookup_knowledge`。
## 关联 OpenSpec
- `openspec/changes/archive/2026-07-06-modular-rag-pipeline`
## 接口影响
级别:L4 breaking interface。
原因:
- 删除旧 `LookupResult.primary` / `LookupResult.supplement`。
- `lookup_knowledge` tool JSON 输出形状变化。
缓解:
- 工具名和输入参数保持不变。
- in-repo 消费方、测试和 prompt 同步迁移。
- `tool_invocation` 表结构不变。
@@ -0,0 +1,72 @@
# Modular RAG Pipeline — Decisions
## D1: `lookup_knowledge` 保持显式工具
不把知识检索做成隐式 Advisor。Agent 仍显式调用 `lookup_knowledge(query)`,这样 trace、Verifier、Eval 都能看到工具调用边界。
## D2: L0 只做 query understanding
L0 产出:
- `domainHints`
- `matchedKeywords`
- `entities`
- `categoryFilter`
- `l0Titles`
- `l0MatchCount`
L0 不再直接转成 fact evidence。L0 hint 可以影响 filter、rerank、trace,但不能在 L1 失败时冒充知识证据。
## D3: MVP 降级策略采用 unfiltered L1 retry
流程:
```text
filtered L1 with L0 category filter
-> empty / no final evidence / below reference threshold
-> raw query unfiltered L1 retry
-> still no evidence => no_evidence
```
取舍:
- 简单、可解释、适合 MVP。
- 避免引入 BM25/RRF/multi-query/cross-encoder 的复杂度。
- 代价是低质量场景多一次向量查询,已通过 trace 记录 attempt duration。
## D4: 删除 `primary/supplement`
这是一次 L4 breaking interface change。
删除原因:
- `primary/supplement` 绑定旧语义:L0 primary、L1 supplement。
- 新设计中事实证据来自 `evidenceBlocks/contextPack`。
迁移结果:
- `LookupResult` 暴露 evidence-first 字段。
- `PrimaryResult` / `SupplementResult` 已删除。
- 生产代码和测试不再引用 `getPrimary()` / `getSupplement()`。
## D5: Trace 表结构保持稳定
`tool_invocation` 表不新增列。新增信息写入 `retrieval_details` JSON:
- `query_transform`
- `retrieval_trace`
- `context_pack_summary`
- `rerank_trace`
- `fallback_reason`
- `evidence_blocks`
原因:当前 trace、Verifier、Eval 已经以 `tool_invocation` 为证据入口,JSON details 足够承载 RAG 细节,避免 schema churn。
## D6: 会话去重不返回可消费证据
Review 后修正:
- dedup result 的 `found=false` 必须和 evidence/context 语义一致。
- 返回消息说明文档已检索过。
- 不再返回 `evidenceBlocks/contextPack`,避免 Agent 重复使用同一证据。
- 保留 `retrievalTrace` 和 `retrievedDomainsThisSession` 便于可观测。
@@ -0,0 +1,56 @@
# Modular RAG Pipeline — Evidence
## 代码证据
关键入口:
- `src/main/java/com/superbiz/agent/tool/LookupKnowledgeTool.java`
- `src/main/java/com/superbiz/agent/service/ToolInvocationRecorder.java`
- `src/main/java/com/superbiz/agent/dto/LookupResult.java`
新增模块:
- `KnowledgeQueryTransformer`:复用 `KnowledgeIndexService.analyzeQuery`,把 L0 转成 query hints 和可选 `categoryFilter`。
- `KnowledgeDocumentRetriever`:封装 `VectorSearchService.searchSimilarDocuments(query, topK, category)`,统一 filtered / unfiltered attempt。
- `KnowledgeEvidencePostProcessor`:归一化 L2、创建 evidence blocks、source dedup、规则 rerank、输出 `RerankTrace`。
- `KnowledgeContextPacker`:按字符预算打包 evidence,保留 source/title/breadcrumb/hit reasons。
- `LookupResultAssembler`:统一组装 evidence-first result、no-evidence result、session dedup result。
## 设计证据
已有文档约束:
- `mvp/architecture/rag-architecture.md`:RAG 应表达为可解释 pipeline,而不是一坨工具逻辑。
- `mvp/architecture/retrieval-observability.md`:L0 是 hint/explainability 层,L1 是语义检索主路径。
- `devflow/glossary/CONTEXT.md`:`lookup_knowledge` 是显式 Agent evidence tool,`tool_invocation` 是 trace / verifier / eval 的证据来源。
OpenSpec 对齐:
- `openspec/changes/modular-rag-pipeline/proposal.md`
- `openspec/changes/modular-rag-pipeline/design.md`
- `openspec/changes/modular-rag-pipeline/specs/rag-knowledge-retrieval/spec.md`
- `openspec/changes/modular-rag-pipeline/tasks.md`
## 用户确认
- 一次到位做模块化 RAG,而不是只做小补丁。
- L0 不再作为事实证据兜底。
- filtered L1 不准时,MVP 降级为 raw query unfiltered L1 retry。
- 可以新增字段,并删除旧字段以换取后续流程清晰。
## Review 发现
Review 中发现一个非阻塞但应修复的问题:
- 会话去重命中时,返回 `found=false` 但仍带 `evidenceBlocks/contextPack`,可能导致 Agent 重复消费同一份证据。
修复:
- `LookupResultAssembler.deduped` 清空可消费 evidence/context,只保留 message、trace、relevance hint 和 session domain memory。
- 新增 `LookupKnowledgeToolTest.sessionDedupDoesNotReturnConsumableEvidenceAgain`。
## 非阻塞观察
- `MilvusConnectionTest` 仍直接依赖 `MILVUS_TOKEN` 环境变量;主配置中已有 token,但测试不读 Spring 配置。
- 测试日志仍有 ANTLR 版本 warning,不影响测试通过。
- 控制台在部分命令输出中仍会出现中文编码显示问题,但源码按 UTF-8 读取时关键用户提示文本正常。
@@ -0,0 +1,78 @@
# Acceptance: rag-eval-pipeline-closure
## Status
Archived.
## Acceptance Criteria
| Item | Status | Notes |
|---|---|---|
| Modular fixture support | Done | Evaluator reads `lookupResult.evidenceBlocks/contextPack/retrievalTrace/rerankTrace`. |
| LookupResult-only contract | Done | Evaluator fails fixtures that do not expose `lookupResult`. |
| Golden modular assertions | Done | Cases assert selected attempt, accepted fallback reason, evidence status, context sources, and rerank top source. |
| Fallback coverage | Done | Added `chat-l0-filter-fallback` for filtered low-quality/no-evidence to unfiltered retry. |
| Real tool snapshot generation | Done | Added `RagLookupSnapshotGeneratorTest` and `generate_rag_lookup_snapshots.ps1`, defaulting to Spring AI VectorStore mode. |
| Seed docs import/reindex | Done | Added canonical seed docs, `RagEvalSeedImporterTest`, and `prepare_rag_eval_seed.ps1`. |
| Eval metadata isolation | Done | Added `kb_scope` metadata and `retrieval.kb-scope` filtering for L0 and L1. |
| Frontmatter body split | Done | Upload chunking embeds Markdown body, while frontmatter feeds metadata and L0. |
| Baseline diff | Done | `--compare-to` writes JSON/Markdown diff and exits non-zero on regression. |
| Documentation | Done | Updated RAG eval README and added `mvp/architecture/rag-eval-closure.md`. |
## Verification
```powershell
python scripts\eval_rag_retrieval.py
```
Result: passed. 7 cases, passRate=1.0, recall@5=1.0.
```powershell
mvn -q "-Dtest=RagLookupSnapshotGeneratorTest" test
```
Result: passed. The snapshot generator stays disabled unless `rag.snapshot.enabled=true` is provided.
```powershell
mvn -q "-Dtest=RagLookupSnapshotGeneratorTest" "-Drag.snapshot.enabled=true" "-Drag.snapshot.fixtures=<temp-fixtures>" "-Drag.snapshot.retrievedAt=2026-07-06T00:00:00Z" "-Dretrieval.kb-scope=rag-eval" "-Dretrieval.vector-store.mode=spring" test
python scripts\eval_rag_retrieval.py --fixtures <temp-fixtures> --json-report <temp-current.json> --markdown-report <temp-current.md>
```
Result: passed. Spring AI VectorStore live snapshot produced 7 cases, passRate=1.0, recall@5=1.0. The fallback case used `selectedAttempt=UNFILTERED_VECTOR_RETRY` and `fallbackReason=filtered_vector_no_evidence`; the expected source `rag-l0-filter-fallback` remained rank 1.
```powershell
python scripts\eval_rag_retrieval.py --json-report <temp-current.json> --markdown-report <temp-current.md> --compare-to eval\rag-retrieval\reports\baseline.json --diff-json-report <temp-diff.json> --diff-markdown-report <temp-diff.md>
```
Result: passed. regressions=0.
```powershell
mvn -q "-Dtest=FrontmatterParserTest,VectorIndexServiceTest,VectorSearchServiceTest,DocumentManagementServiceTest,RagLookupSnapshotGeneratorTest,RagEvalSeedImporterTest" test
```
Result: passed. The seed importer and snapshot generator remain disabled unless their system properties are explicitly enabled.
```powershell
$null = [scriptblock]::Create((Get-Content -Raw scripts\prepare_rag_eval_seed.ps1))
$null = [scriptblock]::Create((Get-Content -Raw scripts\generate_rag_lookup_snapshots.ps1))
```
Result: PowerShell syntax OK.
```powershell
.\scripts\prepare_rag_eval_seed.ps1
```
Result: passed. Seed docs were imported through `DocumentManagementService` and reindexed into the configured runtime DB/vector stack.
```powershell
.\scripts\generate_rag_lookup_snapshots.ps1 -Fixtures <temp-fixtures> -RetrievedAt 2026-07-06T00:00:00Z -SkipEval
```
Result: passed after defaulting the script to `retrieval.vector-store.mode=spring`. The script generated live fixtures through the real `LookupKnowledgeTool` and then the offline evaluator reported 7 cases, passRate=1.0, recall@5=1.0.
```powershell
git diff --check
```
Result: no whitespace errors. Git reported only LF/CRLF conversion warnings.
@@ -0,0 +1,21 @@
# Brief: rag-eval-pipeline-closure
## Background
The modular RAG pipeline now returns `LookupResult` with `evidenceBlocks`, `contextPack`, `retrievalTrace`, and `rerankTrace`. The RAG retrieval baseline must validate that full contract, so it can detect regressions in fallback behavior, context packing, or rerank trace.
## Goals
1. Reuse the existing offline RAG retrieval baseline.
2. Extend it to support modular `LookupResult` fixtures.
3. Add golden assertions for selected attempt, fallback reason, evidence status, context sources, and rerank top source.
4. Add a RAG baseline diff path for regression detection.
5. Add a snapshot generator that calls the real `LookupKnowledgeTool`.
6. Document how RAG baseline and diagnosis baseline form a quality loop.
## Non-Goals
- No new production API.
- No LLM-as-judge scoring.
- No production API behavior changes.
- No replacement for diagnosis eval.
@@ -0,0 +1,57 @@
# Decisions: rag-eval-pipeline-closure
## D1. Reuse the existing evaluator
Decision: extend `scripts/eval_rag_retrieval.py` instead of creating a second evaluator.
Reason: the old evaluator already owns golden cases, fixtures, hit-level classification, and Markdown/JSON reports. Extending it keeps one RAG baseline path.
## D2. Use LookupResult as the only fixture contract
Decision: support `lookupResult` only.
Reason: the MVP has moved to evidence-first RAG. Keeping an older fixture contract would weaken the baseline and let incomplete fixtures bypass context packing, retrieval trace, and rerank checks.
## D3. Make modular assertions opt-in per case
Decision: use fields such as `expectedSelectedAttempt`, `expectedFallbackReason`/`expectedFallbackReasons`, `expectedEvidenceStatus`, `expectedContextSources`, and `expectedRerankTopSource`.
Reason: golden cases can be strict where the pipeline path matters without forcing every historical case to assert every new field.
## D4. Diff remains deterministic
Decision: RAG diff compares report fields only and does not call live services or models.
Reason: this keeps it suitable for local regression checks and CI-style gates.
## D5. Isolate live eval docs with kb_scope
Decision: add `kb_scope` metadata and use `rag-eval` for canonical eval seed documents.
Reason: local production documents are not stable enough for golden retrieval expectations. Scope isolation lets real `LookupKnowledgeTool` snapshots use the same MySQL/Milvus stack while avoiding accidental matches from unrelated local data.
Default runtime keeps `retrieval.kb-scope` empty so legacy documents without `kb_scope` remain searchable. Eval scripts pass `-Dretrieval.kb-scope=rag-eval`. The same scope applies to L0 query hints and L1 vector retrieval.
## D6. Import seed docs through the real upload pipeline
Decision: seed docs are imported by `RagEvalSeedImporterTest` through `DocumentManagementService.uploadDocument`.
Reason: this updates DB metadata, L0 index state, local knowledge files, and Milvus chunks in the same way as normal document ingestion. A direct Milvus-only seed would make the live eval less representative.
## D7. Strip frontmatter before chunk embedding
Decision: uploaded Markdown frontmatter feeds metadata/L0 but is stripped before chunking and embedding.
Reason: frontmatter is a control plane, not evidence text. Keeping it in chunks lets L0-only keywords artificially improve vector similarity, especially for fallback decoy cases.
## D8. Treat retry behavior as the stable fallback contract
Decision: the fallback golden case accepts both `filtered_vector_low_quality` and `filtered_vector_no_evidence`, while still requiring `selectedAttempt=UNFILTERED_VECTOR_RETRY`, expected evidence source, context packing, and rerank top source.
Reason: Spring AI VectorStore and the Milvus SDK can differ on whether an over-filtered first pass returns a weak candidate or no candidate. The MVP contract is that the retriever skips only the L0 category filter, keeps `kb_scope`, retries the original query, and returns the correct evidence.
## D9. Default live snapshots to Spring AI VectorStore
Decision: `generate_rag_lookup_snapshots.ps1` defaults to `retrieval.vector-store.mode=spring`.
Reason: Spring AI VectorStore is the current framework path for the project and should be the default live verification route. SDK mode remains available through `-VectorStoreMode sdk` for comparison.
@@ -0,0 +1,87 @@
# Evidence: rag-eval-pipeline-closure
## Changed Assets
- `scripts/eval_rag_retrieval.py`
- `eval/rag-retrieval/cases/golden-cases.json`
- `eval/rag-retrieval/fixtures/*.json`
- `eval/rag-retrieval/reports/baseline.json`
- `eval/rag-retrieval/reports/baseline.md`
- `eval/rag-retrieval/README.md`
- `mvp/architecture/rag-eval-closure.md`
- `src/test/java/com/superbiz/agent/eval/RagLookupSnapshotGeneratorTest.java`
- `src/test/java/com/superbiz/agent/eval/RagEvalSeedImporterTest.java`
- `scripts/generate_rag_lookup_snapshots.ps1`
- `scripts/prepare_rag_eval_seed.ps1`
- `eval/rag-retrieval/seed-docs/*.md`
- `src/main/java/com/superbiz/agent/dto/Frontmatter.java`
- `src/main/java/com/superbiz/agent/dto/KnowledgeEntry.java`
- `src/main/java/com/superbiz/agent/service/FrontmatterParser.java`
- `src/main/java/com/superbiz/agent/service/KnowledgeIndexService.java`
- `src/main/java/com/superbiz/agent/service/DocumentManagementService.java`
- `src/main/java/com/superbiz/agent/service/VectorIndexService.java`
- `src/main/java/com/superbiz/agent/service/VectorSearchService.java`
- `src/main/java/com/superbiz/agent/service/SpringAiVectorStoreSidecarService.java`
- `src/main/resources/application.yml`
## Baseline Result
```text
Evaluated 7 cases: passRate=1.0, recall@5=1.0, failed=0
```
## Regression Signals
The evaluator now fails on:
- missing expected source
- missing breadcrumb or evidence keyword
- non-`lookupResult` fixture
- selected attempt mismatch
- fallback reason mismatch
- fallback reason outside accepted values
- evidence status mismatch
- missing context source
- rerank top source mismatch
The snapshot generator now provides:
- real `LookupKnowledgeTool` invocation
- one fixture per golden case
- explicit opt-in through `rag.snapshot.enabled=true`
- optional post-generation baseline evaluation
- scoped retrieval through `retrieval.kb-scope=rag-eval`
- Spring AI VectorStore by default through `retrieval.vector-store.mode=spring`
- scoped L0 hints through the same `retrieval.kb-scope`
The seed importer now provides:
- canonical eval docs under `eval/rag-retrieval/seed-docs`
- real `DocumentManagementService` import/reindex
- stable `source`/`docId` metadata
- `kb_scope=rag-eval` isolation from local non-eval documents
- frontmatter stripping before chunk embedding
- an over-filter decoy seed doc for fallback-path evaluation
The diff now detects:
- aggregate pass/recall regression
- case pass regression
- hit-level regression
- first-rank regression
- selected attempt/fallback/evidence/rerank changes
## Final Spring Live Snapshot
```text
.\scripts\generate_rag_lookup_snapshots.ps1 -Fixtures <temp-fixtures> -RetrievedAt 2026-07-06T00:00:00Z
Evaluated 7 cases: passRate=1.0, recall@5=1.0, failed=0
```
Key fallback trace:
```text
selectedAttempt=UNFILTERED_VECTOR_RETRY
fallbackReason=filtered_vector_no_evidence
rerankTopSource=rag-l0-filter-fallback
```
@@ -0,0 +1,16 @@
# Acceptance: executor-evidence-output-contract
## Draft Acceptance
- [x] Issue exists: `mvp/issues/executor-evidence-attribution-hallucination.md`.
- [x] OpenSpec change artifacts exist.
- [x] devflow tracking files exist.
- [x] OpenSpec validation passes.
- [x] Implementation updates Chat Executor prompt.
- [x] Implementation passes structured Executor output to Verifier.
- [x] Verifier prefers structured claims and still falls back safely.
- [x] Focused tests cover parsing, payload assembly, and unsupported confirmed claims.
## Notes
This project is currently in proposal/design stage. Runtime code is intentionally not changed yet.
@@ -0,0 +1,23 @@
# Brief: executor-evidence-output-contract
## Summary
Executor currently returns natural-language diagnosis answers that may mix confirmed evidence, runbook guidance, historical patterns, and unsupported inference. Verifier catches many unsupported facts, but only after extracting claims from prose.
This project defines a structured Executor evidence-attribution contract and updates the Verifier input/verification path to consume it.
## Goal
Make Chat Executor output machine-checkable so confirmed claims are explicitly bound to current-session evidence, while hypotheses and evidence gaps remain visibly separate.
## Scope
- Chat Executor prompt contract.
- Executor structured output parsing.
- Verifier payload extension.
- Chat Verifier prompt behavior.
- Focused tests/eval fixtures.
## Related OpenSpec
`openspec/changes/executor-evidence-output-contract/`
@@ -0,0 +1,71 @@
# Decisions: executor-evidence-output-contract
## sm-flow Progress
### Clarify
Entry summary: recent Chat diagnosis sessions are `LOW_CONFID` because Executor presents unsupported or weakly supported details as confirmed facts after successful tool calls.
Slug: `executor-evidence-output-contract`
Scale: standard. This affects prompts, verifier input assembly, parsing behavior, and tests, but does not require a database schema change.
### Context
Relevant history:
- `executor-action-memory-relevance`: Executor already has retrieval quality constraints and should avoid repeated `lookup_knowledge`.
- `chat-verifier-agent`: Verifier should not see intermediate reasoning; it receives explicit `original_query`, `executor_final_answer`, `tool_trace_summary`, and `retry_context`.
- `evidence-trace-hardening`: evidence-bearing tools persist stable traces and no-evidence semantics.
- `modular-rag-pipeline`: `lookup_knowledge` exposes evidence blocks and context packs; L0 hints are not fact evidence.
Current code shape:
- `src/main/resources/prompts/chat-executor-prompt.md` is the Chat Executor prompt.
- `src/main/resources/prompts/executor-prompt.md` is for the AiOps flow and is not the target of this Chat change.
- `VerifierInputHook` currently builds a payload with `original_query`, `executor_final_answer`, `tool_trace_summary`, and `retry_context`.
- Verifier prompt currently extracts facts from `executor_final_answer`.
### Grill
Question: Should Executor output only JSON or JSON plus readable answer?
Decision: use one JSON object containing both machine fields and `user_facing_answer`. This avoids losing a readable Chinese answer while giving Verifier structured claims.
Question: Should evidence binding use `chunk_id`?
Decision: no. Use generic binding fields because `query_logs` and `query_metrics` do not naturally expose RAG chunks.
Question: Should Verifier trust Executor-provided claims completely?
Decision: no. Verifier should verify structured claims first, then scan `user_facing_answer` for extra confirmed-sounding facts omitted from `claims`.
Question: What happens when Executor JSON is malformed?
Decision: preserve raw final answer, mark parse failure, and fall back to existing natural-language verification.
### Specify
OpenSpec artifacts:
- `proposal.md`: why and scope
- `design.md`: contract, verifier behavior, risks
- `specs/chat-verifier-agent/spec.md`: modified and added requirements
- `tasks.md`: implementation checklist
### Audit
Cross-artifact alignment:
- Issue describes evidence attribution hallucination.
- Proposal scopes the fix to Executor output and Verifier consumption.
- Design preserves existing verifier isolation.
- Spec adds observable behavior without changing database schema.
- Tasks remain implementation-oriented and unchecked.
Interface impact:
- Prompt/output contract: L2 internal Agent contract change.
- Verifier payload: L2 internal structured input extension.
- Database schema: no change.
- External HTTP API: no intended change.
@@ -0,0 +1,17 @@
# Evidence: executor-evidence-output-contract
## Repository Evidence
- `chat-executor-prompt.md` currently requires using real tool data but does not require a structured evidence-attribution output.
- `chat-verifier-prompt.md` currently extracts facts from `executor_final_answer` prose and compares them with `tool_trace_summary`.
- `VerifierInputHook` currently provides `original_query`, `executor_final_answer`, `tool_trace_summary`, and `retry_context`.
- `openspec/specs/chat-verifier-agent/spec.md` already requires explicit verifier inputs, auditable evidence refs, fixed verdicts, and low-confidence handling.
- `openspec/specs/evidence-trace-hardening/spec.md` already distinguishes failed, no-hit, deduped, and successful evidence-tool traces.
## Runtime Evidence From Recent Sessions
Recent MySQL inspection showed repeated `LOW_CONFID` verifier results with many `no_evidence` facts. Typical unsupported claims included OOM, Full GC frequency, specific slow SQL timings, lock waits, and service-specific timeout details that were not supported by current-session tool traces.
## Design Evidence
This change preserves the previous design that Verifier should not inspect intermediate reasoning. The new structured output is still final Executor output, not hidden chain-of-thought.
+233
View File
@@ -0,0 +1,233 @@
# RAG Retrieval Baseline
This directory contains the offline retrieval baseline for the RAG refactor.
The baseline is intentionally narrower than full diagnosis evaluation. It checks
whether fixed retrieval queries can recover expected documents, breadcrumbs, and
evidence keywords before changing L0 behavior, query augmentation, evidence
post-processing, or Spring AI VectorStore integration.
## Layout
```text
eval/rag-retrieval/
cases/golden-cases.json Fixed retrieval golden cases
seed-docs/*.md Canonical docs imported into the live KB for real-tool eval
fixtures/*.json Saved retrieval fixtures for each case
reports/baseline.json Machine-readable baseline report
reports/baseline.md Human-readable baseline report
reports/baseline-diff.* Optional diff reports
reports/live-post-reindex.* Optional live acceptance reports
```
## Seed Docs + Import/Reindex
The live-tool eval uses canonical seed documents so the real
`LookupKnowledgeTool` can retrieve stable evidence from MySQL/Milvus instead of
whatever ad hoc documents happen to exist in the local knowledge base.
Seed documents live in:
```text
eval/rag-retrieval/seed-docs/*.md
```
Each seed doc uses frontmatter fields that are propagated into vector metadata:
```yaml
source: mysql-connection-pool
breadcrumb: Database > MySQL > Connection Pool
kb_scope: rag-eval
```
Import or reindex the seed docs through the real upload pipeline:
```powershell
.\scripts\prepare_rag_eval_seed.ps1
```
The script runs `RagEvalSeedImporterTest` with `rag.seed.enabled=true`. It
deletes the existing document with the same `source`/`docId`, uploads the seed
doc through `DocumentManagementService`, updates DB metadata and L0, and rebuilds
Milvus chunks.
`kb_scope` isolates eval data:
- default application config leaves `retrieval.kb-scope` empty, so legacy docs
without `kb_scope` remain searchable;
- eval scripts pass `-Dretrieval.kb-scope=rag-eval`, so L0 query hints and L1
vector retrieval both use only the canonical eval seed docs;
- the fallback retry skips only the L0 category filter, not the `kb_scope`
boundary.
Frontmatter is not embedded as chunk content during upload. It feeds metadata,
L0, and document enrichment; only the Markdown body is chunked and embedded.
This keeps controlled L0 decoys from becoming semantically relevant just because
their frontmatter keywords matched the query.
## Run
From the repository root:
```bash
python scripts/eval_rag_retrieval.py
```
Custom paths are also supported:
```bash
python scripts/eval_rag_retrieval.py \
--cases eval/rag-retrieval/cases/golden-cases.json \
--fixtures eval/rag-retrieval/fixtures \
--json-report eval/rag-retrieval/reports/baseline.json \
--markdown-report eval/rag-retrieval/reports/baseline.md
```
## Generate Fixtures From LookupKnowledgeTool
Use the snapshot generator when fixtures should reflect the real
`LookupKnowledgeTool` pipeline:
```powershell
.\scripts\generate_rag_lookup_snapshots.ps1
```
For the intended live loop, run seed import first:
```powershell
.\scripts\prepare_rag_eval_seed.ps1
.\scripts\generate_rag_lookup_snapshots.ps1
python scripts\eval_rag_retrieval.py
```
The script runs a Spring test harness:
```text
mvn -q -Dtest=RagLookupSnapshotGeneratorTest -Drag.snapshot.enabled=true -Dretrieval.kb-scope=rag-eval -Dretrieval.vector-store.mode=spring test
```
The generator reads `golden-cases.json`, injects the real `LookupKnowledgeTool`
bean, calls `lookupKnowledge(query)` for each case, writes
`fixtures/{caseId}.json`, and then runs `eval_rag_retrieval.py` unless
`-SkipEval` is provided. It defaults to Spring AI VectorStore mode; pass
`-VectorStoreMode sdk` only when intentionally comparing the legacy SDK path.
Custom paths are supported:
```powershell
.\scripts\generate_rag_lookup_snapshots.ps1 `
-Cases eval\rag-retrieval\cases\golden-cases.json `
-Fixtures eval\rag-retrieval\fixtures `
-RetrievedAt 2026-07-06T00:00:00Z
```
The generator is disabled in normal test runs. It only executes when
`rag.snapshot.enabled=true` is provided because it writes repository files and
depends on the configured runtime retrieval stack.
If generated fixtures fail the offline baseline, treat that as a real alignment
signal: either the golden expectations need to be adjusted to the current
knowledge base, or the knowledge base/indexing path needs to be fixed.
## Modular RAG Contract
Fixtures must use the current `lookupResult` shape, which mirrors the
`lookup_knowledge` output:
```text
lookupResult.evidenceBlocks
lookupResult.contextPack
lookupResult.retrievalTrace
lookupResult.rerankTrace
```
Golden cases can assert both retrieval quality and pipeline behavior:
- `expectedSources` / `expectedDocIds`
- `expectedBreadcrumbs`
- `expectedKeywords`
- `expectedSelectedAttempt`
- `expectedFallbackReason`
- `expectedFallbackReasons`
- `expectedEvidenceStatus`
- `expectedContextSources`
- `expectedRerankTopSource`
This lets the baseline catch regressions such as losing the expected evidence
source, skipping context packing, changing the selected retrieval attempt, or
breaking the filtered-vector to unfiltered-retry fallback.
## Baseline Diff
To compare a freshly generated report against an existing baseline:
```bash
python scripts/eval_rag_retrieval.py \
--json-report eval/rag-retrieval/reports/current.json \
--markdown-report eval/rag-retrieval/reports/current.md \
--compare-to eval/rag-retrieval/reports/baseline.json \
--diff-json-report eval/rag-retrieval/reports/baseline-diff.json \
--diff-markdown-report eval/rag-retrieval/reports/baseline-diff.md
```
The diff reports aggregate regressions and case-level changes for:
- pass rate, recall@K, strong hit rate, miss count
- pass state
- hit level
- first expected rank
- selected attempt
- fallback reason
- evidence status
- rerank top source
The command exits non-zero when a case fails or the diff contains a regression.
## Hit Levels
- `strong`: expected document is found and breadcrumb or evidence keyword coverage is satisfied.
- `medium`: expected document is found, but breadcrumb or keyword coverage is incomplete.
- `weak`: expected evidence keyword is found, but expected document is missing.
- `miss`: expected document and expected evidence are not found.
`Recall@K` counts `strong` and `medium` as retrieved.
## Scope
This baseline runs fully offline and does not call MySQL, Redis, Milvus, an LLM,
or the Spring Boot application. It is a regression harness for retrieval behavior,
not a claim that live production retrieval accuracy is complete.
## Live Post-Reindex Acceptance
When embedding input changes, existing vectors do not update by themselves. For
example, after adding `title` and `breadcrumb` to the embedding text, the live
Milvus/Zilliz collection must be reindexed before retrieval can reflect that new
semantic signal.
Use this optional live acceptance flow after the application is running and the
knowledge base has been reindexed:
```bash
python scripts/eval_rag_live_acceptance.py
```
Custom service URL and output paths are supported:
```bash
python scripts/eval_rag_live_acceptance.py \
--base-url http://127.0.0.1:9900 \
--json-report eval/rag-retrieval/reports/live-post-reindex.json \
--markdown-report eval/rag-retrieval/reports/live-post-reindex.md
```
The script calls:
```text
GET /api/search/similar
```
It writes JSON and Markdown reports with query, topK, result count, top
results, breadcrumb, score labels, and raw response fields. This is a live
smoke check for environment readiness and post-reindex behavior; it does not
replace the deterministic offline baseline above.
+112
View File
@@ -0,0 +1,112 @@
{
"version": 1,
"description": "Offline golden retrieval cases for RAG refactor baseline.",
"topK": 5,
"cases": [
{
"caseId": "chat-mysql-connection-pool",
"scenario": "chat",
"query": "MySQL connection pool is exhausted. How should I diagnose it?",
"expectedDocIds": ["mysql-connection-pool"],
"expectedSources": ["mysql-connection-pool"],
"expectedBreadcrumbs": ["Database > MySQL > Connection Pool"],
"expectedKeywords": ["connection pool", "max_connections", "HikariCP"],
"expectedSelectedAttempt": "FILTERED_VECTOR",
"expectedFallbackReason": null,
"expectedEvidenceStatus": "supported",
"expectedContextSources": ["mysql-connection-pool"],
"expectedRerankTopSource": "mysql-connection-pool",
"notes": "Covers precise database troubleshooting retrieval."
},
{
"caseId": "chat-diagnosis-flow",
"scenario": "chat",
"query": "What is the standard troubleshooting flow for an application incident?",
"expectedDocIds": ["incident-diagnosis-flow"],
"expectedSources": ["incident-diagnosis-flow"],
"expectedBreadcrumbs": ["AIOps > Diagnosis Flow"],
"expectedKeywords": ["collect evidence", "verify", "remediation"],
"expectedSelectedAttempt": "FILTERED_VECTOR",
"expectedFallbackReason": null,
"expectedEvidenceStatus": "supported",
"expectedContextSources": ["incident-diagnosis-flow"],
"expectedRerankTopSource": "incident-diagnosis-flow",
"notes": "Covers process-style knowledge where breadcrumb matters."
},
{
"caseId": "aiops-payment-latency-alert",
"scenario": "aiops",
"query": "Alert HighLatency on payment-service with p95 latency above threshold",
"expectedDocIds": ["payment-service-latency"],
"expectedSources": ["payment-service-latency"],
"expectedBreadcrumbs": ["AIOps > Service Alerts > Payment Latency"],
"expectedKeywords": ["p95 latency", "payment-service", "downstream dependency"],
"expectedSelectedAttempt": "FILTERED_VECTOR",
"expectedFallbackReason": null,
"expectedEvidenceStatus": "supported",
"expectedContextSources": ["payment-service-latency"],
"expectedRerankTopSource": "payment-service-latency",
"notes": "Covers alert payload terms that should become retrieval hints."
},
{
"caseId": "aiops-prometheus-alert-scope",
"scenario": "aiops",
"query": "When an AIOps request already includes alert payload, should the agent diagnose unrelated active alerts?",
"expectedDocIds": ["aiops-alert-scope-control"],
"expectedSources": ["aiops-alert-scope-control"],
"expectedBreadcrumbs": ["AIOps > Alert Scope Control"],
"expectedKeywords": ["payload", "unrelated active alerts", "scope"],
"expectedSelectedAttempt": "FILTERED_VECTOR",
"expectedFallbackReason": null,
"expectedEvidenceStatus": "supported",
"expectedContextSources": ["aiops-alert-scope-control"],
"expectedRerankTopSource": "aiops-alert-scope-control",
"notes": "Covers scoped alert diagnosis behavior."
},
{
"caseId": "chat-rag-chunk-context",
"scenario": "chat",
"query": "If a long section is split into multiple chunks, how do we keep retrieval context?",
"expectedDocIds": ["rag-chunk-context-reconstruction"],
"expectedSources": ["rag-chunk-context-reconstruction"],
"expectedBreadcrumbs": ["RAG > Chunking > Context Reconstruction"],
"expectedKeywords": ["neighbor chunk", "same section", "breadcrumb"],
"expectedSelectedAttempt": "FILTERED_VECTOR",
"expectedFallbackReason": null,
"expectedEvidenceStatus": "supported",
"expectedContextSources": ["rag-chunk-context-reconstruction"],
"expectedRerankTopSource": "rag-chunk-context-reconstruction",
"notes": "Covers the known RAG refactor issue around context reconstruction."
},
{
"caseId": "chat-l0-domain-hint",
"scenario": "chat",
"query": "Should L0 keyword matching decide the final retrieval result?",
"expectedDocIds": ["rag-l0-domain-entity-hint"],
"expectedSources": ["rag-l0-domain-entity-hint"],
"expectedBreadcrumbs": ["RAG > L0 > Domain Entity Hint"],
"expectedKeywords": ["domain detector", "entity extractor", "metadata filter"],
"expectedSelectedAttempt": "FILTERED_VECTOR",
"expectedFallbackReason": null,
"expectedEvidenceStatus": "supported",
"expectedContextSources": ["rag-l0-domain-entity-hint"],
"expectedRerankTopSource": "rag-l0-domain-entity-hint",
"notes": "Covers the target L0 role after refactor."
},
{
"caseId": "chat-l0-filter-fallback",
"scenario": "chat",
"query": "RAG query was over-filtered by L0 and filtered vector search returned low quality evidence. What should happen?",
"expectedDocIds": ["rag-l0-filter-fallback"],
"expectedSources": ["rag-l0-filter-fallback"],
"expectedBreadcrumbs": ["RAG > Fallback > Unfiltered Retry"],
"expectedKeywords": ["skip the L0 filter", "unfiltered vector retry", "low quality"],
"expectedSelectedAttempt": "UNFILTERED_VECTOR_RETRY",
"expectedFallbackReasons": ["filtered_vector_low_quality", "filtered_vector_no_evidence"],
"expectedEvidenceStatus": "supported",
"expectedContextSources": ["rag-l0-filter-fallback"],
"expectedRerankTopSource": "rag-l0-filter-fallback",
"notes": "Covers the MVP fallback rule: if filtered L1 is low quality, retry raw query without L0 filter."
}
]
}
@@ -0,0 +1,81 @@
{
"caseId": "aiops-payment-latency-alert",
"query": "Alert HighLatency on payment-service with p95 latency above threshold",
"retrievedAt": "2026-07-06T00:00:00Z",
"lookupResult": {
"found": true,
"evidenceBlocks": [
{
"source": "payment-service-latency",
"title": "Payment Service Latency Alert Playbook",
"breadcrumb": "AIOps > Service Alerts > Payment Latency",
"retrievalLayer": "L1",
"content": "For payment-service p95 latency alerts, check downstream dependency latency, thread pool saturation, gateway retries, and recent deployment changes.",
"score": 0.84,
"hitReasons": ["domain_match:+0.15", "entity_match:+0.20", "keyword_match:+0.10"]
},
{
"source": "mysql-connection-pool",
"title": "MySQL Connection Pool Troubleshooting",
"breadcrumb": "Database > MySQL > Connection Pool",
"retrievalLayer": "L1",
"content": "Database connection pool saturation can increase payment latency when checkout paths wait for connections.",
"score": 0.68,
"hitReasons": ["keyword_match:+0.10"]
}
],
"contextPack": {
"packedText": "[1] Payment Service Latency Alert Playbook\nAIOps > Service Alerts > Payment Latency\nFor payment-service p95 latency alerts, check downstream dependency latency, thread pool saturation, gateway retries, and recent deployment changes.",
"strategy": "top_evidence_blocks",
"charBudget": 3500,
"usedChars": 236,
"includedSources": ["payment-service-latency", "mysql-connection-pool"],
"omittedSources": []
},
"retrievalTrace": {
"originalQuery": "Alert HighLatency on payment-service with p95 latency above threshold",
"rewrittenQuery": "HighLatency payment-service p95 latency alert downstream dependency diagnosis",
"categoryFilter": "AIOps",
"selectedAttempt": "FILTERED_VECTOR",
"fallbackReason": null,
"evidenceStatus": "supported",
"queryHints": {
"domains": ["AIOps"],
"matched_keywords": ["p95 latency", "payment-service", "downstream dependency"],
"entities": ["payment-service", "HighLatency"],
"l0_titles": ["Payment Service Latency Alert Playbook"],
"l0_match_count": 1
},
"attempts": [
{
"name": "FILTERED_VECTOR",
"query": "HighLatency payment-service p95 latency alert downstream dependency diagnosis",
"categoryFilter": "AIOps",
"candidateCount": 2,
"usable": true,
"durationMs": 11,
"topScore": 0.84,
"topSimilarity": 0.84
}
]
},
"rerankTrace": {
"items": [
{
"finalRank": 1,
"source": "payment-service-latency",
"baseScore": 0.84,
"finalScore": 1.29,
"boostReasons": ["domain_match:+0.15", "entity_match:+0.20", "keyword_match:+0.10"]
},
{
"finalRank": 2,
"source": "mysql-connection-pool",
"baseScore": 0.68,
"finalScore": 0.78,
"boostReasons": ["keyword_match:+0.10"]
}
]
}
}
}
@@ -0,0 +1,65 @@
{
"caseId": "aiops-prometheus-alert-scope",
"query": "When an AIOps request already includes alert payload, should the agent diagnose unrelated active alerts?",
"retrievedAt": "2026-07-06T00:00:00Z",
"lookupResult": {
"found": true,
"evidenceBlocks": [
{
"source": "aiops-alert-scope-control",
"title": "AIOps Alert Scope Control",
"breadcrumb": "AIOps > Alert Scope Control",
"retrievalLayer": "L1",
"content": "When payload mode is used, diagnose the input alert payload and do not expand unrelated active alerts into the main diagnosis scope.",
"score": 0.88,
"hitReasons": ["domain_match:+0.15", "keyword_match:+0.10"]
}
],
"contextPack": {
"packedText": "[1] AIOps Alert Scope Control\nAIOps > Alert Scope Control\nWhen payload mode is used, diagnose the input alert payload and do not expand unrelated active alerts into the main diagnosis scope.",
"strategy": "top_evidence_blocks",
"charBudget": 3500,
"usedChars": 188,
"includedSources": ["aiops-alert-scope-control"],
"omittedSources": []
},
"retrievalTrace": {
"originalQuery": "When an AIOps request already includes alert payload, should the agent diagnose unrelated active alerts?",
"rewrittenQuery": "AIOps alert payload scope unrelated active alerts diagnosis",
"categoryFilter": "AIOps",
"selectedAttempt": "FILTERED_VECTOR",
"fallbackReason": null,
"evidenceStatus": "supported",
"queryHints": {
"domains": ["AIOps"],
"matched_keywords": ["payload", "unrelated active alerts", "scope"],
"entities": ["alert payload"],
"l0_titles": ["AIOps Alert Scope Control"],
"l0_match_count": 1
},
"attempts": [
{
"name": "FILTERED_VECTOR",
"query": "AIOps alert payload scope unrelated active alerts diagnosis",
"categoryFilter": "AIOps",
"candidateCount": 1,
"usable": true,
"durationMs": 8,
"topScore": 0.88,
"topSimilarity": 0.88
}
]
},
"rerankTrace": {
"items": [
{
"finalRank": 1,
"source": "aiops-alert-scope-control",
"baseScore": 0.88,
"finalScore": 1.13,
"boostReasons": ["domain_match:+0.15", "keyword_match:+0.10"]
}
]
}
}
}
@@ -0,0 +1,81 @@
{
"caseId": "chat-diagnosis-flow",
"query": "What is the standard troubleshooting flow for an application incident?",
"retrievedAt": "2026-07-06T00:00:00Z",
"lookupResult": {
"found": true,
"evidenceBlocks": [
{
"source": "incident-diagnosis-flow",
"title": "Incident Diagnosis Flow",
"breadcrumb": "AIOps > Diagnosis Flow",
"retrievalLayer": "L1",
"content": "The standard flow is to collect evidence, identify the suspected fault domain, verify the hypothesis, apply remediation, and confirm recovery.",
"score": 0.82,
"hitReasons": ["domain_match:+0.15", "keyword_match:+0.10"]
},
{
"source": "rag-chunk-context-reconstruction",
"title": "RAG Chunk Context Reconstruction",
"breadcrumb": "RAG > Chunking > Context Reconstruction",
"retrievalLayer": "L1",
"content": "Long sections may require neighbor chunk expansion and breadcrumb-aware packing.",
"score": 0.55,
"hitReasons": []
}
],
"contextPack": {
"packedText": "[1] Incident Diagnosis Flow\nAIOps > Diagnosis Flow\nThe standard flow is to collect evidence, identify the suspected fault domain, verify the hypothesis, apply remediation, and confirm recovery.",
"strategy": "top_evidence_blocks",
"charBudget": 3500,
"usedChars": 192,
"includedSources": ["incident-diagnosis-flow", "rag-chunk-context-reconstruction"],
"omittedSources": []
},
"retrievalTrace": {
"originalQuery": "What is the standard troubleshooting flow for an application incident?",
"rewrittenQuery": "standard application incident troubleshooting flow collect evidence verify remediation",
"categoryFilter": "AIOps",
"selectedAttempt": "FILTERED_VECTOR",
"fallbackReason": null,
"evidenceStatus": "supported",
"queryHints": {
"domains": ["AIOps"],
"matched_keywords": ["collect evidence", "verify", "remediation"],
"entities": ["application incident"],
"l0_titles": ["Incident Diagnosis Flow"],
"l0_match_count": 1
},
"attempts": [
{
"name": "FILTERED_VECTOR",
"query": "standard application incident troubleshooting flow collect evidence verify remediation",
"categoryFilter": "AIOps",
"candidateCount": 2,
"usable": true,
"durationMs": 10,
"topScore": 0.82,
"topSimilarity": 0.82
}
]
},
"rerankTrace": {
"items": [
{
"finalRank": 1,
"source": "incident-diagnosis-flow",
"baseScore": 0.82,
"finalScore": 1.07,
"boostReasons": ["domain_match:+0.15", "keyword_match:+0.10"]
},
{
"finalRank": 2,
"source": "rag-chunk-context-reconstruction",
"baseScore": 0.55,
"finalScore": 0.55,
"boostReasons": []
}
]
}
}
}
@@ -0,0 +1,81 @@
{
"caseId": "chat-l0-domain-hint",
"query": "Should L0 keyword matching decide the final retrieval result?",
"retrievedAt": "2026-07-06T00:00:00Z",
"lookupResult": {
"found": true,
"evidenceBlocks": [
{
"source": "rag-l0-domain-entity-hint",
"title": "RAG L0 Domain Entity Hint",
"breadcrumb": "RAG > L0 > Domain Entity Hint",
"retrievalLayer": "L1",
"content": "L0 should be retained as a domain detector, entity extractor, metadata filter generator, and explainability signal, not as the final retrieval decision.",
"score": 0.88,
"hitReasons": ["domain_match:+0.15", "keyword_match:+0.10"]
},
{
"source": "rag-l0-l1-fusion-ranking",
"title": "RAG L0 L1 Fusion Ranking",
"breadcrumb": "RAG > Ranking > Fusion",
"retrievalLayer": "L1",
"content": "L0 and L1 candidates should eventually be fused rather than handled as an early-return branch.",
"score": 0.75,
"hitReasons": ["domain_match:+0.15"]
}
],
"contextPack": {
"packedText": "[1] RAG L0 Domain Entity Hint\nRAG > L0 > Domain Entity Hint\nL0 should be retained as a domain detector, entity extractor, metadata filter generator, and explainability signal, not as the final retrieval decision.",
"strategy": "top_evidence_blocks",
"charBudget": 3500,
"usedChars": 219,
"includedSources": ["rag-l0-domain-entity-hint", "rag-l0-l1-fusion-ranking"],
"omittedSources": []
},
"retrievalTrace": {
"originalQuery": "Should L0 keyword matching decide the final retrieval result?",
"rewrittenQuery": "RAG L0 keyword matching domain entity hint final retrieval decision",
"categoryFilter": "RAG",
"selectedAttempt": "FILTERED_VECTOR",
"fallbackReason": null,
"evidenceStatus": "supported",
"queryHints": {
"domains": ["RAG"],
"matched_keywords": ["domain detector", "entity extractor", "metadata filter"],
"entities": ["L0"],
"l0_titles": ["RAG L0 Domain Entity Hint"],
"l0_match_count": 1
},
"attempts": [
{
"name": "FILTERED_VECTOR",
"query": "RAG L0 keyword matching domain entity hint final retrieval decision",
"categoryFilter": "RAG",
"candidateCount": 2,
"usable": true,
"durationMs": 9,
"topScore": 0.88,
"topSimilarity": 0.88
}
]
},
"rerankTrace": {
"items": [
{
"finalRank": 1,
"source": "rag-l0-domain-entity-hint",
"baseScore": 0.88,
"finalScore": 1.13,
"boostReasons": ["domain_match:+0.15", "keyword_match:+0.10"]
},
{
"finalRank": 2,
"source": "rag-l0-l1-fusion-ranking",
"baseScore": 0.75,
"finalScore": 0.9,
"boostReasons": ["domain_match:+0.15"]
}
]
}
}
}
@@ -0,0 +1,91 @@
{
"caseId": "chat-l0-filter-fallback",
"query": "RAG query was over-filtered by L0 and filtered vector search returned low quality evidence. What should happen?",
"retrievedAt": "2026-07-06T00:00:00Z",
"lookupResult": {
"found": true,
"evidenceBlocks": [
{
"source": "rag-l0-filter-fallback",
"title": "RAG L0 Filter Fallback",
"breadcrumb": "RAG > Fallback > Unfiltered Retry",
"retrievalLayer": "L1",
"content": "When filtered vector retrieval is low quality, skip the L0 filter and run an unfiltered vector retry with the raw query before returning no evidence.",
"score": 0.83,
"hitReasons": ["domain_match:+0.15", "keyword_match:+0.10"]
},
{
"source": "rag-l0-domain-entity-hint",
"title": "RAG L0 Domain Entity Hint",
"breadcrumb": "RAG > L0 > Domain Entity Hint",
"retrievalLayer": "L1",
"content": "L0 supplies hints for metadata filtering and explanation, but it should not be treated as final fact evidence.",
"score": 0.66,
"hitReasons": ["domain_match:+0.15"]
}
],
"contextPack": {
"packedText": "[1] RAG L0 Filter Fallback\nRAG > Fallback > Unfiltered Retry\nWhen filtered vector retrieval is low quality, skip the L0 filter and run an unfiltered vector retry with the raw query before returning no evidence.",
"strategy": "top_evidence_blocks",
"charBudget": 3500,
"usedChars": 214,
"includedSources": ["rag-l0-filter-fallback", "rag-l0-domain-entity-hint"],
"omittedSources": []
},
"retrievalTrace": {
"originalQuery": "RAG query was over-filtered by L0 and filtered vector search returned low quality evidence. What should happen?",
"rewrittenQuery": "RAG L0 filtered vector low quality fallback unfiltered retry",
"categoryFilter": "RAG",
"selectedAttempt": "UNFILTERED_VECTOR_RETRY",
"fallbackReason": "filtered_vector_low_quality",
"evidenceStatus": "supported",
"queryHints": {
"domains": ["RAG"],
"matched_keywords": ["L0", "low quality", "unfiltered vector retry"],
"entities": ["L0"],
"l0_titles": ["RAG L0 Domain Entity Hint"],
"l0_match_count": 1
},
"attempts": [
{
"name": "FILTERED_VECTOR",
"query": "RAG L0 filtered vector low quality fallback unfiltered retry",
"categoryFilter": "RAG",
"candidateCount": 1,
"usable": false,
"durationMs": 7,
"topScore": 1.35,
"topSimilarity": 0.325
},
{
"name": "UNFILTERED_VECTOR_RETRY",
"query": "RAG query was over-filtered by L0 and filtered vector search returned low quality evidence. What should happen?",
"categoryFilter": null,
"candidateCount": 2,
"usable": true,
"durationMs": 13,
"topScore": 0.83,
"topSimilarity": 0.83
}
]
},
"rerankTrace": {
"items": [
{
"finalRank": 1,
"source": "rag-l0-filter-fallback",
"baseScore": 0.83,
"finalScore": 1.08,
"boostReasons": ["domain_match:+0.15", "keyword_match:+0.10"]
},
{
"finalRank": 2,
"source": "rag-l0-domain-entity-hint",
"baseScore": 0.66,
"finalScore": 0.81,
"boostReasons": ["domain_match:+0.15"]
}
]
}
}
}
@@ -0,0 +1,81 @@
{
"caseId": "chat-mysql-connection-pool",
"query": "MySQL connection pool is exhausted. How should I diagnose it?",
"retrievedAt": "2026-07-06T00:00:00Z",
"lookupResult": {
"found": true,
"evidenceBlocks": [
{
"source": "mysql-connection-pool",
"title": "MySQL Connection Pool Troubleshooting",
"breadcrumb": "Database > MySQL > Connection Pool",
"retrievalLayer": "L1",
"content": "When the connection pool is exhausted, inspect HikariCP active connections, max_connections, slow SQL, leak detection, and database wait events.",
"score": 0.86,
"hitReasons": ["domain_match:+0.15", "keyword_match:+0.10"]
},
{
"source": "incident-diagnosis-flow",
"title": "Incident Diagnosis Flow",
"breadcrumb": "AIOps > Diagnosis Flow",
"retrievalLayer": "L1",
"content": "Collect evidence, compare metrics and logs, then verify remediation before closing the incident.",
"score": 0.61,
"hitReasons": []
}
],
"contextPack": {
"packedText": "[1] MySQL Connection Pool Troubleshooting\nDatabase > MySQL > Connection Pool\nWhen the connection pool is exhausted, inspect HikariCP active connections, max_connections, slow SQL, leak detection, and database wait events.",
"strategy": "top_evidence_blocks",
"charBudget": 3500,
"usedChars": 216,
"includedSources": ["mysql-connection-pool", "incident-diagnosis-flow"],
"omittedSources": []
},
"retrievalTrace": {
"originalQuery": "MySQL connection pool is exhausted. How should I diagnose it?",
"rewrittenQuery": "MySQL connection pool exhausted HikariCP max_connections diagnosis",
"categoryFilter": "Database",
"selectedAttempt": "FILTERED_VECTOR",
"fallbackReason": null,
"evidenceStatus": "supported",
"queryHints": {
"domains": ["Database", "MySQL"],
"matched_keywords": ["connection pool", "HikariCP", "max_connections"],
"entities": ["MySQL", "HikariCP"],
"l0_titles": ["MySQL Connection Pool Troubleshooting"],
"l0_match_count": 1
},
"attempts": [
{
"name": "FILTERED_VECTOR",
"query": "MySQL connection pool exhausted HikariCP max_connections diagnosis",
"categoryFilter": "Database",
"candidateCount": 2,
"usable": true,
"durationMs": 12,
"topScore": 0.86,
"topSimilarity": 0.86
}
]
},
"rerankTrace": {
"items": [
{
"finalRank": 1,
"source": "mysql-connection-pool",
"baseScore": 0.86,
"finalScore": 1.11,
"boostReasons": ["domain_match:+0.15", "keyword_match:+0.10"]
},
{
"finalRank": 2,
"source": "incident-diagnosis-flow",
"baseScore": 0.61,
"finalScore": 0.61,
"boostReasons": []
}
]
}
}
}
@@ -0,0 +1,81 @@
{
"caseId": "chat-rag-chunk-context",
"query": "If a long section is split into multiple chunks, how do we keep retrieval context?",
"retrievedAt": "2026-07-06T00:00:00Z",
"lookupResult": {
"found": true,
"evidenceBlocks": [
{
"source": "rag-chunk-context-reconstruction",
"title": "RAG Chunk Context Reconstruction",
"breadcrumb": "RAG > Chunking > Context Reconstruction",
"retrievalLayer": "L1",
"content": "After a chunk hit, expand to neighbor chunk candidates from the same section and preserve breadcrumb metadata in the evidence pack.",
"score": 0.79,
"hitReasons": ["domain_match:+0.15", "keyword_match:+0.10"]
},
{
"source": "rag-breadcrumb-embedding-gap",
"title": "RAG Breadcrumb Embedding Gap",
"breadcrumb": "RAG > Embedding > Breadcrumb",
"retrievalLayer": "L1",
"content": "Embedding title and breadcrumb with content helps recover section semantics.",
"score": 0.72,
"hitReasons": ["domain_match:+0.15"]
}
],
"contextPack": {
"packedText": "[1] RAG Chunk Context Reconstruction\nRAG > Chunking > Context Reconstruction\nAfter a chunk hit, expand to neighbor chunk candidates from the same section and preserve breadcrumb metadata in the evidence pack.",
"strategy": "top_evidence_blocks",
"charBudget": 3500,
"usedChars": 203,
"includedSources": ["rag-chunk-context-reconstruction", "rag-breadcrumb-embedding-gap"],
"omittedSources": []
},
"retrievalTrace": {
"originalQuery": "If a long section is split into multiple chunks, how do we keep retrieval context?",
"rewrittenQuery": "RAG chunk context reconstruction neighbor chunk same section breadcrumb",
"categoryFilter": "RAG",
"selectedAttempt": "FILTERED_VECTOR",
"fallbackReason": null,
"evidenceStatus": "supported",
"queryHints": {
"domains": ["RAG"],
"matched_keywords": ["neighbor chunk", "same section", "breadcrumb"],
"entities": ["chunk", "breadcrumb"],
"l0_titles": ["RAG Chunk Context Reconstruction"],
"l0_match_count": 1
},
"attempts": [
{
"name": "FILTERED_VECTOR",
"query": "RAG chunk context reconstruction neighbor chunk same section breadcrumb",
"categoryFilter": "RAG",
"candidateCount": 2,
"usable": true,
"durationMs": 9,
"topScore": 0.79,
"topSimilarity": 0.79
}
]
},
"rerankTrace": {
"items": [
{
"finalRank": 1,
"source": "rag-chunk-context-reconstruction",
"baseScore": 0.79,
"finalScore": 1.04,
"boostReasons": ["domain_match:+0.15", "keyword_match:+0.10"]
},
{
"finalRank": 2,
"source": "rag-breadcrumb-embedding-gap",
"baseScore": 0.72,
"finalScore": 0.87,
"boostReasons": ["domain_match:+0.15"]
}
]
}
}
}
+1
View File
@@ -0,0 +1 @@
+223
View File
@@ -0,0 +1,223 @@
{
"generatedAt": "2026-07-06T13:37:59.726351+00:00",
"caseFile": "eval/rag-retrieval/cases/golden-cases.json",
"fixtureDir": "eval/rag-retrieval/fixtures",
"aggregate": {
"caseCount": 7,
"topK": 5,
"passedCount": 7,
"failedCount": 0,
"passRate": 1.0,
"lookupResultCaseCount": 7,
"strongHitCount": 7,
"mediumHitCount": 0,
"weakHitCount": 0,
"missCount": 0,
"recallAtK": 1.0,
"strongHitRate": 1.0,
"averageFirstHitRank": 1.0
},
"results": [
{
"caseId": "chat-mysql-connection-pool",
"scenario": "chat",
"query": "MySQL connection pool is exhausted. How should I diagnose it?",
"dataShape": "lookupResult",
"hitLevel": "strong",
"passed": true,
"firstExpectedRank": 1,
"topCandidates": [
"1:mysql-connection-pool",
"2:incident-diagnosis-flow"
],
"matchedKeywords": [
"connection pool",
"max_connections",
"hikaricp"
],
"breadcrumbMatched": true,
"selectedAttempt": "FILTERED_VECTOR",
"fallbackReason": null,
"evidenceStatus": "supported",
"includedSources": [
"mysql-connection-pool",
"incident-diagnosis-flow"
],
"omittedSources": [],
"rerankTopSource": "mysql-connection-pool",
"failedChecks": []
},
{
"caseId": "chat-diagnosis-flow",
"scenario": "chat",
"query": "What is the standard troubleshooting flow for an application incident?",
"dataShape": "lookupResult",
"hitLevel": "strong",
"passed": true,
"firstExpectedRank": 1,
"topCandidates": [
"1:incident-diagnosis-flow",
"2:rag-chunk-context-reconstruction"
],
"matchedKeywords": [
"collect evidence",
"verify",
"remediation"
],
"breadcrumbMatched": true,
"selectedAttempt": "FILTERED_VECTOR",
"fallbackReason": null,
"evidenceStatus": "supported",
"includedSources": [
"incident-diagnosis-flow",
"rag-chunk-context-reconstruction"
],
"omittedSources": [],
"rerankTopSource": "incident-diagnosis-flow",
"failedChecks": []
},
{
"caseId": "aiops-payment-latency-alert",
"scenario": "aiops",
"query": "Alert HighLatency on payment-service with p95 latency above threshold",
"dataShape": "lookupResult",
"hitLevel": "strong",
"passed": true,
"firstExpectedRank": 1,
"topCandidates": [
"1:payment-service-latency",
"2:mysql-connection-pool"
],
"matchedKeywords": [
"p95 latency",
"payment-service",
"downstream dependency"
],
"breadcrumbMatched": true,
"selectedAttempt": "FILTERED_VECTOR",
"fallbackReason": null,
"evidenceStatus": "supported",
"includedSources": [
"payment-service-latency",
"mysql-connection-pool"
],
"omittedSources": [],
"rerankTopSource": "payment-service-latency",
"failedChecks": []
},
{
"caseId": "aiops-prometheus-alert-scope",
"scenario": "aiops",
"query": "When an AIOps request already includes alert payload, should the agent diagnose unrelated active alerts?",
"dataShape": "lookupResult",
"hitLevel": "strong",
"passed": true,
"firstExpectedRank": 1,
"topCandidates": [
"1:aiops-alert-scope-control"
],
"matchedKeywords": [
"payload",
"unrelated active alerts",
"scope"
],
"breadcrumbMatched": true,
"selectedAttempt": "FILTERED_VECTOR",
"fallbackReason": null,
"evidenceStatus": "supported",
"includedSources": [
"aiops-alert-scope-control"
],
"omittedSources": [],
"rerankTopSource": "aiops-alert-scope-control",
"failedChecks": []
},
{
"caseId": "chat-rag-chunk-context",
"scenario": "chat",
"query": "If a long section is split into multiple chunks, how do we keep retrieval context?",
"dataShape": "lookupResult",
"hitLevel": "strong",
"passed": true,
"firstExpectedRank": 1,
"topCandidates": [
"1:rag-chunk-context-reconstruction",
"2:rag-breadcrumb-embedding-gap"
],
"matchedKeywords": [
"neighbor chunk",
"same section",
"breadcrumb"
],
"breadcrumbMatched": true,
"selectedAttempt": "FILTERED_VECTOR",
"fallbackReason": null,
"evidenceStatus": "supported",
"includedSources": [
"rag-chunk-context-reconstruction",
"rag-breadcrumb-embedding-gap"
],
"omittedSources": [],
"rerankTopSource": "rag-chunk-context-reconstruction",
"failedChecks": []
},
{
"caseId": "chat-l0-domain-hint",
"scenario": "chat",
"query": "Should L0 keyword matching decide the final retrieval result?",
"dataShape": "lookupResult",
"hitLevel": "strong",
"passed": true,
"firstExpectedRank": 1,
"topCandidates": [
"1:rag-l0-domain-entity-hint",
"2:rag-l0-l1-fusion-ranking"
],
"matchedKeywords": [
"domain detector",
"entity extractor",
"metadata filter"
],
"breadcrumbMatched": true,
"selectedAttempt": "FILTERED_VECTOR",
"fallbackReason": null,
"evidenceStatus": "supported",
"includedSources": [
"rag-l0-domain-entity-hint",
"rag-l0-l1-fusion-ranking"
],
"omittedSources": [],
"rerankTopSource": "rag-l0-domain-entity-hint",
"failedChecks": []
},
{
"caseId": "chat-l0-filter-fallback",
"scenario": "chat",
"query": "RAG query was over-filtered by L0 and filtered vector search returned low quality evidence. What should happen?",
"dataShape": "lookupResult",
"hitLevel": "strong",
"passed": true,
"firstExpectedRank": 1,
"topCandidates": [
"1:rag-l0-filter-fallback",
"2:rag-l0-domain-entity-hint"
],
"matchedKeywords": [
"skip the l0 filter",
"unfiltered vector retry",
"low quality"
],
"breadcrumbMatched": true,
"selectedAttempt": "UNFILTERED_VECTOR_RETRY",
"fallbackReason": "filtered_vector_low_quality",
"evidenceStatus": "supported",
"includedSources": [
"rag-l0-filter-fallback",
"rag-l0-domain-entity-hint"
],
"omittedSources": [],
"rerankTopSource": "rag-l0-filter-fallback",
"failedChecks": []
}
]
}
+33
View File
@@ -0,0 +1,33 @@
# RAG Retrieval Baseline
Generated at: `2026-07-06T13:37:59.726351+00:00`
## Aggregate
| Metric | Value |
|---|---:|
| Cases | 7 |
| Top K | 5 |
| Passed | 7 |
| Failed | 0 |
| Pass rate | 1.0 |
| LookupResult fixtures | 7 |
| Recall@K | 1.0 |
| Strong hit rate | 1.0 |
| Strong hits | 7 |
| Medium hits | 0 |
| Weak hits | 0 |
| Misses | 0 |
| Average first hit rank | 1.0 |
## Cases
| Case | Scenario | Pass | Hit | Attempt | Fallback | Evidence | First Expected Rank | Top Candidates | Failed Checks |
|---|---|---|---|---|---|---|---:|---|---|
| chat-mysql-connection-pool | chat | true | strong | FILTERED_VECTOR | | supported | 1 | 1:mysql-connection-pool<br>2:incident-diagnosis-flow | |
| chat-diagnosis-flow | chat | true | strong | FILTERED_VECTOR | | supported | 1 | 1:incident-diagnosis-flow<br>2:rag-chunk-context-reconstruction | |
| aiops-payment-latency-alert | aiops | true | strong | FILTERED_VECTOR | | supported | 1 | 1:payment-service-latency<br>2:mysql-connection-pool | |
| aiops-prometheus-alert-scope | aiops | true | strong | FILTERED_VECTOR | | supported | 1 | 1:aiops-alert-scope-control | |
| chat-rag-chunk-context | chat | true | strong | FILTERED_VECTOR | | supported | 1 | 1:rag-chunk-context-reconstruction<br>2:rag-breadcrumb-embedding-gap | |
| chat-l0-domain-hint | chat | true | strong | FILTERED_VECTOR | | supported | 1 | 1:rag-l0-domain-entity-hint<br>2:rag-l0-l1-fusion-ranking | |
| chat-l0-filter-fallback | chat | true | strong | UNFILTERED_VECTOR_RETRY | filtered_vector_low_quality | supported | 1 | 1:rag-l0-filter-fallback<br>2:rag-l0-domain-entity-hint | |
@@ -0,0 +1,26 @@
---
title: AIOps Alert Scope Control
keywords: [alert payload, unrelated active alerts, scope control]
summary: Keep diagnosis scoped to the request payload and avoid diagnosing unrelated active alerts.
category: aiops
source: aiops-alert-scope-control
breadcrumb: AIOps > Alert Scope Control
kb_scope: rag-eval
covers: [alert scope, payload, active alerts]
when_to_retrieve: Use when an AIOps request includes a concrete alert payload and scope boundaries matter.
---
# AIOps
## Alert Scope Control
When an AIOps request already includes an alert payload, the agent should diagnose that payload first.
It must not expand the task into unrelated active alerts unless the user asks for broad alert triage.
Scope rules:
1. Treat the provided payload as the primary incident boundary.
2. Use unrelated active alerts only as correlation evidence when they share service, dependency, time window, or trace context.
3. Do not replace the requested alert with a louder but unrelated alert.
This runbook anchors payload, unrelated active alerts, and scope behavior.
@@ -0,0 +1,27 @@
---
title: Incident Diagnosis Flow
keywords: [standard troubleshooting flow, application incident, collect evidence, verify, remediation]
summary: Standard flow for diagnosing application incidents with evidence, hypothesis verification, and remediation.
category: ops
source: incident-diagnosis-flow
breadcrumb: AIOps > Diagnosis Flow
kb_scope: rag-eval
covers: [incident diagnosis, evidence collection, remediation]
when_to_retrieve: Use when the user asks for a standard troubleshooting flow or incident diagnosis sequence.
---
# AIOps
## Diagnosis Flow
The standard troubleshooting flow is evidence first, hypothesis second, remediation last.
Recommended sequence:
1. Collect evidence from alerts, metrics, logs, traces, deployments, and recent configuration changes.
2. Define a small hypothesis that explains the observed symptoms.
3. Verify the hypothesis with a targeted metric, log query, or reproduction step.
4. Choose remediation that directly addresses the verified cause.
5. Record the outcome and the evidence used to make the decision.
Do not skip collect evidence, verify, and remediation ordering during an application incident.
@@ -0,0 +1,29 @@
---
title: MySQL Connection Pool Runbook
keywords: [MySQL connection pool, pool exhausted, max_connections, HikariCP]
summary: Diagnose exhausted MySQL connection pools and distinguish application leaks from database limits.
category: database
source: mysql-connection-pool
breadcrumb: Database > MySQL > Connection Pool
kb_scope: rag-eval
covers: [mysql, connection pool, database capacity]
when_to_retrieve: Use when MySQL clients report exhausted pools, connection acquisition timeout, max_connections pressure, or HikariCP saturation.
---
# Database
## MySQL
### Connection Pool
When MySQL connection pool is exhausted, first compare application pool usage with database `max_connections`.
For HikariCP, check `active`, `idle`, `pending`, and connection acquisition timeout metrics.
Recommended diagnosis:
1. Verify whether HikariCP active connections stay near maximum while pending threads grow.
2. Check MySQL `Threads_connected`, `Threads_running`, and `max_connections`.
3. Inspect slow SQL and long transactions that keep connections checked out.
4. If the database is healthy, look for application connection leaks or missing transaction boundaries.
Use this runbook as evidence for connection pool, max_connections, and HikariCP incidents.
@@ -0,0 +1,28 @@
---
title: Payment Service Latency Alert
keywords: [HighLatency, payment-service, p95 latency, downstream dependency]
summary: Diagnose payment-service p95 latency alerts and identify downstream dependency bottlenecks.
category: aiops
source: payment-service-latency
breadcrumb: AIOps > Service Alerts > Payment Latency
kb_scope: rag-eval
covers: [payment-service, latency, downstream dependency]
when_to_retrieve: Use when an alert mentions payment-service, HighLatency, or elevated p95 latency.
---
# AIOps
## Service Alerts
### Payment Latency
For `HighLatency` alerts on `payment-service`, treat p95 latency as the primary symptom.
Diagnosis steps:
1. Confirm whether p95 latency is isolated to payment-service or shared across upstream callers.
2. Compare payment-service latency with downstream dependency latency for gateway, risk, and order services.
3. Check connection pool wait time, retry spikes, and timeout rates.
4. If downstream dependency latency increased first, classify payment-service as affected rather than root cause.
The expected evidence terms are p95 latency, payment-service, and downstream dependency.
@@ -0,0 +1,28 @@
---
title: RAG Chunk Context Reconstruction
keywords: [split into multiple chunks, retrieval context, neighbor chunk, same section, breadcrumb context]
summary: Preserve context when long RAG sections are split into multiple retrievable chunks.
category: rag
source: rag-chunk-context-reconstruction
breadcrumb: RAG > Chunking > Context Reconstruction
kb_scope: rag-eval
covers: [rag chunking, context packing, breadcrumbs]
when_to_retrieve: Use when a retrieval question asks how to preserve context across split chunks.
---
# RAG
## Chunking
### Context Reconstruction
When a long section is split into multiple chunks, retrieval should keep enough local structure for the answer.
Recommended behavior:
1. Store the breadcrumb with every chunk.
2. Preserve the same section identity across adjacent chunks.
3. During context packing, include a neighbor chunk when the selected chunk depends on nearby setup or definitions.
4. Prefer concise evidence blocks that show the breadcrumb and the relevant content span.
The key concepts are neighbor chunk, same section, and breadcrumb.
@@ -0,0 +1,29 @@
---
title: RAG L0 Domain Entity Hint
keywords: [L0 keyword matching, final retrieval result, domain detector, entity extractor, metadata filter]
summary: Define L0 as a query transformation hint layer instead of final retrieval evidence.
category: rag
source: rag-l0-domain-entity-hint
breadcrumb: RAG > L0 > Domain Entity Hint
kb_scope: rag-eval
covers: [l0 hint, query transformation, metadata filter]
when_to_retrieve: Use when a question asks whether L0 should decide final retrieval or only provide hints.
---
# RAG
## L0
### Domain Entity Hint
L0 keyword matching should not decide the final retrieval result.
In the modular RAG pipeline, L0 behaves like a lightweight domain detector and entity extractor.
The output can provide:
1. Candidate domain hints.
2. Matched entities and keywords.
3. An optional metadata filter for the first vector retrieval attempt.
Final evidence still comes from L1 vector retrieval, post-retrieval normalization, rerank, and context packing.
The important terms are domain detector, entity extractor, and metadata filter.
@@ -0,0 +1,19 @@
---
title: RAG L0 Filter Decoy
keywords: [over-filtered by L0, filtered vector search, low quality evidence]
summary: Decoy document used to force the first filtered retrieval attempt into a low-quality category.
category: overfilter-decoy
source: rag-l0-filter-decoy
breadcrumb: RAG > Fallback > Decoy
kb_scope: rag-eval
covers: [fallback test decoy]
when_to_retrieve: Use only as a controlled eval decoy for over-filter fallback testing.
---
# Release Calendar
## Approval Window
This document describes an unrelated release calendar approval window.
It intentionally avoids the real fallback instructions so the filtered retrieval
attempt is low quality and the retriever must retry without the L0 category filter.
@@ -0,0 +1,25 @@
---
title: RAG L0 Filter Fallback
keywords: [golden retry contract, second pass retrieval]
summary: Retry the raw query without the L0 category filter when filtered vector evidence is missing or low quality.
category: fallback
source: rag-l0-filter-fallback
breadcrumb: RAG > Fallback > Unfiltered Retry
kb_scope: rag-eval
covers: [fallback, unfiltered retry, retrieval quality]
when_to_retrieve: Use when validating the fallback contract for low-quality filtered vector retrieval.
---
# RAG
## Fallback
### Unfiltered Retry
If the first vector search is over-constrained by an L0 metadata filter and returns low quality evidence,
the retriever should skip the L0 filter and run an unfiltered vector retry with the original query.
The fallback reason should be `filtered_vector_low_quality` when the filtered candidate exists but is below the
reference threshold. If there is no usable evidence at all, use `filtered_vector_no_evidence`.
This document is the expected evidence for skip the L0 filter, unfiltered vector retry, and low quality behavior.
+66
View File
@@ -0,0 +1,66 @@
# SuperBizAgent 面试资料包
## 一句话定位
SuperBizAgent 是一个面向企业故障诊断场景的 Agent 工程项目。它把用户问题或 AIOps 告警转换成可追踪的 Agent 执行链路,并把工具证据、模型步骤、最终答案、自评估和用户反馈统一沉淀到诊断 Trace 中。
## 面试重点
- **Agent 编排**:Chat 复杂问题走 `Planner -> Executor -> Verifier`;AIOps 告警入口走 `Supervisor -> Planner / Executor`。
- **工具证据链**:知识库、日志、指标、Prometheus 告警都通过显式工具调用进入链路,并记录到 `tool_invocation`。
- **可追踪诊断**:一次诊断对应一个 `sessionId`,可通过 `GET /api/diagnosis/{sessionId}/trace` 回放。
- **质量门禁**:Chat Verifier 校验 groundedness;AIOps 规则评估检查报告完整性、payload 聚焦和证据工具覆盖。
- **RAG 工程化**:`lookup_knowledge` 是显式 Agent Tool,底层通过 Spring AI VectorStore 主路径 + Milvus SDK fallback。
- **反馈闭环**:用户反馈 `useful` 会沉淀 `case_library`,`not_useful` 保留 bad case 信号。
## 推荐阅读顺序
1. `mvp/architecture/interview-one-pager.md`:一页式架构图和 2-5 分钟讲解。
2. `mvp/demo/ten-minute-interview-demo.md`:10 分钟现场演示脚本。
3. `interview/story-cases.md`:可复用的面试故事案例。
4. `interview/architecture.md`:面试版系统架构。
5. `interview/design-tradeoffs.md`:关键设计取舍。
6. `interview/demo-script.md`:更细的命令式演示脚本。
7. `interview/acceptance-checklist.md`:面试前验收清单。
8. RAG 专题文档:`rag-refactor-story.md`、`rag-vectorstore-interview-notes.md`、`rag-retrieval-quality-report.md`。
## 核心演示链路
### Chat 诊断
```text
POST /api/chat
-> ChatService
-> Planner -> Executor -> Verifier
-> lookup_knowledge / query_logs / query_metrics
-> diagnosis_session + agent_step + tool_invocation
-> GET /api/diagnosis/{sessionId}/trace
-> POST /api/feedback
```
### AIOps 告警诊断
```text
POST /api/ai_ops
-> AiOpsService
-> PAYLOAD_TARGETED / AUTO_DISCOVERY
-> ai_ops_supervisor
-> planner_agent / executor_agent
-> queryPrometheusAlerts + logs + metrics + lookup_knowledge
-> alert report
-> aiops_rule_evaluation
-> GET /api/diagnosis/{sessionId}/trace
```
## 当前完成度
- Chat 诊断链路:可运行、可追踪、有 Verifier。
- AIOps 告警链路:可运行、可追踪、支持 payload scope control。
- RAG 检索链路:Spring AI VectorStore 主路径、Milvus SDK fallback、L0 hint、检索评测 baseline。
- Trace API:统一返回 session、agent steps、tool invocations 和 summary。
- Demo 材料:`mvp/demo/README.md`、`mvp/demo/ten-minute-interview-demo.md`。
## 主叙事
这个项目不是简单调用大模型,而是在做一个可审计、可验证、可回归的 Agent 诊断系统。模型可以规划和推理,但每一步工具证据、最终结论、Verifier 结果和用户反馈都能被 Trace API 回放。面试时重点展示“从问题到证据到答案到验证再到反馈”的闭环。
+166
View File
@@ -0,0 +1,166 @@
# 面试前验收清单
## 1. 环境检查
- 当前分支包含最新架构文档和面试材料。
- MySQL 可连接。
- Redis 可连接。
- Milvus/Zilliz 可连接。
- 模型 API key 可用。
- `mvp-demo` profile 开启 mock Prometheus 和 mock CLS。
启动服务:
```powershell
mvn spring-boot:run "-Dspring-boot.run.profiles=mvp-demo"
```
编译检查:
```powershell
mvn -q -DskipTests compile
```
目标测试:
```powershell
mvn -q "-Dtest=AiOpsServiceTest,ChatServiceSequentialAgentTest,DiagnosisTraceServiceTest,VectorSearchServiceTest,LookupKnowledgeToolTest" test
```
## 2. Chat Demo 验收
请求:
```powershell
$sessionId = "interview-chat-payment-timeout-001"
$body = @{
Id = $sessionId
Question = "支付接口最近出现超时,请结合知识库、日志和指标判断可能原因,并给出修复建议。"
} | ConvertTo-Json
Invoke-RestMethod `
-Method Post `
-Uri "http://localhost:9900/api/chat" `
-ContentType "application/json" `
-Body $body
```
验收:
- 返回 `data.success = true`。
- 返回 `data.sessionId = interview-chat-payment-timeout-001`。
- `diagnosis_session.agent_flow = CHAT`。
- Trace API 返回 session、steps、toolInvocations。
- 复杂问题下 trace 中能看到 verifier 相关数据。
SQL:
```powershell
python scripts/query_mysql.py "SELECT session_id, agent_flow, status, step_count, tool_call_count FROM diagnosis_session WHERE session_id='interview-chat-payment-timeout-001'"
```
## 3. AIOps Demo 验收
请求:
```powershell
$aiopsSessionId = "interview-aiops-payment-cpu-001"
$aiopsBody = @{
sessionId = $aiopsSessionId
alertName = "HighCPUUsage"
service = "payment-service"
severity = "P1"
description = "服务 payment-service 的 CPU 使用率持续超过 80%,当前值为 92%。实例: pod-payment-service-7d8f9c6b5-x2k4m。"
timeRange = "last_15m"
userRequest = "请结合 Prometheus 活动告警、system-metrics 日志和知识库生成告警分析报告。"
} | ConvertTo-Json
Invoke-WebRequest `
-Method Post `
-Uri "http://localhost:9900/api/ai_ops" `
-ContentType "application/json" `
-Body $aiopsBody
```
验收:
- SSE 首条包含 `type=session`。
- SSE 最后包含 `type=done`。
- `diagnosis_session.agent_flow = AI_OPS`。
- `diagnosis_session.status = SUCCESS`。
- `diagnosis_session.answer` 有最终报告。
- Trace API 返回 AIOps steps 和 tool invocations。
- 报告主章节聚焦 `HighCPUUsage/payment-service`。
- 其他 active alerts 不应展开成独立主根因章节。
- `self_evaluation.aiops_rule_evaluation` 存在。
SQL:
```powershell
python scripts/query_mysql.py "SELECT session_id, agent_flow, status, total_duration_ms, step_count, tool_call_count FROM diagnosis_session WHERE session_id='interview-aiops-payment-cpu-001'"
```
```powershell
python scripts/query_mysql.py "SELECT tool_name, COUNT(*) AS cnt FROM tool_invocation WHERE session_id='interview-aiops-payment-cpu-001' GROUP BY tool_name ORDER BY tool_name"
```
Scope 检查:
```powershell
python scripts/query_mysql.py "SELECT (answer LIKE '%HighCPUUsage%') AS has_main_alert, (answer LIKE '%payment-service%') AS has_service FROM diagnosis_session WHERE session_id='interview-aiops-payment-cpu-001'"
```
## 4. Trace API 验收
```powershell
Invoke-RestMethod `
-Method Get `
-Uri "http://localhost:9900/api/diagnosis/interview-aiops-payment-cpu-001/trace"
```
如果 PowerShell 对长 JSON 或特殊字符不稳定,可以用:
```powershell
curl.exe --silent --show-error --max-time 60 "http://localhost:9900/api/diagnosis/interview-aiops-payment-cpu-001/trace"
```
## 5. RAG 验收
```powershell
Invoke-RestMethod `
-Uri "http://127.0.0.1:9900/api/search/similar?query=ERR_TIMEOUT&topK=3" `
-Method Get
```
验收:
- 返回 `code = 200`。
- top candidates 中包含 `ERR_TIMEOUT` 相关文档。
- `scoreLabel` 能体现当前检索路径语义。
- 如果走 VectorStore,日志应出现 Spring AI VectorStore search。
## 6. 常见问题
### MySQL stale connection
现象:
```text
HikariPool - Connection is not available
No operations allowed after connection closed
```
处理:
- 重启服务。
- 确认 HikariPool 使用当前配置启动成功。
- 再跑 trace 或 AIOps 请求。
### SSE 客户端显示异常
PowerShell `Invoke-WebRequest` 有时对 SSE 或长 JSON 处理不稳定。可以改用 `curl.exe` 或直接查询 MySQL 和 Trace API 验证结果。
### OpenSpec 全量校验失败
`openspec validate --all --strict` 可能因为历史未完成 change 失败。面试演示主要依赖已归档 spec、MVP trace 和 RAG 验收材料,可以单独验证相关 spec。
+74
View File
@@ -0,0 +1,74 @@
# AIOps 轻量规则验证器
## 1. 改动是什么
AIOps 现在有一个确定性的后置质量门禁。最终告警报告持久化后,`AiOpsRuleEvaluationService` 会检查:
- 最终报告是否存在,且不是明显过短。
- payload 模式下,报告是否提到输入的告警和服务。
- 是否有证据工具调用,例如 `lookup_knowledge`、`query_metrics`、`query_logs`。
结果写入:
```text
diagnosis_session.self_evaluation.aiops_rule_evaluation
```
Trace API 会通过 session self-evaluation 展示这个结果。
## 2. 为什么先做规则型
这还不是完整 LLM Verifier。
AIOps 第一阶段质量风险比较具体,适合先用规则:
- 报告有没有生成。
- 报告有没有聚焦 payload。
- 有没有使用证据工具。
- 有没有把无关告警展开成主诊断对象。
规则验证稳定、便宜、容易解释,也不会在当前链路里额外引入一次隐藏模型调用。
## 3. 判定结果
当前评估器输出:
```text
PASS
WARN
FAIL
```
含义:
- `PASS`:核心检查通过。
- `WARN`:报告存在,但可能缺少 payload 关键词或证据工具。
- `FAIL`:缺少最终报告、报告过短等关键问题。
缺少 payload 关键词或证据工具先给 `WARN`,因为 demo/mock 环境下证据可能不可用,且报告措辞可能与 payload 字段不完全一致。
## 4. 面试回答
如果被问:为什么 AIOps 也需要验证器?
```text
Chat 已经有 LLM Verifier,因为用户问题开放度高。
AIOps 的第一阶段质量风险更明确:报告是否聚焦输入告警、是否使用证据工具、报告是否完整。
所以我先做了轻量规则验证器,把结果写入 self_evaluation,让 Trace 不只展示 Agent 做了什么,也展示输出是否通过基础质量门。
```
如果被问:为什么不直接复用 Chat Verifier?
```text
AIOps 验证语义和 Chat 不一样。
它要检查 alert scope、payload focus、证据工具覆盖,以及是否过度展开无关 active alerts。
直接复用 Chat Verifier 会混淆这些语义。
规则评估先提供稳定质量门,后续 AIOps LLM Verifier 可以基于同一套 trace contract 扩展。
```
## 5. 后续增强
- 引入 AIOps LLM Verifier,逐条校验根因和建议是否有 evidence refs。
- 把 rule evaluation 的 checks 在 Trace API 中结构化展示。
- 将 payload scope violation 沉淀为 bad case。

Some files were not shown because too many files have changed in this diff Show More