Add diagnosis eval baseline diff

2026-07-05 00:59:53 +08:00
parent 4c7c53b024
commit 69deb15330
22 changed files with 1069 additions and 0 deletions
@@ -8,6 +8,7 @@ This folder contains the first fixed-case evaluation set for the MVP diagnosis A
 - Offline trace fixtures: `fixtures/*.json`
 - Field definitions: `schema.md`
 - Baseline reports: `reports/baseline-report.json` and `reports/baseline-report.md`
+- Baseline diff sample: `reports/baseline-diff-sample.json` and `reports/baseline-diff-sample.md`
 - Evaluator implementation: `DiagnosisTraceEvaluator`
 - Report writer: `DiagnosisEvalReportWriter`

@@ -45,3 +46,16 @@ fixed diagnosis case
 -> JSON / Markdown report
 -> regression signal for prompts, tools, retrieval, and verifier behavior
 ```
+
+## Baseline Diff
+
+Baseline diff compares a current report against `reports/baseline-report.json`.
+
+```text
+baseline report
+current report
+-> deterministic diff
+-> regressions, improvements, and changed signals
+```
+
+Use it to answer: did a prompt, tool, retrieval, or verifier change make the Agent worse than the fixed baseline?
@@ -0,0 +1,85 @@
+{
+  "baselineTotalCases" : 5,
+  "currentTotalCases" : 5,
+  "baselinePassedCases" : 5,
+  "currentPassedCases" : 4,
+  "baselinePassRate" : 1.0,
+  "currentPassRate" : 0.8,
+  "regressionCount" : 6,
+  "improvementCount" : 0,
+  "changedCount" : 2,
+  "hasRegression" : true,
+  "items" : [ {
+    "type" : "REGRESSION",
+    "scope" : "aggregate",
+    "caseId" : null,
+    "metric" : "passRate",
+    "baselineValue" : "1.0",
+    "currentValue" : "0.8",
+    "delta" : -0.19999999999999996,
+    "message" : "passRate changed"
+  }, {
+    "type" : "REGRESSION",
+    "scope" : "aggregate",
+    "caseId" : null,
+    "metric" : "averageToolCallCount",
+    "baselineValue" : "2.0",
+    "currentValue" : "3.0",
+    "delta" : 1.0,
+    "message" : "averageToolCallCount changed"
+  }, {
+    "type" : "CHANGED",
+    "scope" : "aggregate",
+    "caseId" : null,
+    "metric" : "verdictDistribution.LOW_CONFID",
+    "baselineValue" : "3",
+    "currentValue" : "2",
+    "delta" : -1.0,
+    "message" : "verdict count changed for LOW_CONFID"
+  }, {
+    "type" : "CHANGED",
+    "scope" : "aggregate",
+    "caseId" : null,
+    "metric" : "verdictDistribution.REJECT",
+    "baselineValue" : "0",
+    "currentValue" : "1",
+    "delta" : 1.0,
+    "message" : "verdict count changed for REJECT"
+  }, {
+    "type" : "REGRESSION",
+    "scope" : "case",
+    "caseId" : "redis-timeout",
+    "metric" : "passed",
+    "baselineValue" : "true",
+    "currentValue" : "false",
+    "delta" : null,
+    "message" : "redis-timeout pass state changed"
+  }, {
+    "type" : "REGRESSION",
+    "scope" : "case",
+    "caseId" : "redis-timeout",
+    "metric" : "verdict",
+    "baselineValue" : "LOW_CONFID",
+    "currentValue" : "REJECT",
+    "delta" : -1.0,
+    "message" : "redis-timeout verdict changed"
+  }, {
+    "type" : "REGRESSION",
+    "scope" : "case",
+    "caseId" : "redis-timeout",
+    "metric" : "matchedKeywordCount",
+    "baselineValue" : "2",
+    "currentValue" : "1",
+    "delta" : -1.0,
+    "message" : "redis-timeout matchedKeywordCount changed"
+  }, {
+    "type" : "REGRESSION",
+    "scope" : "case",
+    "caseId" : "redis-timeout",
+    "metric" : "evidenceCoverage.query_logs",
+    "baselineValue" : "true",
+    "currentValue" : "false",
+    "delta" : null,
+    "message" : "redis-timeout evidence coverage changed for query_logs"
+  } ]
+}
@@ -0,0 +1,22 @@
+# Diagnosis Eval Baseline Diff
+
+- Baseline pass rate: 100.00%
+- Current pass rate: 80.00%
+- Baseline passed cases: 5/5
+- Current passed cases: 4/5
+- Regressions: 6
+- Improvements: 0
+- Other changes: 2
+
+## Diff Items
+
+| Type | Scope | Case | Metric | Baseline | Current | Delta | Message |
+| --- | --- | --- | --- | --- | --- | ---: | --- |
+| REGRESSION | aggregate | - | passRate | 1.0 | 0.8 | -0.200 | passRate changed |
+| REGRESSION | aggregate | - | averageToolCallCount | 2.0 | 3.0 | 1.000 | averageToolCallCount changed |
+| CHANGED | aggregate | - | verdictDistribution.LOW_CONFID | 3 | 2 | -1.000 | verdict count changed for LOW_CONFID |
+| CHANGED | aggregate | - | verdictDistribution.REJECT | 0 | 1 | 1.000 | verdict count changed for REJECT |
+| REGRESSION | case | redis-timeout | passed | true | false | - | redis-timeout pass state changed |
+| REGRESSION | case | redis-timeout | verdict | LOW_CONFID | REJECT | -1.000 | redis-timeout verdict changed |
+| REGRESSION | case | redis-timeout | matchedKeywordCount | 2 | 1 | -1.000 | redis-timeout matchedKeywordCount changed |
+| REGRESSION | case | redis-timeout | evidenceCoverage.query_logs | true | false | - | redis-timeout evidence coverage changed for query_logs |
@@ -135,3 +135,68 @@ Java 类型：`DiagnosisEvalReport`
 Agent 每次运行会留下 trace，评测器用代码规则读取 trace，输出结构化报告。
 这样我改 Agent 的时候，可以用同一套基准判断有没有行为回退。
 ```
+
+## 6. Baseline Diff
+
+Baseline diff 是拿两份 report 做对比：
+
+```text
+baseline report：以前认可的基准结果
+current report：这次改动后跑出来的新结果
+diff report：告诉你哪里变好了、哪里变差了、哪里只是变了
+```
+
+Java 类型：
+
+- `DiagnosisEvalDiffReport`
+- `DiagnosisEvalDiffItem`
+
+`DiagnosisEvalDiffReport` 字段：
+
+| 字段 | 意思 |
+| --- | --- |
+| `baselineTotalCases` | baseline 里有多少条 case |
+| `currentTotalCases` | current 里有多少条 case |
+| `baselinePassedCases` | baseline 通过了多少条 |
+| `currentPassedCases` | current 通过了多少条 |
+| `baselinePassRate` | baseline 通过率 |
+| `currentPassRate` | current 通过率 |
+| `regressionCount` | 退化项数量 |
+| `improvementCount` | 改善项数量 |
+| `changedCount` | 普通变化项数量 |
+| `hasRegression` | 是否存在退化 |
+| `items` | 具体 diff 明细 |
+
+`DiagnosisEvalDiffItem` 字段：
+
+| 字段 | 意思 |
+| --- | --- |
+| `type` | `REGRESSION`、`IMPROVEMENT` 或 `CHANGED` |
+| `scope` | `aggregate` 表示整体指标，`case` 表示单条 case |
+| `caseId` | 如果是单条 case 变化，这里记录 case id |
+| `metric` | 哪个指标变了，比如 `passRate` 或 `evidenceCoverage.query_logs` |
+| `baselineValue` | baseline 里的值 |
+| `currentValue` | current 里的值 |
+| `delta` | 数值变化量；非数值变化为空 |
+| `message` | 给人看的变化说明 |
+
+口语化判断规则：
+
+```text
+pass rate 下降：退化
+case 从通过变失败：退化
+证据工具从有变没有：退化
+关键词命中变少：退化
+工具调用或耗时升高：成本上升，记为退化信号
+verdict 分布变化：记录变化，供人工判断是否符合预期
+```
+
+面试里可以这样讲：
+
+```text
+我把 baseline report 和当前 report 做结构化 diff。
+它不是再问 LLM，而是用代码比较固定字段。
+如果某个 case 从 PASS 变 FAIL，或者 query_logs 证据没了，
+diff 会直接标成 regression。
+这样 Agent 改动可以用固定基准做回归判断。
+```
@@ -9,3 +9,4 @@
 | ISS-005 | 证据链补齐与降级契约收敛 | 高 | 已归档 | [ISS-005-evidence-trace-hardening.md](ISS-005-evidence-trace-hardening.md) |
 | ISS-006 | 固定诊断评测集与回归 Harness | 高 | 已归档 | [ISS-006-diagnosis-eval-harness.md](ISS-006-diagnosis-eval-harness.md) |
 | expand-diagnosis-eval-fixtures | 补齐固定诊断评测 fixture 与 baseline | 中 | 已归档 | [expand-diagnosis-eval-fixtures.md](expand-diagnosis-eval-fixtures.md) |
+| diagnosis-eval-baseline-diff | 诊断评测 baseline diff 与回归判断 | 中 | 已归档 | [diagnosis-eval-baseline-diff.md](diagnosis-eval-baseline-diff.md) |
@@ -0,0 +1,73 @@
+# Diagnosis Eval Baseline Diff
+
+**状态**：已归档  
+**严重程度**：中  
+**发现时间**：2026-07-05  
+**来源**：P1-B follow-up  
+**依赖**：`diagnosis-eval-harness`, `expand-diagnosis-eval-fixtures`
+
+---
+
+## 背景
+
+现在项目已经有固定诊断 case、完整 fixture 和 baseline report。下一步需要把 baseline 真正用起来：每次改 Agent 后，把新的 report 和 baseline report 做对比。
+
+---
+
+## 问题
+
+当前 baseline 只能告诉我们“标准状态是什么”，但还不能自动告诉我们“这次改动有没有变差”。
+
+典型问题包括：
+
+- pass rate 是否下降。
+- 某个 case 是否从通过变失败。
+- 某个 evidence tool 是否从覆盖变成缺失。
+- verifier verdict 分布是否异常变化。
+- 平均工具调用数和耗时是否明显上升。
+
+---
+
+## 目标
+
+新增一个 deterministic baseline diff 能力，用代码比较两份 `DiagnosisEvalReport`。
+
+完成后应该做到：
+
+- 输入 baseline report 和 current report。
+- 输出结构化 diff。
+- 标出 regression、improvement 和普通 changed。
+- 支持 JSON 和 Markdown 输出。
+- 文档说明面试时怎么解释这套回归判断。
+
+---
+
+## 范围
+
+### In scope
+
+- report-level diff 数据结构。
+- aggregate 指标比较。
+- case-level 指标比较。
+- JSON / Markdown diff writer。
+- focused tests 和 eval 文档。
+
+### Out of scope
+
+- 不运行真实 Agent。
+- 不生成新 trace。
+- 不引入 LLM-as-judge。
+- 不改现有 evaluator 评分规则。
+
+---
+
+## 面试表达
+
+可以这样讲：
+
+```text
+我不是只保存了一份 baseline，而是加了 baseline diff。
+每次改 prompt、tool、retrieval 或 verifier 后，
+我都能把新 report 和 baseline 比较，
+直接看到哪些 case 退化、哪些证据缺失、成本有没有上升。
+```