refactor(harness): remove legacy agent architecture

2026-07-22 18:02:01 +08:00
parent bc36248cd8
commit 8ee7cc0b70
148 changed files with 3091 additions and 13889 deletions
@@ -0,0 +1,63 @@
+# Evidence Pipeline Demo Scenarios
+
+这份清单用于面试时说明 Chat 证据链路如何覆盖 `PASS`、`LOW_CONFID`、`REJECT` 和 no-evidence 场景。
+
+重点区别：
+
+- Live demo 证明本地服务、工具、Trace、Feedback 主链路能跑通。
+- Fixture-backed eval 证明固定安全场景可以确定性回归，不依赖 LLM 当场随机输出。
+
+## Scenario Matrix
+
+| 场景 | 类型 | 输入/证据 | 期望讲点 |
+|---|---|---|---|
+| Payment timeout | Live 主路径 | `requests/payment-timeout-chat.json` | 完整 Chat -> Trace -> Feedback 闭环 |
+| Narrow HighCPU observation | Live 可尝试 + fixture-backed | `requests/narrow-highcpu-chat.json` / `mvp/eval/fixtures/narrow-highcpu-observation-pass.json` | Executor 只输出观察类 claim，Gatekeeper 验引用，Verifier PASS |
+| Hikari no-evidence | Live 可尝试 + fixture-backed | `requests/hikari-no-evidence-chat.json` / `mvp/eval/fixtures/hikari-no-evidence-negative-observation-pass.json` | `$.no_evidence` 只表示本次查询无匹配证据，Composer 不说“已排除” |
+| Unsupported claim filtering | Fixture-backed | `requests/safety-unsupported-claim-chat.json` / `mvp/eval/fixtures/unsupported-claim-filtering-low-confid.json` | Verifier 将 unsupported claim 降为 LOW_CONFID，最终答案不确认“主库故障” |
+| Fabricated invocation reject | Fixture-backed | `mvp/eval/fixtures/gatekeeper-fabricated-invocation-reject.json` | Gatekeeper 拦截伪造 invocation，最终 REJECT/降级 |
+| Composer fallback | Fixture-backed | `mvp/eval/fixtures/composer-fallback-no-raw-json-low-confid.json` | 即使 Composer 输出异常，也不能把 Executor JSON 泄漏给用户 |
+
+## Trace Fields To Inspect
+
+| 能力 | JSON path |
+|---|---|
+| Executor V2 输出 | `data.session.selfEvaluation.verifier_evaluation.executor_structured_output` |
+| Gatekeeper 结果 | `data.session.selfEvaluation.verifier_evaluation.gatekeeper_result.status` |
+| Gatekeeper 规则版本 | `data.session.selfEvaluation.verifier_evaluation.gatekeeper_result.rule_set_version` |
+| 证据绑定校验 | `data.session.selfEvaluation.verifier_evaluation.gatekeeper_result.checked_bindings` |
+| Verifier claim checks | `data.session.selfEvaluation.verifier_evaluation.claim_checks` |
+| Composer 输出 | `data.session.selfEvaluation.verifier_evaluation.composer_output` |
+| 工具证据引用 | `data.toolInvocations[*].retrievalDetails.evidence_refs` |
+
+## How To Present It
+
+```text
+我把现场 demo 和固定 eval 分开。
+现场 demo 证明系统能跑通真实链路；
+fixture-backed eval 证明反幻觉安全场景可以稳定回归。
+Gatekeeper 的规则版本也进入 trace，所以后续调整阈值或规则时可以审计。
+```
+
+## Optional Live Requests
+
+手动发送某个请求样例：
+
+```powershell
+$body = Get-Content -Raw -Encoding UTF8 "mvp/demo/requests/narrow-highcpu-chat.json"
+Invoke-RestMethod `
+  -Method Post `
+  -Uri "http://localhost:9900/api/chat" `
+  -ContentType "application/json" `
+  -Body $body
+```
+
+然后保留响应里的 `runId`，查询同一 run 的 Trace：
+
+```powershell
+Invoke-RestMethod `
+  -Method Get `
+  -Uri "http://localhost:9900/api/diagnosis/mvp-demo-narrow-highcpu-001/trace?runId=$runId"
+```
+
+注意：除 payment-timeout 主路径外，其它 live 请求是“可尝试”的演示入口；稳定验收以 `mvp/eval` fixture 和 baseline 为准。