64 lines
3.3 KiB
Markdown
64 lines
3.3 KiB
Markdown
# Evidence Pipeline Demo Scenarios
|
||
|
||
这份清单用于面试时说明 Chat 证据链路如何覆盖 `PASS`、`LOW_CONFID`、`REJECT` 和 no-evidence 场景。
|
||
|
||
重点区别：
|
||
|
||
- Live demo 证明本地服务、工具、Trace、Feedback 主链路能跑通。
|
||
- Fixture-backed eval 证明固定安全场景可以确定性回归，不依赖 LLM 当场随机输出。
|
||
|
||
## Scenario Matrix
|
||
|
||
| 场景 | 类型 | 输入/证据 | 期望讲点 |
|
||
|---|---|---|---|
|
||
| Payment timeout | Live 主路径 | `requests/payment-timeout-chat.json` | 完整 Chat -> Trace -> Feedback 闭环 |
|
||
| Narrow HighCPU observation | Live 可尝试 + fixture-backed | `requests/narrow-highcpu-chat.json` / `mvp/eval/fixtures/narrow-highcpu-observation-pass.json` | Executor 只输出观察类 claim，Gatekeeper 验引用，Verifier PASS |
|
||
| Hikari no-evidence | Live 可尝试 + fixture-backed | `requests/hikari-no-evidence-chat.json` / `mvp/eval/fixtures/hikari-no-evidence-negative-observation-pass.json` | `$.no_evidence` 只表示本次查询无匹配证据，Composer 不说“已排除” |
|
||
| Unsupported claim filtering | Fixture-backed | `requests/safety-unsupported-claim-chat.json` / `mvp/eval/fixtures/unsupported-claim-filtering-low-confid.json` | Verifier 将 unsupported claim 降为 LOW_CONFID，最终答案不确认“主库故障” |
|
||
| Fabricated invocation reject | Fixture-backed | `mvp/eval/fixtures/gatekeeper-fabricated-invocation-reject.json` | Gatekeeper 拦截伪造 invocation，最终 REJECT/降级 |
|
||
| Composer fallback | Fixture-backed | `mvp/eval/fixtures/composer-fallback-no-raw-json-low-confid.json` | 即使 Composer 输出异常，也不能把 Executor JSON 泄漏给用户 |
|
||
|
||
## Trace Fields To Inspect
|
||
|
||
| 能力 | JSON path |
|
||
|---|---|
|
||
| Executor V2 输出 | `data.session.selfEvaluation.verifier_evaluation.executor_structured_output` |
|
||
| Gatekeeper 结果 | `data.session.selfEvaluation.verifier_evaluation.gatekeeper_result.status` |
|
||
| Gatekeeper 规则版本 | `data.session.selfEvaluation.verifier_evaluation.gatekeeper_result.rule_set_version` |
|
||
| 证据绑定校验 | `data.session.selfEvaluation.verifier_evaluation.gatekeeper_result.checked_bindings` |
|
||
| Verifier claim checks | `data.session.selfEvaluation.verifier_evaluation.claim_checks` |
|
||
| Composer 输出 | `data.session.selfEvaluation.verifier_evaluation.composer_output` |
|
||
| 工具证据引用 | `data.toolInvocations[*].retrievalDetails.evidence_refs` |
|
||
|
||
## How To Present It
|
||
|
||
```text
|
||
我把现场 demo 和固定 eval 分开。
|
||
现场 demo 证明系统能跑通真实链路；
|
||
fixture-backed eval 证明反幻觉安全场景可以稳定回归。
|
||
Gatekeeper 的规则版本也进入 trace，所以后续调整阈值或规则时可以审计。
|
||
```
|
||
|
||
## Optional Live Requests
|
||
|
||
手动发送某个请求样例：
|
||
|
||
```powershell
|
||
$body = Get-Content -Raw -Encoding UTF8 "mvp/demo/requests/narrow-highcpu-chat.json"
|
||
Invoke-RestMethod `
|
||
-Method Post `
|
||
-Uri "http://localhost:9900/api/chat" `
|
||
-ContentType "application/json" `
|
||
-Body $body
|
||
```
|
||
|
||
然后保留响应里的 `runId`，查询同一 run 的 Trace：
|
||
|
||
```powershell
|
||
Invoke-RestMethod `
|
||
-Method Get `
|
||
-Uri "http://localhost:9900/api/diagnosis/mvp-demo-narrow-highcpu-001/trace?runId=$runId"
|
||
```
|
||
|
||
注意：除 payment-timeout 主路径外，其它 live 请求是“可尝试”的演示入口；稳定验收以 `mvp/eval` fixture 和 baseline 为准。
|