zhuyongxin
7ae9707a3b
Persist provider reasoning and assistant text separately on agent_reasoning_audit (DeepSeekAssistantMessage path), extract diagnosis_run.conclusion, enrich RAG tool audit (step_id/query/qualityScore), gate empty mysql tools, drop devtools, and align MVP docs after live E2E verification.
164 lines
5.5 KiB
Markdown
164 lines
5.5 KiB
Markdown
# RAG 审计补丁 E2E：`step_id` + query（含业务 FALLBACK 样本）
|
||
|
||
**日期**：2026-07-28
|
||
**状态**：审计字段 live 验收记录
|
||
**关联主文档**：[一次诊断到底发生了什么（SUCCESS 全流程）](一次诊断全流程-E2E导读.md)
|
||
|
||
> 主文档只保留 **SUCCESS 完整诊断** 与 **现行审计能力说明**。
|
||
> 本页单独记录：改造后的一次 live 验收——**审计字段 PASS**，业务因 Milvus 空结果走了 **FALLBACK**。
|
||
|
||
---
|
||
|
||
## 1. 样本身份
|
||
|
||
| 项 | 值 |
|
||
|----|----|
|
||
| `session_id` | `e2e-audit-20260728171858` |
|
||
| `run_id` | `0be605f6-e036-40d3-b364-11b73672241f` |
|
||
| 入口 | `POST /api/chat`（与 SUCCESS 样例同构的 RAG-only 约束题） |
|
||
| 冷启动 | 含 `AgentStepAuditTracker` 等补丁后的 `mvn spring-boot:run` |
|
||
| 业务结局 | `release_outcome=FALLBACK`，`content_type=SAFE_FALLBACK` |
|
||
| 工具 | 1× `lookup_knowledge` |
|
||
|
||
本地产物（若仍在）：`target/e2e-audit-sse.txt`、`target/e2e-audit-session.txt`、`target/e2e-audit-run.txt`。
|
||
|
||
---
|
||
|
||
## 2. 验收目标 vs 非目标
|
||
|
||
| 目标 | 是否本页重点 |
|
||
|------|----------------|
|
||
| `tool_invocation.step_id` = 发出 tool_call 的 `agent_step.id` | **是** |
|
||
| `input_params` 含安全 `query` 预览 | **是** |
|
||
| Trace / timeline 带回 `step_id` | **是** |
|
||
| 业务必须 SUCCESS | **否**（本 run 为 FALLBACK，归因环境） |
|
||
|
||
---
|
||
|
||
## 3. 审计结果：PASS
|
||
|
||
### 3.1 `agent_step`
|
||
|
||
| id | step_index | has_tool_call | 说明 |
|
||
|----|------------|---------------|------|
|
||
| **962** | 0 | 1 | 发出 `lookup_knowledge` |
|
||
| 963 | 1 | 0 | 无证据后的收尾轮 |
|
||
|
||
### 3.2 `tool_invocation`（id=860）
|
||
|
||
| 字段 | 值 |
|
||
|------|-----|
|
||
| `step_id` | **962**（= step0） |
|
||
| `tool_name` | `lookup_knowledge` |
|
||
| `success` | 1（工具跑完；无证据也算执行成功） |
|
||
| `search_mode` | hybrid |
|
||
| `evidence_status` | `NO_EVIDENCE` |
|
||
| `candidate_count` | 0 |
|
||
| `relevance_level` | null |
|
||
|
||
**`input_params` 实值：**
|
||
|
||
```json
|
||
{
|
||
"query": "MySQL connection pool exhausted HikariCP diagnosis",
|
||
"step_id": 962,
|
||
"tool_call_id": "call_00_LireJzbiFEsbZ9ZVvgYp8330",
|
||
"request_bytes": 62
|
||
}
|
||
```
|
||
|
||
### 3.3 Trace API
|
||
|
||
- `toolInvocations[0].stepId = 962`
|
||
- `inputParams.query` 有值
|
||
- timeline `TOOL_INVOCATION.details.step_id = 962`
|
||
|
||
### 3.4 对照表
|
||
|
||
| 检查项 | 预期 | 实际 | 判定 |
|
||
|--------|------|------|------|
|
||
| `tool_invocation.step_id` | = agent_step.id | 962 | **PASS** |
|
||
| `input_params.query` | 有预览 | 有 | **PASS** |
|
||
| `input_params.step_id` | 与列一致 | 962 | **PASS** |
|
||
| Trace `stepId` | 非空 | 962 | **PASS** |
|
||
| timeline `step_id` | 非空 | 962 | **PASS** |
|
||
| `search_mode` | hybrid | hybrid | **PASS** |
|
||
| 业务 outcome | （非本页 KPI） | FALLBACK | 见 §4 |
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
S0[agent_step 962<br/>r1 tool_call] --> TI[tool_invocation 860<br/>step_id=962]
|
||
TI --> IP[input_params.query]
|
||
TI --> TR[Trace / timeline]
|
||
TI --> RAG[hybrid candidate_count=0]
|
||
RAG --> FB[FALLBACK<br/>NO_EVIDENCE]
|
||
```
|
||
|
||
---
|
||
|
||
## 4. 业务 FALLBACK 原因（与审计无关）
|
||
|
||
| 现象 | 说明 |
|
||
|------|------|
|
||
| 日志 | `found=false`，`evidenceBlocks=0` |
|
||
| audit | `attempts[].usable=false`，`candidate_count=0` |
|
||
| warm 检索 | 同期 `GET /api/search/similar` 亦失败 |
|
||
| 日志噪音 | Milvus channel 未正确 shutdown 等提示（环境/客户端生命周期） |
|
||
|
||
**结论**：hybrid **路径进了**，但当次 **0 候选** → 无证据可写报告 → `SAFE_FALLBACK` / `INSUFFICIENT_EVIDENCE`。
|
||
这不否定 `step_id` / `query` 落库；完整 SUCCESS 业务故事见主文档。
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
Q[同一 RAG-only 题] --> LK[lookup_knowledge]
|
||
LK --> AUD[审计: step_id + query PASS]
|
||
LK --> HIT{有候选?}
|
||
HIT -->|SUCCESS 主文档 run| OK[PRECISE → DIAGNOSIS_REPORT]
|
||
HIT -->|本页 run| NO[0 候选 → FALLBACK]
|
||
```
|
||
|
||
---
|
||
|
||
## 5. 实现索引（补丁代码）
|
||
|
||
| 组件 | 职责 |
|
||
|------|------|
|
||
| `AgentStepAuditTracker` | run 级 bind/current/clear `agent_step.id` |
|
||
| `HarnessAgentAuditHook` | `beforeModel` 落 step 后 `bind` |
|
||
| `ToolBoundary.auditSafely` | 读 tracker，带上 `stepId` + `requestJson` |
|
||
| `JpaToolInvocationAuditSink` | 写 `step_id` 列；安全展开 `input_params` |
|
||
| `TraceAuditEvents.toolInvocation` | timeline details 的 `step_id` |
|
||
| `JpaChatRunStore.finish` | `clear(runId)` |
|
||
|
||
**`input_params` 安全规则摘要**：
|
||
|
||
- 始终：`tool_call_id`、`request_bytes`；有则：`step_id`
|
||
- 顶层 string/number/boolean/纯字符串数组；文本 ≤160 字符
|
||
- 键名含 password/token/secret/apikey 等 → 不落库
|
||
|
||
更完整的字段词典与 SUCCESS 阶段拆解见主文档 §3.4 / §3.4.1。
|
||
|
||
---
|
||
|
||
## 6. 复盘 SQL
|
||
|
||
```bash
|
||
python scripts/query_mysql.py "SELECT id, step_id, tool_name, relevance_level, CAST(input_params AS CHAR) AS inputp, LEFT(CAST(retrieval_details AS CHAR), 500) AS details FROM tool_invocation WHERE run_id = '0be605f6-e036-40d3-b364-11b73672241f'"
|
||
|
||
python scripts/query_mysql.py "SELECT id, step_index, agent_name, has_tool_call, token_count FROM agent_step WHERE run_id = '0be605f6-e036-40d3-b364-11b73672241f' ORDER BY step_index"
|
||
|
||
python scripts/query_mysql.py "SELECT run_id, status, release_outcome, tool_call_count, total_token_count FROM diagnosis_run WHERE run_id = '0be605f6-e036-40d3-b364-11b73672241f'"
|
||
```
|
||
|
||
```http
|
||
GET /api/diagnosis/e2e-audit-20260728171858/trace?runId=0be605f6-e036-40d3-b364-11b73672241f
|
||
```
|
||
|
||
---
|
||
|
||
## 7. 修订记录
|
||
|
||
| 日期 | 说明 |
|
||
|------|------|
|
||
| 2026-07-28 | 从主 walkthrough 拆出：专门记录 audit 验收 run（FALLBACK 业务 + step_id/query PASS） |
|