feat(trace): finish run-aware demo verification
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# 反馈与自评估架构
|
||||
|
||||
**更新日期**:2026-07-08
|
||||
**更新日期**:2026-07-10
|
||||
**状态**:当前可运行架构
|
||||
**参考历史文档**:`archive/2026-07-05-legacy/confidence-feedback.md`
|
||||
|
||||
@@ -8,8 +8,8 @@
|
||||
|
||||
反馈架构包含两条闭环:
|
||||
|
||||
1. 系统自评估:基于工具调用、Gatekeeper、Verifier、Composer、AIOps 规则检查,写入 `diagnosis_session.self_evaluation`。
|
||||
2. 用户反馈:用户标记 `useful` 或 `not_useful`,写入 `diagnosis_session.feedback`,其中 `useful` 会沉淀案例。
|
||||
1. 系统自评估:基于当前 run 的工具调用、Gatekeeper、Verifier、Composer、AIOps 规则检查,写入 `diagnosis_run.self_evaluation`。
|
||||
2. 用户反馈:用户标记 `useful` 或 `not_useful`,优先写入 `diagnosis_run.feedback`,其中 `useful` 会沉淀案例。
|
||||
|
||||
当前重要边界:
|
||||
|
||||
@@ -21,7 +21,7 @@
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Answer["Chat / AIOps final answer"] --> Session["diagnosis_session.answer"]
|
||||
Answer["Chat / AIOps final answer"] --> Run["diagnosis_run.answer"]
|
||||
|
||||
subgraph SelfEval["Self evaluation"]
|
||||
Invocation["tool_invocation"] --> RuleEval["EvaluationService: rule_evaluation"]
|
||||
@@ -40,24 +40,24 @@ flowchart TD
|
||||
RuleEval --> Merge["SelfEvaluationMergeService"]
|
||||
VerifierEval --> Merge
|
||||
AiOpsEval --> Merge
|
||||
Merge --> SelfJson["diagnosis_session.self_evaluation"]
|
||||
Merge --> SelfJson["diagnosis_run.self_evaluation"]
|
||||
|
||||
subgraph UserFeedback["User feedback"]
|
||||
UI["Feedback bar"] --> API["POST /api/feedback"]
|
||||
API --> FeedbackService["FeedbackService"]
|
||||
FeedbackService --> FeedbackField["diagnosis_session.feedback"]
|
||||
FeedbackService --> FeedbackField["diagnosis_run.feedback"]
|
||||
FeedbackService --> Useful{"feedback == useful?"}
|
||||
Useful -->|yes| CaseService["CaseLibraryService.createFromSession"]
|
||||
Useful -->|yes| CaseService["CaseLibraryService.createFromRun"]
|
||||
CaseService --> Case["case_library"]
|
||||
Useful -->|no| BadCase["Bad case by feedback=not_useful"]
|
||||
end
|
||||
|
||||
Session --> UI
|
||||
Run --> UI
|
||||
```
|
||||
|
||||
## 3. self_evaluation JSON
|
||||
|
||||
`SelfEvaluationMergeService` 统一维护 `diagnosis_session.self_evaluation`。
|
||||
`SelfEvaluationMergeService` 统一维护当前运行的 `diagnosis_run.self_evaluation`。历史兼容数据可能仍存在于 `diagnosis_session.self_evaluation`,但新 Chat/AIOps 执行不再写旧表。
|
||||
|
||||
当前结构:
|
||||
|
||||
@@ -149,7 +149,7 @@ flowchart LR
|
||||
Composer --> ComposerOutput["composer_output"]
|
||||
Output --> Merge["SelfEvaluationMergeService.mergeVerifierEvaluation"]
|
||||
ComposerOutput --> Merge
|
||||
Merge --> Session["diagnosis_session.self_evaluation.verifier_evaluation"]
|
||||
Merge --> Run["diagnosis_run.self_evaluation.verifier_evaluation"]
|
||||
```
|
||||
|
||||
Verifier 输出:
|
||||
@@ -198,6 +198,7 @@ POST /api/feedback
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"runId": "run-xxx",
|
||||
"sessionId": "xxx",
|
||||
"feedback": "useful" | "not_useful"
|
||||
}
|
||||
@@ -209,6 +210,8 @@ Content-Type: application/json
|
||||
{
|
||||
"success": true,
|
||||
"message": "反馈已记录",
|
||||
"runId": "run-xxx",
|
||||
"fallbackToLatestRun": false,
|
||||
"caseId": "uuid 或 null"
|
||||
}
|
||||
```
|
||||
@@ -217,10 +220,16 @@ Content-Type: application/json
|
||||
|
||||
| feedback | 行为 |
|
||||
|---|---|
|
||||
| `useful` | 写入 `DiagnosisSession.feedback`,调用 `CaseLibraryService.createFromSession` |
|
||||
| `not_useful` | 写入 `DiagnosisSession.feedback`,不改变 session status |
|
||||
| `useful` | 写入 `DiagnosisRun.feedback`,调用 `CaseLibraryService.createFromRun` |
|
||||
| `not_useful` | 写入 `DiagnosisRun.feedback`,不改变 run status |
|
||||
| 其他值 | 返回 HTTP 400 |
|
||||
|
||||
兼容行为:
|
||||
|
||||
- 请求带 `runId` 时,后端验证 `runId` 属于 `sessionId`。
|
||||
- 请求缺少 `runId` 且存在 run-backed 数据时,后端绑定 latest run,并返回 `fallbackToLatestRun=true` 和实际 `runId`。
|
||||
- 仅当没有 `diagnosis_run` 但存在历史 `diagnosis_session` 时,才使用历史 fallback;该路径不声明 latest-run fallback。
|
||||
|
||||
## 8. 案例沉淀
|
||||
|
||||
`useful` 反馈会生成或复用 `case_library` 记录。
|
||||
@@ -230,7 +239,7 @@ Content-Type: application/json
|
||||
| CaseLibrary 字段 | 来源 |
|
||||
|---|---|
|
||||
| `caseId` | UUID |
|
||||
| `diagnosisId` | `DiagnosisSession.sessionId` |
|
||||
| `diagnosisId` | 新数据为 `DiagnosisRun.runId`;历史数据可能为 `DiagnosisSession.sessionId` |
|
||||
| `sourceType` | `AUTO` |
|
||||
| `faultCategory` | 当前固定为 `GENERAL` |
|
||||
| `title` | `query` 前 100 字符 |
|
||||
@@ -241,7 +250,7 @@ Content-Type: application/json
|
||||
幂等性:
|
||||
|
||||
```text
|
||||
case_library.diagnosisId == sessionId
|
||||
case_library.diagnosisId == runId
|
||||
-> existing case: return existing
|
||||
-> missing case: create new
|
||||
```
|
||||
@@ -260,9 +269,9 @@ Trace API 会展示:
|
||||
|
||||
| 视角 | 数据来源 |
|
||||
|---|---|
|
||||
| 执行是否成功 | `diagnosis_session.status` |
|
||||
| 执行是否成功 | `diagnosis_run.status` |
|
||||
| 证据是否充分 | `self_evaluation.rule_evaluation` / `verifier_evaluation` |
|
||||
| 用户是否认可 | `diagnosis_session.feedback` |
|
||||
| 用户是否认可 | `diagnosis_run.feedback` |
|
||||
|
||||
## 10. 后续增强
|
||||
|
||||
|
||||
Reference in New Issue
Block a user