docs(architecture): align evidence pipeline design
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
# 当前 MVP 架构
|
||||
|
||||
**更新日期**:2026-07-05
|
||||
**状态**:当前可运行架构
|
||||
**更新日期**:2026-07-08
|
||||
**状态**:当前可运行架构
|
||||
**适用范围**:Demo、面试讲解、后续迭代规划
|
||||
|
||||
## 1. 系统定位
|
||||
@@ -38,7 +38,9 @@ flowchart TB
|
||||
Supervisor["Supervisor"]
|
||||
Planner["Planner"]
|
||||
Executor["Executor"]
|
||||
Gatekeeper["Gatekeeper"]
|
||||
Verifier["Verifier"]
|
||||
Composer["Composer"]
|
||||
end
|
||||
|
||||
subgraph Tools["Evidence Tools"]
|
||||
@@ -105,7 +107,9 @@ Agent Orchestration
|
||||
-> Supervisor
|
||||
-> Planner
|
||||
-> Executor
|
||||
-> Gatekeeper
|
||||
-> Verifier
|
||||
-> Composer
|
||||
|
||||
Evidence Tools
|
||||
-> lookup_knowledge
|
||||
@@ -133,6 +137,7 @@ Persistence
|
||||
-> Milvus/Zilliz collection
|
||||
|
||||
Quality Gates
|
||||
-> executor gatekeeper
|
||||
-> chat verifier
|
||||
-> AIOps rule evaluation
|
||||
-> diagnosis eval baseline
|
||||
@@ -150,7 +155,9 @@ sequenceDiagram
|
||||
participant Planner as Planner Agent
|
||||
participant Executor as Executor Agent
|
||||
participant Tool as Evidence Tools
|
||||
participant Gatekeeper as Gatekeeper Hook
|
||||
participant Verifier as Verifier Agent
|
||||
participant Composer as Composer Agent
|
||||
participant DB as Trace Tables
|
||||
participant Trace as Trace API
|
||||
|
||||
@@ -162,8 +169,12 @@ sequenceDiagram
|
||||
Executor->>Tool: lookup_knowledge / logs / metrics
|
||||
Tool->>DB: 写入 tool_invocation
|
||||
Tool-->>Executor: 返回证据
|
||||
Executor->>Verifier: 生成候选诊断并校验
|
||||
Executor->>Gatekeeper: 输出 executor_evidence_v2
|
||||
Gatekeeper->>DB: 读取 tool_invocation.evidence_refs 并校验引用
|
||||
Gatekeeper->>Verifier: 传入已验真的 claims / excerpts
|
||||
Verifier->>DB: 合并 self_evaluation.verifier_evaluation
|
||||
Verifier->>Composer: 传入 allowed_claims / missing_info / actions
|
||||
Composer->>Chat: 生成最终用户答复
|
||||
Chat->>DB: 保存 diagnosis_session.answer
|
||||
User->>Trace: GET /api/diagnosis/{sessionId}/trace
|
||||
Trace->>DB: 聚合 session / step / tool
|
||||
@@ -180,14 +191,16 @@ POST /api/chat
|
||||
-> lookup_knowledge
|
||||
-> query_logs
|
||||
-> query_metrics
|
||||
-> Verifier 校验最终诊断
|
||||
-> Gatekeeper 校验 Executor 证据引用真实性
|
||||
-> Verifier 判断 claim 是否能由已核验证据推出
|
||||
-> Composer 生成最终用户答复
|
||||
-> 保存 diagnosis_session
|
||||
-> 保存 agent_step
|
||||
-> 保存 tool_invocation
|
||||
-> 合并 self_evaluation.verifier_evaluation
|
||||
```
|
||||
|
||||
Chat 链路的质量门禁是 LLM Verifier。Verifier 输出合并到 `diagnosis_session.self_evaluation.verifier_evaluation`,Trace API 会展示该验证结果。
|
||||
Chat 链路的质量门禁由三段组成:Gatekeeper 先做代码级引用验真,Verifier 再做 LLM 可推导性判断,Composer 最后控制对用户的表达边界。Gatekeeper、Verifier、Composer 的输出合并到 `diagnosis_session.self_evaluation.verifier_evaluation`,Trace API 会展示该验证结果。
|
||||
|
||||
Agent 编排细节见 [agent-orchestration.md](agent-orchestration.md)。
|
||||
|
||||
@@ -323,6 +336,7 @@ tool_invocation
|
||||
-> 工具调用事实
|
||||
-> tool_name / input_params / output_preview
|
||||
-> retrieval_layer / retrieval_details
|
||||
-> retrieval_details.evidence_refs
|
||||
-> relevance_level / dedup_reason
|
||||
-> duration / success
|
||||
```
|
||||
@@ -346,12 +360,12 @@ Trace API 聚合:
|
||||
- 会话状态和最终报告。
|
||||
- Agent step 序列。
|
||||
- 工具调用和检索细节。
|
||||
- Chat verifier 结果。
|
||||
- Chat Gatekeeper / Verifier / Composer 结果。
|
||||
- AIOps rule evaluation 结果。
|
||||
|
||||
Trace 是本项目区别于普通问答系统的关键:答案不是孤立文本,而是可以追溯到 Agent 决策、工具调用和证据来源。
|
||||
|
||||
Prompt、Hook、Verifier 和评测门禁的完整说明见 [harness-quality-gates.md](harness-quality-gates.md),用户反馈与 `self_evaluation` 闭环见 [feedback-architecture.md](feedback-architecture.md)。
|
||||
Prompt、Hook、Gatekeeper、Verifier、Composer 和评测门禁的完整说明见 [harness-quality-gates.md](harness-quality-gates.md),用户反馈与 `self_evaluation` 闭环见 [feedback-architecture.md](feedback-architecture.md)。
|
||||
|
||||
## 8. 质量门禁
|
||||
|
||||
@@ -359,7 +373,9 @@ Prompt、Hook、Verifier 和评测门禁的完整说明见 [harness-quality-gate
|
||||
|
||||
| 门禁 | 位置 | 作用 |
|
||||
|---|---|---|
|
||||
| Chat Verifier | `ChatService` | 校验普通诊断回答质量 |
|
||||
| Executor Gatekeeper | `VerifierInputHook` / `ExecutorGatekeeperService` | 校验 Executor 引用的 invocation、`raw_path`、`evidence_excerpt` 是否真实 |
|
||||
| Chat Verifier | `ChatService` | 判断已验真证据是否能推出 Executor claims |
|
||||
| Chat Composer | `ChatService` | 只表达 Verifier 允许输出的内容,避免把 no-evidence 说成已排除 |
|
||||
| AIOps Rule Evaluation | `AiOpsRuleEvaluationService` | 校验告警诊断是否聚焦 payload 并使用证据 |
|
||||
| Diagnosis Eval Baseline | `mvp/eval/` | 固化诊断 trace 和报告行为 |
|
||||
| RAG Retrieval Baseline | `eval/rag-retrieval/` | 固化检索召回行为,避免 RAG 重构回退 |
|
||||
@@ -379,6 +395,10 @@ Prompt、Hook、Verifier 和评测门禁的完整说明见 [harness-quality-gate
|
||||
- `title`、`breadcrumb`、`content` 参与 embedding 文本。
|
||||
- `tool_invocation` 记录检索层、relevance level、dedup reason。
|
||||
- Chat verifier 和 AIOps rule evaluation 合并进 `self_evaluation`。
|
||||
- Chat Executor 结构化输出 `executor_evidence_v2`,不再直接承担最终用户答复。
|
||||
- `tool_invocation.retrieval_details.evidence_refs` 支持 `raw_path` 精确引用和 `$.no_evidence` 负向证据。
|
||||
- Gatekeeper 对 Executor 引用做代码级验真,Verifier 只判断可推导性。
|
||||
- Composer 在 Verifier 之后生成最终用户表达,并限制 negative observation 过度表述。
|
||||
- RAG offline baseline 和 live acceptance 脚本。
|
||||
|
||||
暂不作为当前已完成能力声明:
|
||||
@@ -406,4 +426,5 @@ Prompt、Hook、Verifier 和评测门禁的完整说明见 [harness-quality-gate
|
||||
| Spring AI VectorStore 配置辅助 | `SpringAiVectorStoreSidecarService` |
|
||||
| Trace 聚合 | `DiagnosisTraceService` |
|
||||
| 工具调用记录 | `ToolInvocationRecorder` |
|
||||
| Executor 引用验真 | `ExecutorGatekeeperService`, `VerifierInputHook` |
|
||||
| self_evaluation 合并 | `SelfEvaluationMergeService` |
|
||||
|
||||
Reference in New Issue
Block a user