osiman/SuperBizAgent-java
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: reorganize MVP interview documentation

Browse Source
This commit is contained in:
aruo
2026-07-05 15:29:28 +08:00
parent b22f2d22c8
commit 88e0a6c944
51 changed files with 4352 additions and 1318 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mvp/architecture/current-mvp-architecture.md
+341 -170
View File
@@ -1,220 +1,391 @@
# Current MVP Architecture Snapshot
# 当前 MVP 架构
**Updated**: 2026-07-05
**更新日期**：2026-07-05
**状态**：当前可运行架构
**适用范围**：Demo、面试讲解、后续迭代规划
This document records the current runnable MVP architecture. Older architecture notes in this folder still represent design history; this file should be read as the current snapshot for demos, interviews, and next-step planning.
## 1. 系统定位
## 1. Positioning
SuperBizAgent MVP 不是通用 Chatbot，而是面向故障诊断的 Agent 工程项目。
The MVP is an Agent engineering project for traceable troubleshooting, not a generic chatbot.
核心目标：
Core goals:
- 支持用户主动发起的 Chat 诊断。
- 支持 AIOps 告警触发的自动诊断。
- 保留 Agent 的规划、执行、验证过程。
- 工具调用必须显式、可追踪、可回放。
- RAG 检索必须通过 `lookup_knowledge` 暴露证据链。
- 每次诊断都沉淀 session、step、tool invocation 和 self evaluation。
- Support normal chat-based diagnosis.
- Support AIOps alert-triggered diagnosis.
- Keep tool calls explicit and traceable.
- Keep RAG retrieval observable through `lookup_knowledge`.
- Persist enough execution evidence for replay, evaluation, and interview explanation.
## 2. 总体分层
## 2. Runtime Architecture
```mermaid
flowchart TB
subgraph API["API Layer"]
ChatController["ChatController"]
TraceController["DiagnosisTraceController"]
SearchController["SearchController"]
DocumentController["DocumentController"]
end
```text
HTTP API
-> ChatService / AiOpsService
-> Agent orchestration
-> Supervisor / Planner / Executor / Verifier
-> Tools
-> lookup_knowledge
-> query_logs
-> query_metrics
-> other diagnosis tools
-> Persistence
-> diagnosis_session
-> agent_step
-> tool_invocation
-> Trace API
-> DiagnosisTraceService
subgraph App["Application Service"]
ChatService["ChatService"]
AiOpsService["AiOpsService"]
TraceService["DiagnosisTraceService"]
end
subgraph Agent["Agent Orchestration"]
Supervisor["Supervisor"]
Planner["Planner"]
Executor["Executor"]
Verifier["Verifier"]
end
subgraph Tools["Evidence Tools"]
KnowledgeTool["lookup_knowledge"]
LogsTool["query_logs"]
MetricsTool["query_metrics"]
AlertsTool["queryPrometheusAlerts"]
end
subgraph RAG["RAG Retrieval"]
L0["KnowledgeIndexService"]
VectorSearch["VectorSearchService"]
VectorStore["Spring AI VectorStore"]
SdkFallback["Milvus SDK fallback"]
end
subgraph Store["Persistence and Trace"]
Session["diagnosis_session"]
Step["agent_step"]
Invocation["tool_invocation"]
ApiDoc["api_document"]
Milvus["Milvus/Zilliz"]
end
API --> App
ChatService --> Agent
AiOpsService --> Agent
Agent --> Tools
KnowledgeTool --> RAG
RAG --> Store
Tools --> Invocation
Agent --> Step
App --> Session
TraceService --> Session
TraceService --> Step
TraceService --> Invocation
```
Current entry points:
- `ChatService`: user-driven troubleshooting and follow-up diagnosis.
- `AiOpsService`: alert-driven diagnosis, including payload mode and auto-discovery mode.
- `DiagnosisTraceService`: trace view of session, steps, tool calls, and self-evaluation.
## 3. Chat Diagnosis Flow
```text
User question
API Layer
-> ChatController
-> DiagnosisTraceController
-> SearchController
-> DocumentController
Application Service
-> ChatService
-> simple response or diagnosis flow
-> Planner creates investigation direction
-> Executor calls tools for evidence
-> lookup_knowledge
-> query_logs
-> query_metrics
-> Verifier checks final diagnosis quality
-> self_evaluation.verifier_evaluation
-> diagnosis trace
```
The chat path uses the LLM verifier as the main quality gate. The verifier result is persisted under `diagnosis_session.self_evaluation.verifier_evaluation`.
## 4. AIOps Diagnosis Flow
```text
AIOps request
-> AiOpsService
-> payload mode or auto-discovery mode
-> build alert-focused diagnosis prompt
-> append recommended lookup_knowledge query when payload exists
-> Agent diagnosis flow
-> Supervisor / Planner / Executor
-> evidence tools
-> final report
-> AiOpsRuleEvaluationService
-> self_evaluation.aiops_rule_evaluation
-> diagnosis trace
```
-> DiagnosisTraceService
AIOps keeps two modes:
Agent Orchestration
-> Supervisor
-> Planner
-> Executor
-> Verifier
- Payload mode: the request already contains alert fields such as alert name, service, metric, severity, and symptom. The system builds a recommended knowledge query from these fields.
- Auto-discovery mode: the system follows the original alert-discovery behavior and lets the Agent collect alert context through tools.
Evidence Tools
-> lookup_knowledge
-> query_logs
-> query_metrics
-> queryPrometheusAlerts
The AIOps verifier is currently lightweight and rule-based. It checks:
- Whether the final report exists.
- Whether the result stays focused on the alert payload when payload exists.
- Whether evidence tools were used, especially `lookup_knowledge`, `query_logs`, and `query_metrics`.
## 5. RAG Architecture
```text
lookup_knowledge
-> L0 domain/entity hint
-> matched domain
-> matched keywords/entities
-> metadata filter signal
RAG Retrieval
-> KnowledgeIndexService
-> VectorSearchService
-> Spring AI VectorStore path
-> Milvus SDK fallback path
-> evidence post-processing
-> score / rawScore / scoreLabel
-> source metadata
-> title / breadcrumb / content evidence block
-> tool_invocation record
-> Spring AI VectorStore
-> Milvus SDK fallback
Persistence
-> diagnosis_session
-> agent_step
-> tool_invocation
-> api_document
-> Milvus/Zilliz collection
Quality Gates
-> chat verifier
-> AIOps rule evaluation
-> diagnosis eval baseline
-> RAG retrieval baseline
```
Important decisions:
## 3. Chat 诊断链路
- `lookup_knowledge` remains an explicit Agent tool. It is not replaced by an implicit chat Advisor because the project needs visible Agent decision-making.
- L0 is retained but downgraded. It is a domain/entity hint and explainability signal, not the final recall decision.
- L1 retrieval now goes through `VectorSearchService`.
- Spring AI `VectorStore` is the preferred retrieval path.
- The original Milvus SDK path is retained as fallback and compatibility path.
- `title`, `breadcrumb`, and `content` participate in embedding text so chunk context is less likely to be lost.
- Retrieval output keeps compatibility fields: `score`, `rawScore`, and `scoreLabel`.
```mermaid
sequenceDiagram
autonumber
actor User as 用户
participant API as POST /api/chat
participant Chat as ChatService
participant Planner as Planner Agent
participant Executor as Executor Agent
participant Tool as Evidence Tools
participant Verifier as Verifier Agent
participant DB as Trace Tables
participant Trace as Trace API
Vector retrieval modes:
User->>API: 提交诊断问题
API->>Chat: execute chat strategy
Chat->>Planner: 复杂问题进入规划
Planner->>DB: 写入 agent_step
Planner->>Executor: 下发排查方向
Executor->>Tool: lookup_knowledge / logs / metrics
Tool->>DB: 写入 tool_invocation
Tool-->>Executor: 返回证据
Executor->>Verifier: 生成候选诊断并校验
Verifier->>DB: 合并 self_evaluation.verifier_evaluation
Chat->>DB: 保存 diagnosis_session.answer
User->>Trace: GET /api/diagnosis/{sessionId}/trace
Trace->>DB: 聚合 session / step / tool
Trace-->>User: 返回可回放诊断链路
```
```text
retrieval.vector-store.mode=auto # Prefer Spring AI VectorStore, fallback to SDK
retrieval.vector-store.mode=spring-ai # Use Spring AI VectorStore only
retrieval.vector-store.mode=sdk # Use original Milvus SDK path
POST /api/chat
-> ChatService
-> 简单问题：轻量回答
-> 复杂诊断：Agent 编排
-> Planner 制定排查方向
-> Executor 调用证据工具
-> lookup_knowledge
-> query_logs
-> query_metrics
-> Verifier 校验最终诊断
-> 保存 diagnosis_session
-> 保存 agent_step
-> 保存 tool_invocation
-> 合并 self_evaluation.verifier_evaluation
```
## 6. Persistence And Trace
Chat 链路的质量门禁是 LLM Verifier。Verifier 输出合并到 `diagnosis_session.self_evaluation.verifier_evaluation`，Trace API 会展示该验证结果。
Current trace-related persistence:
Agent 编排细节见 [agent-orchestration.md](agent-orchestration.md)。
关键代码：
- `src/main/java/com/superbiz/agent/controller/ChatController.java`
- `src/main/java/com/superbiz/agent/service/ChatService.java`
- `src/main/java/com/superbiz/agent/service/SelfEvaluationMergeService.java`
- `src/main/java/com/superbiz/agent/service/DiagnosisTraceService.java`
## 4. AIOps 诊断链路
```mermaid
flowchart TD
Request["POST /api/ai_ops"] --> Payload{"包含告警 payload?"}
Payload -->|是| Targeted["PAYLOAD_TARGETED"]
Payload -->|否| Discovery["AUTO_DISCOVERY"]
Targeted --> BuildPrompt["构造聚焦 payload 的诊断 prompt"]
Targeted --> QueryAug["生成 recommended lookup_knowledge query"]
Discovery --> DiscoverAlert["通过 queryPrometheusAlerts 发现活跃告警"]
BuildPrompt --> Plan["Planner 规划排查"]
QueryAug --> Plan
DiscoverAlert --> Plan
Plan --> Execute["Executor 收集证据"]
Execute --> Knowledge["lookup_knowledge"]
Execute --> Metrics["query_metrics / Prometheus"]
Execute --> Logs["query_logs"]
Knowledge --> Report["告警分析报告"]
Metrics --> Report
Logs --> Report
Report --> RuleEval["AiOpsRuleEvaluationService"]
RuleEval --> SelfEval["self_evaluation.aiops_rule_evaluation"]
Report --> Trace["DiagnosisTraceService"]
SelfEval --> Trace
```
```text
POST /api/ai_ops
-> AiOpsService
-> 判断是否有告警 payload
-> PAYLOAD_TARGETED
-> AUTO_DISCOVERY
-> 构造 AIOps 诊断 prompt
-> payload 模式补充 recommended lookup_knowledge query
-> Agent 编排
-> Planner / Executor
-> Prometheus / logs / knowledge tools
-> 生成告警分析报告
-> AiOpsRuleEvaluationService
-> 合并 self_evaluation.aiops_rule_evaluation
-> Trace API 可查看全链路
```
AIOps 保留两种模式：
| 模式 | 触发条件 | 行为 |
|---|---|---|
| `PAYLOAD_TARGETED` | 请求包含 alertName、service、severity、description、timeRange 等字段 | 以 payload 为唯一主诊断对象，并生成推荐知识库 query |
| `AUTO_DISCOVERY` | 请求没有明确告警 payload | 先查询当前活跃告警，再选择目标排查 |
AIOps 当前使用轻量规则验证器，重点检查：
- 最终报告是否存在。
- payload 模式是否聚焦输入告警。
- 是否使用关键证据工具，例如 `lookup_knowledge`、日志、指标。
关键代码：
- `src/main/java/com/superbiz/agent/service/AiOpsService.java`
- `src/main/java/com/superbiz/agent/service/AiOpsRuleEvaluationService.java`
## 5. RAG 位置
RAG 不是隐藏在 Chat Advisor 里的隐式能力，而是 Executor 可以显式调用的工具：
```mermaid
flowchart LR
Executor["Executor Agent"] --> Tool["lookup_knowledge Tool"]
Tool --> L0["L0 domain/entity hint"]
Tool --> Search["VectorSearchService"]
L0 --> Search
Search --> VectorStore["Spring AI VectorStore"]
Search --> Fallback["Milvus SDK fallback"]
VectorStore --> Normalize["score/rawScore/scoreLabel"]
Fallback --> Normalize
Normalize --> Evidence["evidence output"]
Evidence --> Invocation["tool_invocation"]
Evidence --> Executor
```
```text
Executor
-> lookup_knowledge(query)
-> L0 domain/entity hint
-> VectorSearchService
-> Spring AI VectorStore
-> Milvus SDK fallback
-> evidence shaping
-> tool_invocation
```
保留显式工具的原因：
- Agent 何时检索、检索什么、证据是什么，必须能在 trace 中解释。
- AIOps payload 到 query 的业务映射需要项目内控制。
- `tool_invocation` 是后续评测、回放和面试讲解的核心材料。
RAG 总体设计见 [rag-architecture.md](rag-architecture.md)，检索运行细节见 [retrieval-observability.md](retrieval-observability.md)。
## 6. 持久化模型
当前诊断持久化以三张表为核心：
```text
diagnosis_session
-> final_report
-> 一次诊断会话的主记录
-> query / status / agent_flow / answer
-> self_evaluation
-> verifier_evaluation
-> aiops_rule_evaluation
-> step_count / tool_call_count / duration
agent_step
-> role
-> step input/output
-> execution order
-> Agent 模型调用步骤
-> step_index / agent_name
-> model_input / model_output / thought
-> duration / token_count
tool_invocation
-> tool_name
-> query
-> retrieval_layer
-> retrieval_details
-> evidence blocks
-> duration
-> 工具调用事实
-> tool_name / input_params / output_preview
-> retrieval_layer / retrieval_details
-> relevance_level / dedup_reason
-> duration / success
```
Trace API aggregates these records into a session-level view:
说明：
- Agent step sequence.
- Tool calls and retrieval details.
- Final diagnosis report.
- Chat verifier status.
- AIOps rule verifier status.
- 旧的 `diagnosis_record` 已不是当前主模型，迁移脚本中已经由 `diagnosis_session + agent_step + tool_invocation` 取代。
- `api_document` 仍用于文档元数据管理。
- 文档向量内容存放在 Milvus/Zilliz collection 中。
## 7. Quality Gates
会话和 Trace 生命周期见 [session-trace-lifecycle.md](session-trace-lifecycle.md)，完整数据关系见 [data-model.md](data-model.md)。
Current quality gates:
## 7. Trace API
- Chat verifier: LLM-based final answer verification for normal diagnosis.
- AIOps rule verifier: lightweight deterministic checks for alert-focused diagnosis.
- Diagnosis eval baseline: fixture-based evaluation for trace and evidence behavior.
- RAG retrieval baseline: golden query set with offline baseline report.
- Live RAG acceptance: post-reindex script for validating retrieval against the running stack.
```text
GET /api/diagnosis/{sessionId}/trace
```
These gates are intentionally layered. The MVP proves the Agent chain can produce evidence, persist it, and be inspected after execution.
Trace API 聚合：
## 8. Current Completion State
- 会话状态和最终报告。
- Agent step 序列。
- 工具调用和检索细节。
- Chat verifier 结果。
- AIOps rule evaluation 结果。
Completed for the current MVP stage:
Trace 是本项目区别于普通问答系统的关键：答案不是孤立文本，而是可以追溯到 Agent 决策、工具调用和证据来源。
- Explicit `lookup_knowledge` Agent tool.
- L0 + L1 retrieval shape retained.
- L0 downgraded to domain/entity hint.
- Spring AI VectorStore retrieval path integrated.
- Milvus SDK fallback retained.
- RAG evidence post-processing added.
- Breadcrumb/title/content embedding text improved.
- RAG offline baseline and live acceptance script added.
- AIOps payload query augmentation added.
- AIOps lightweight verifier added.
- Trace summary includes both chat verifier and AIOps verifier signals.
Prompt、Hook、Verifier 和评测门禁的完整说明见 [harness-quality-gates.md](harness-quality-gates.md)，用户反馈与 `self_evaluation` 闭环见 [feedback-architecture.md](feedback-architecture.md)。
Deferred future enhancements:
## 8. 质量门禁
- LLM QueryTransformer / MultiQuery.
- BM25, RRF, and reranker.
- Neighbor chunk or section-level context expansion.
- VectorStore write path migration.
- Full LLM-based AIOps verifier.
- More complete golden set for recall, MRR, and nDCG metrics.
当前质量门禁分层如下：
## 9. Key Code References
| 门禁 | 位置 | 作用 |
|---|---|---|
| Chat Verifier | `ChatService` | 校验普通诊断回答质量 |
| AIOps Rule Evaluation | `AiOpsRuleEvaluationService` | 校验告警诊断是否聚焦 payload 并使用证据 |
| Diagnosis Eval Baseline | `mvp/eval/` | 固化诊断 trace 和报告行为 |
| RAG Retrieval Baseline | `eval/rag-retrieval/` | 固化检索召回行为，避免 RAG 重构回退 |
| Live RAG Acceptance | `scripts/eval_rag_live_acceptance.py` | 在运行环境中验证重建索引后的真实检索 |
- `src/main/java/com/superbiz/agent/service/ChatService.java`
- `src/main/java/com/superbiz/agent/service/AiOpsService.java`
- `src/main/java/com/superbiz/agent/service/AiOpsRuleEvaluationService.java`
- `src/main/java/com/superbiz/agent/tool/LookupKnowledgeTool.java`
- `src/main/java/com/superbiz/agent/service/VectorSearchService.java`
- `src/main/java/com/superbiz/agent/service/VectorIndexService.java`
- `src/main/java/com/superbiz/agent/service/SpringAiVectorStoreSidecarService.java`
- `src/main/java/com/superbiz/agent/service/DiagnosisTraceService.java`
- `src/main/java/com/superbiz/agent/service/ToolInvocationRecorder.java`
- `src/main/java/com/superbiz/agent/service/SelfEvaluationMergeService.java`
## 9. 当前完成状态
## 10. Supporting Materials
已经完成：
- `mvp/issues/rag-refactor-plan.md`
- `eval/rag-retrieval/README.md`
- `scripts/eval_rag_live_acceptance.py`
- `interview/rag-refactor-story.md`
- `interview/rag-vectorstore-interview-notes.md`
- `interview/rag-retrieval-quality-report.md`
- `interview/rag-breadcrumb-embedding-acceptance.md`
- `interview/aiops-query-augmentation.md`
- `interview/aiops-lightweight-verifier.md`
- Chat 和 AIOps 两条入口链路。
- 显式 `lookup_knowledge` Agent Tool。
- L0 从最终决策降级为 domain/entity hint。
- `VectorSearchService` 作为稳定检索门面。
- Spring AI VectorStore 读取路径。
- Milvus SDK fallback。
- `score` / `rawScore` / `scoreLabel` 分数语义拆分。
- `title`、`breadcrumb`、`content` 参与 embedding 文本。
- `tool_invocation` 记录检索层、relevance level、dedup reason。
- Chat verifier 和 AIOps rule evaluation 合并进 `self_evaluation`。
- RAG offline baseline 和 live acceptance 脚本。
暂不作为当前已完成能力声明：
- 完整 QueryTransformer / MultiQuery。
- BM25、RRF、cross-encoder rerank。
- 完整邻居 chunk / section context expansion。
- VectorStore 写入路径全面迁移。
- 完整 LLM-based AIOps verifier。
后续 Agent 拆分、Skill/Playbook、MCP 工具协议化和进程隔离等方向见 [evolution-roadmap.md](evolution-roadmap.md)。
## 10. 关键代码索引
| 能力 | 代码 |
|---|---|
| Chat 入口与编排 | `ChatController`, `ChatService` |
| AIOps 入口与编排 | `ChatController.aiOps`, `AiOpsService` |
| AIOps 规则验证 | `AiOpsRuleEvaluationService` |
| 知识库工具 | `LookupKnowledgeTool` |
| L0 hint | `KnowledgeIndexService` |
| 向量检索门面 | `VectorSearchService` |
| 文档切片 | `DocumentChunkService` |
| 向量写入 | `VectorIndexService` |
| Spring AI VectorStore 配置辅助 | `SpringAiVectorStoreSidecarService` |
| Trace 聚合 | `DiagnosisTraceService` |
| 工具调用记录 | `ToolInvocationRecorder` |
| self_evaluation 合并 | `SelfEvaluationMergeService` |