**变更概述:** - 将 MVP 架构设计文档独立到项目根目录 `mvp/` - 整理 `docs/` 为纯学习和分析文档目录 - 按类型分类:learning(学习)、analysis(分析)、reports(报告)、guides(指南) **目录结构:** ``` mvp/ # MVP 架构设计(独立) ├── README.md # 数据库设计总览 ├── architecture/ # 架构文档 │ ├── agent-architecture-mvp.md │ ├── implementation-plan.md │ └── ... └── tables/ # 数据表设计 docs/ # 学习和分析文档 ├── learning/ # 学习笔记(00-08 编号) ├── analysis/ # 分析笔记 + 重构计划 ├── reports/ # 临时报告 └── guides/ # 指南文档 ``` **详细变更:** - docs/README.md → mvp/README.md(数据库设计入口) - docs/architecture/ → mvp/architecture/(架构设计) - docs/tables/ → mvp/tables/(数据表设计) - docs/学习笔记-*.md → docs/learning/07-*.md, 08-*.md - docs/项目学习路径.md → docs/learning/00-*.md - docs/功能分析报告.md → docs/analysis/ - docs/修复报告-*.md → docs/reports/ - docs/日志配置*.md → docs/guides/ 或 docs/reports/ - docs/design/ → docs/analysis/(问题分析和重构计划)
1712 lines
48 KiB
Markdown
1712 lines
48 KiB
Markdown
# 数据库设计文档
|
||
|
||
## 一、设计原则
|
||
|
||
### 1.1 核心原则
|
||
- ✅ **简单优先**:满足诊断流程需要,避免过度设计
|
||
- ✅ **渐进增强**:先实现核心功能,再逐步扩展
|
||
- ✅ **数据分离**:诊断结果持久化(MySQL),会话上下文临时化(Redis)
|
||
- ✅ **适度冗余**:避免过度范式化,适当冗余提升查询性能
|
||
|
||
### 1.2 系统定位
|
||
**自动化诊断系统**
|
||
- 核心:一键诊断 → 返回完整报告
|
||
- 辅助:支持追问,但不是主要场景
|
||
- 特点:大部分用户单次诊断即结束,少数用户会追问细节
|
||
|
||
---
|
||
|
||
## 二、核心表设计
|
||
|
||
### 2.1 diagnosis_record(诊断记录表)
|
||
|
||
#### 设计理念:兼容多种故障类型
|
||
|
||
**问题背景**:
|
||
- 初始设计过于聚焦"外部接口故障"(省份、接口URL、业务错误码)
|
||
- 实际故障类型更丰富:空指针异常、数据库死锁、缓存穿透、线程池耗尽等
|
||
- 需要字段泛化,支持**外部接口故障 + 系统内部错误**
|
||
|
||
**解决方案**:
|
||
- 字段泛化:business_id 替代 order_id,fault_source 替代 province
|
||
- 增加分类:fault_category 显式区分故障类别
|
||
- 增强错误信息:error_message、stack_trace 支持内部错误
|
||
|
||
---
|
||
|
||
#### 表结构(v2.0 泛化版)
|
||
|
||
```sql
|
||
CREATE TABLE diagnosis_record (
|
||
-- 主键
|
||
id BIGINT PRIMARY KEY AUTO_INCREMENT,
|
||
diagnosis_id VARCHAR(64) UNIQUE NOT NULL COMMENT '诊断唯一ID(UUID)',
|
||
|
||
-- 关联信息
|
||
session_id VARCHAR(64) COMMENT '会话ID(关联Redis)',
|
||
business_id VARCHAR(128) COMMENT '业务标识(订单号/请求ID/线程ID/任务ID...)',
|
||
trace_id VARCHAR(64) COMMENT '链路追踪ID',
|
||
|
||
-- 故障分类(泛化设计)
|
||
fault_category VARCHAR(32) COMMENT '故障类别(EXTERNAL_API/INTERNAL_ERROR/DATABASE/CACHE/NETWORK/THREAD/MEMORY/CONFIG)',
|
||
fault_source VARCHAR(128) COMMENT '故障源(省份/服务名/类名/数据库实例/Redis集群...)',
|
||
fault_target VARCHAR(256) COMMENT '故障目标(接口URL/方法名/SQL语句/缓存键...)',
|
||
|
||
-- 错误信息(通用)
|
||
error_code VARCHAR(64) COMMENT '错误码(业务错误码/HTTP状态码/异常类名/数据库错误码)',
|
||
error_message TEXT COMMENT '错误消息',
|
||
stack_trace TEXT COMMENT '堆栈信息(内部错误时记录)',
|
||
|
||
-- 诊断结果
|
||
problem_type VARCHAR(32) COMMENT '问题类型(参数/网络/权限/逻辑/空指针/死锁/缓存穿透/线程池满...)',
|
||
root_cause TEXT COMMENT '根因分析',
|
||
solution TEXT COMMENT '修复方案',
|
||
report_markdown TEXT COMMENT '完整诊断报告(Markdown格式)',
|
||
|
||
-- 评估指标
|
||
status VARCHAR(16) DEFAULT 'PENDING' COMMENT '诊断状态(PENDING/RUNNING/SUCCESS/FAILED)',
|
||
confidence INT COMMENT '诊断置信度(0-100)',
|
||
duration INT COMMENT '诊断耗时(毫秒)',
|
||
|
||
-- 调试字段(可选)
|
||
tool_calls JSON COMMENT '工具调用记录',
|
||
|
||
-- 元数据
|
||
created_by VARCHAR(64) COMMENT '创建人',
|
||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
|
||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间',
|
||
|
||
-- 索引
|
||
INDEX idx_business_id (business_id),
|
||
INDEX idx_trace_id (trace_id),
|
||
INDEX idx_session_id (session_id),
|
||
INDEX idx_fault_category (fault_category),
|
||
INDEX idx_fault_source_target (fault_source, fault_target(100)),
|
||
INDEX idx_error_code (error_code),
|
||
INDEX idx_created_at (created_at),
|
||
INDEX idx_status (status)
|
||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='诊断记录表(v2.0 泛化版)';
|
||
```
|
||
|
||
#### 字段说明(泛化后)
|
||
|
||
| 字段 | 类型 | 必填 | 说明 |
|
||
|------|------|------|------|
|
||
| diagnosis_id | VARCHAR(64) | 是 | 诊断唯一标识(UUID) |
|
||
| session_id | VARCHAR(64) | 否 | 会话ID,关联Redis会话上下文 |
|
||
| business_id | VARCHAR(128) | 否 | **泛化**:业务标识(订单号/请求ID/线程ID/任务ID),根据故障类型灵活填写 |
|
||
| trace_id | VARCHAR(64) | 否 | 链路追踪ID,用于串联日志 |
|
||
| fault_category | VARCHAR(32) | 否 | **新增**:故障类别(EXTERNAL_API/INTERNAL_ERROR/DATABASE/CACHE/NETWORK/THREAD/MEMORY/CONFIG) |
|
||
| fault_source | VARCHAR(128) | 否 | **泛化**:故障源(省份/服务名/类名/数据库实例),根据故障类别填写 |
|
||
| fault_target | VARCHAR(256) | 否 | **泛化**:故障目标(接口URL/方法名/SQL语句/缓存键),描述具体位置 |
|
||
| error_code | VARCHAR(64) | 否 | **扩展**:错误码(业务错误码/HTTP状态码/异常类名/数据库错误码) |
|
||
| error_message | TEXT | 否 | **新增**:错误消息,通用描述 |
|
||
| stack_trace | TEXT | 否 | **新增**:堆栈信息,内部错误时记录 |
|
||
| problem_type | VARCHAR(32) | 否 | 问题类型(参数/网络/权限/逻辑/空指针/死锁/缓存穿透/线程池满...) |
|
||
| root_cause | TEXT | 否 | 根因分析 |
|
||
| solution | TEXT | 否 | 修复方案 |
|
||
| report_markdown | TEXT | 否 | 完整诊断报告(Markdown格式) |
|
||
| status | VARCHAR(16) | 是 | 诊断状态(PENDING/RUNNING/SUCCESS/FAILED) |
|
||
| confidence | INT | 否 | 置信度(0-100) |
|
||
| duration | INT | 否 | 诊断耗时(毫秒) |
|
||
| tool_calls | JSON | 否 | 工具调用记录 |
|
||
|
||
#### 核心设计决策
|
||
|
||
**1. 字段泛化:支持多种故障类型**
|
||
|
||
**泛化前 → 泛化后**:
|
||
```
|
||
order_id (订单号) → business_id (业务标识)
|
||
province (省份) → fault_source (故障源)
|
||
api_name (接口名称) → 移除(信息合并到 fault_target)
|
||
api_url (接口URL) → fault_target (故障目标)
|
||
error_code (业务错误码) → error_code (通用错误码,扩展支持)
|
||
新增 fault_category (故障类别)
|
||
新增 error_message (错误消息)
|
||
新增 stack_trace (堆栈信息)
|
||
```
|
||
|
||
**设计理由**:
|
||
- 原设计假设"故障 = 外部接口调用失败",但实际故障类型更多样
|
||
- 泛化后支持:外部接口故障、内部异常、数据库问题、缓存问题、线程问题等
|
||
- 字段语义更通用,根据故障类型灵活填写
|
||
|
||
---
|
||
|
||
**2. fault_category 枚举值**
|
||
|
||
```
|
||
故障类别分类:
|
||
├─ EXTERNAL_API:外部接口调用失败(第三方API、政府接口)
|
||
├─ INTERNAL_ERROR:系统内部错误(空指针、NPE、业务异常)
|
||
├─ DATABASE:数据库问题(死锁、慢查询、连接池耗尽)
|
||
├─ CACHE:缓存问题(穿透、雪崩、击穿)
|
||
├─ NETWORK:网络问题(超时、连接失败、DNS解析失败)
|
||
├─ THREAD:线程问题(线程池满、死锁)
|
||
├─ MEMORY:内存问题(OOM、内存泄漏)
|
||
└─ CONFIG:配置问题(配置错误、配置缺失)
|
||
|
||
用途:
|
||
- 统计不同类别故障的分布
|
||
- 路由不同的诊断策略(不同类别使用不同工具)
|
||
- 支持按类别过滤查询
|
||
```
|
||
|
||
---
|
||
|
||
**3. 字段映射示例**
|
||
|
||
**示例1:外部接口故障(原场景)**
|
||
```
|
||
business_id: "202406150001" (订单号)
|
||
fault_category: "EXTERNAL_API"
|
||
fault_source: "广东" (省份)
|
||
fault_target: "/api/v1/guangdong/social-security" (接口URL)
|
||
error_code: "40003" (业务错误码)
|
||
error_message: "参数缺失:idCard"
|
||
stack_trace: NULL (无堆栈)
|
||
```
|
||
|
||
**示例2:空指针异常(内部错误)**
|
||
```
|
||
business_id: "req-xyz789" (请求ID)
|
||
fault_category: "INTERNAL_ERROR"
|
||
fault_source: "order-service" (服务名)
|
||
fault_target: "OrderController.createOrder()" (方法名)
|
||
error_code: "NullPointerException" (异常类名)
|
||
error_message: "Cannot invoke 'User.getName()' because 'user' is null"
|
||
stack_trace: "java.lang.NullPointerException: ...\n at OrderController.java:45\n ..." (完整堆栈)
|
||
```
|
||
|
||
**示例3:数据库死锁**
|
||
```
|
||
business_id: "txn-20240615-001" (事务ID)
|
||
fault_category: "DATABASE"
|
||
fault_source: "mysql-master-01" (数据库实例)
|
||
fault_target: "UPDATE orders SET status=? WHERE order_id=?" (SQL)
|
||
error_code: "1213" (MySQL死锁错误码)
|
||
error_message: "Deadlock found when trying to get lock"
|
||
stack_trace: NULL (数据库错误无堆栈)
|
||
```
|
||
|
||
**示例4:缓存穿透**
|
||
```
|
||
business_id: "cache-key-user:99999" (缓存键)
|
||
fault_category: "CACHE"
|
||
fault_source: "redis-cluster" (Redis集群)
|
||
fault_target: "user:99999" (缓存键)
|
||
error_code: "CACHE_MISS" (自定义)
|
||
error_message: "恶意查询不存在的用户ID,导致缓存穿透"
|
||
stack_trace: NULL
|
||
```
|
||
|
||
**示例5:线程池耗尽**
|
||
```
|
||
business_id: "pool-async-executor" (线程池名称)
|
||
fault_category: "THREAD"
|
||
fault_source: "order-service" (服务名)
|
||
fault_target: "asyncExecutor ThreadPool" (线程池)
|
||
error_code: "RejectedExecutionException" (异常类名)
|
||
error_message: "Task rejected from ThreadPoolExecutor"
|
||
stack_trace: "java.util.concurrent.RejectedExecutionException: ...\n at ThreadPoolExecutor.java:2063\n ..."
|
||
```
|
||
|
||
---
|
||
|
||
**4. 向下兼容策略**
|
||
|
||
如果已有数据使用旧字段(order_id、province、api_url),可以通过以下方式迁移:
|
||
|
||
```sql
|
||
-- 数据迁移脚本
|
||
UPDATE diagnosis_record SET
|
||
business_id = order_id,
|
||
fault_category = 'EXTERNAL_API',
|
||
fault_source = province,
|
||
fault_target = api_url,
|
||
error_message = CONCAT('错误码: ', error_code)
|
||
WHERE fault_category IS NULL;
|
||
```
|
||
|
||
应用层可以同时支持新旧字段:
|
||
```
|
||
读取时:优先使用新字段,兼容旧字段
|
||
写入时:只写新字段
|
||
```
|
||
|
||
---
|
||
|
||
**5. 一次诊断 = 一条记录**
|
||
```
|
||
特点:
|
||
- 用户发起一次诊断任务,创建一条记录
|
||
- 不是聊天记录(不存多轮对话)
|
||
- 追问对话存在 conversation_history 表(可选)
|
||
|
||
示例:
|
||
用户:"诊断订单 202406150001"
|
||
→ 创建 diagnosis_record(status=RUNNING)
|
||
→ Agent 执行
|
||
→ 更新 diagnosis_record(status=SUCCESS)
|
||
→ 返回报告
|
||
```
|
||
|
||
---
|
||
|
||
**6. report_markdown 字段的必要性**
|
||
```
|
||
为什么要存储完整报告?
|
||
- 固化结果:Prompt变化不影响历史报告
|
||
- 快速展示:不需要重新生成
|
||
- 历史审计:可以看到当时的诊断结果
|
||
|
||
成本:
|
||
- 字段较大(TEXT类型)
|
||
- 有一定冗余
|
||
|
||
结论:存储,因为报告是最终产物
|
||
```
|
||
|
||
---
|
||
|
||
**7. session_id 的作用**
|
||
```
|
||
用途:
|
||
- 关联 Redis 会话(支持追问)
|
||
- 同一会话可能有多次诊断
|
||
- 用于会话级别的数据分析
|
||
|
||
场景:
|
||
用户:"诊断订单 A"(session_id=sess-001, diagnosis_id=diag-001)
|
||
用户:"再诊断订单 B"(session_id=sess-001, diagnosis_id=diag-002)
|
||
→ 同一会话,两次诊断
|
||
```
|
||
|
||
#### 典型查询场景
|
||
|
||
**1. 查询历史诊断**
|
||
```sql
|
||
-- 按业务标识查询(兼容订单号、请求ID等)
|
||
SELECT * FROM diagnosis_record
|
||
WHERE business_id = '202406150001'
|
||
ORDER BY created_at DESC;
|
||
|
||
-- 按链路ID查询
|
||
SELECT * FROM diagnosis_record
|
||
WHERE trace_id = 'trace-abc-123'
|
||
ORDER BY created_at DESC;
|
||
```
|
||
|
||
**2. 按故障类别统计**
|
||
```sql
|
||
-- 统计最近7天各类别故障分布
|
||
SELECT
|
||
fault_category,
|
||
COUNT(*) as count,
|
||
ROUND(AVG(duration), 2) as avg_duration_ms,
|
||
ROUND(AVG(confidence), 2) as avg_confidence
|
||
FROM diagnosis_record
|
||
WHERE created_at >= DATE_SUB(NOW(), INTERVAL 7 DAY)
|
||
GROUP BY fault_category
|
||
ORDER BY count DESC;
|
||
|
||
-- 结果示例:
|
||
-- +------------------+-------+-----------------+----------------+
|
||
-- | fault_category | count | avg_duration_ms | avg_confidence |
|
||
-- +------------------+-------+-----------------+----------------+
|
||
-- | EXTERNAL_API | 120 | 5234.5 | 85.3 |
|
||
-- | INTERNAL_ERROR | 45 | 3456.2 | 90.1 |
|
||
-- | DATABASE | 20 | 4567.8 | 88.5 |
|
||
-- | CACHE | 15 | 2345.1 | 82.0 |
|
||
-- | THREAD | 5 | 6789.3 | 87.2 |
|
||
-- +------------------+-------+-----------------+----------------+
|
||
```
|
||
|
||
**3. 内部错误Top异常统计**
|
||
```sql
|
||
-- 统计内部错误中最频繁的异常
|
||
SELECT
|
||
error_code,
|
||
fault_target,
|
||
COUNT(*) as count,
|
||
AVG(duration) as avg_duration
|
||
FROM diagnosis_record
|
||
WHERE fault_category = 'INTERNAL_ERROR'
|
||
AND created_at >= DATE_SUB(NOW(), INTERVAL 30 DAY)
|
||
GROUP BY error_code, fault_target
|
||
ORDER BY count DESC
|
||
LIMIT 10;
|
||
|
||
-- 结果示例:
|
||
-- +---------------------------+--------------------------------+-------+--------------+
|
||
-- | error_code | fault_target | count | avg_duration |
|
||
-- +---------------------------+--------------------------------+-------+--------------+
|
||
-- | NullPointerException | OrderController.createOrder() | 15 | 3245.2 |
|
||
-- | IllegalArgumentException | UserService.validateUser() | 10 | 2567.8 |
|
||
-- | NullPointerException | PaymentService.processPayment()| 8 | 4123.5 |
|
||
-- +---------------------------+--------------------------------+-------+--------------+
|
||
```
|
||
|
||
**4. 外部接口故障统计(按省份)**
|
||
```sql
|
||
-- 统计外部接口故障(按省份)
|
||
SELECT
|
||
fault_source as province,
|
||
COUNT(*) as count
|
||
FROM diagnosis_record
|
||
WHERE fault_category = 'EXTERNAL_API'
|
||
AND created_at >= DATE_SUB(NOW(), INTERVAL 30 DAY)
|
||
GROUP BY fault_source
|
||
ORDER BY count DESC;
|
||
|
||
-- 结果示例:
|
||
-- +----------+-------+
|
||
-- | province | count |
|
||
-- +----------+-------+
|
||
-- | 广东 | 45 |
|
||
-- | 江苏 | 32 |
|
||
-- | 浙江 | 28 |
|
||
-- +----------+-------+
|
||
```
|
||
|
||
**5. 数据库问题分析**
|
||
```sql
|
||
-- 统计数据库问题(按错误码)
|
||
SELECT
|
||
error_code,
|
||
COUNT(*) as count,
|
||
fault_target as example_sql
|
||
FROM diagnosis_record
|
||
WHERE fault_category = 'DATABASE'
|
||
AND created_at >= DATE_SUB(NOW(), INTERVAL 30 DAY)
|
||
GROUP BY error_code, fault_target
|
||
ORDER BY count DESC
|
||
LIMIT 5;
|
||
|
||
-- 结果示例:
|
||
-- +------------+-------+---------------------------------------------+
|
||
-- | error_code | count | example_sql |
|
||
-- +------------+-------+---------------------------------------------+
|
||
-- | 1213 | 12 | UPDATE orders SET status=? WHERE order_id=? |
|
||
-- | 1205 | 8 | SELECT * FROM orders WHERE user_id=? |
|
||
-- +------------+-------+---------------------------------------------+
|
||
```
|
||
|
||
**6. 诊断成功率统计**
|
||
```sql
|
||
-- 统计最近7天的诊断成功率
|
||
SELECT
|
||
COUNT(*) as total,
|
||
SUM(CASE WHEN status = 'SUCCESS' THEN 1 ELSE 0 END) as success,
|
||
ROUND(SUM(CASE WHEN status = 'SUCCESS' THEN 1 ELSE 0 END) * 100.0 / COUNT(*), 2) as success_rate
|
||
FROM diagnosis_record
|
||
WHERE created_at >= DATE_SUB(NOW(), INTERVAL 7 DAY);
|
||
```
|
||
|
||
**7. 性能监控(P50/P90/P95)**
|
||
```sql
|
||
-- MySQL 8.0+ 使用 PERCENTILE_CONT
|
||
SELECT
|
||
'P50' as metric,
|
||
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY duration) as value_ms
|
||
FROM diagnosis_record
|
||
WHERE status = 'SUCCESS'
|
||
AND created_at >= DATE_SUB(NOW(), INTERVAL 7 DAY)
|
||
UNION ALL
|
||
SELECT 'P90', PERCENTILE_CONT(0.9) WITHIN GROUP (ORDER BY duration)
|
||
FROM diagnosis_record
|
||
WHERE status = 'SUCCESS'
|
||
AND created_at >= DATE_SUB(NOW(), INTERVAL 7 DAY)
|
||
UNION ALL
|
||
SELECT 'P95', PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration)
|
||
FROM diagnosis_record
|
||
WHERE status = 'SUCCESS'
|
||
AND created_at >= DATE_SUB(NOW(), INTERVAL 7 DAY);
|
||
|
||
-- 或使用近似方式(兼容旧版本MySQL)
|
||
SELECT
|
||
'P50' as metric,
|
||
duration as value_ms
|
||
FROM (
|
||
SELECT duration, ROW_NUMBER() OVER (ORDER BY duration) as rn,
|
||
COUNT(*) OVER() as total
|
||
FROM diagnosis_record
|
||
WHERE status = 'SUCCESS'
|
||
AND created_at >= DATE_SUB(NOW(), INTERVAL 7 DAY)
|
||
) t
|
||
WHERE rn = FLOOR(total * 0.5);
|
||
```
|
||
|
||
---
|
||
|
||
### 2.2 case_library(案例库表)
|
||
|
||
#### 设计理念:知识沉淀,系统越用越智能
|
||
|
||
**核心价值**:
|
||
- 质量过滤:只存储高质量案例(成功诊断 + 用户反馈有用)
|
||
- 知识沉淀:历史诊断经验可复用
|
||
- 提升准确率:相似问题提供历史参考
|
||
- 加速诊断:快速推荐相似案例
|
||
|
||
**MVP版本设计原则**:
|
||
- ✅ 能用:满足基本案例推荐功能
|
||
- ✅ 简单:字段不多,逻辑清晰
|
||
- ✅ 可扩展:后续可增加字段
|
||
- ❌ 不做(Phase 2):复杂评分、版本管理、标签分类
|
||
|
||
---
|
||
|
||
#### 表结构(MVP版)
|
||
|
||
```sql
|
||
CREATE TABLE case_library (
|
||
-- 主键
|
||
id BIGINT PRIMARY KEY AUTO_INCREMENT,
|
||
case_id VARCHAR(64) UNIQUE NOT NULL COMMENT '案例唯一ID(UUID)',
|
||
|
||
-- 来源关联
|
||
diagnosis_id VARCHAR(64) COMMENT '关联诊断记录(可选,人工录入时为空)',
|
||
source_type VARCHAR(16) DEFAULT 'AUTO' COMMENT '来源类型(AUTO:自动生成/MANUAL:人工录入)',
|
||
|
||
-- 案例分类(复用 diagnosis_record 的分类字段)
|
||
fault_category VARCHAR(32) COMMENT '故障类别(EXTERNAL_API/INTERNAL_ERROR/DATABASE/CACHE/NETWORK/THREAD/MEMORY/CONFIG)',
|
||
fault_source VARCHAR(128) COMMENT '故障源(省份/服务名/类名/数据库实例...)',
|
||
error_code VARCHAR(64) COMMENT '错误码(业务错误码/HTTP状态码/异常类名)',
|
||
|
||
-- 案例内容
|
||
title VARCHAR(256) NOT NULL COMMENT '案例标题(简短描述,如"广东社保查询idCard字段缺失")',
|
||
root_cause TEXT NOT NULL COMMENT '根因分析',
|
||
solution TEXT NOT NULL COMMENT '解决方案',
|
||
|
||
-- 简单统计
|
||
reference_count INT DEFAULT 0 COMMENT '引用次数(被推荐的次数,用于排序)',
|
||
|
||
-- 元数据
|
||
created_by VARCHAR(64) COMMENT '创建人',
|
||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
|
||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间',
|
||
|
||
-- 索引
|
||
INDEX idx_fault_category (fault_category),
|
||
INDEX idx_error_code (error_code),
|
||
INDEX idx_fault_source (fault_source),
|
||
INDEX idx_diagnosis_id (diagnosis_id),
|
||
INDEX idx_reference_count (reference_count),
|
||
INDEX idx_created_at (created_at)
|
||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='案例库表(MVP版)';
|
||
```
|
||
|
||
#### 字段说明
|
||
|
||
| 字段 | 类型 | 必填 | 说明 |
|
||
|------|------|------|------|
|
||
| case_id | VARCHAR(64) | 是 | 案例唯一标识(UUID) |
|
||
| diagnosis_id | VARCHAR(64) | 否 | 关联诊断记录,追溯案例来源(人工录入时为空)|
|
||
| source_type | VARCHAR(16) | 是 | 来源类型:AUTO(自动生成)/MANUAL(人工录入)|
|
||
| fault_category | VARCHAR(32) | 否 | 故障类别,与 diagnosis_record 一致 |
|
||
| fault_source | VARCHAR(128) | 否 | 故障源,按省份/服务检索 |
|
||
| error_code | VARCHAR(64) | 否 | 错误码,精确匹配检索 |
|
||
| title | VARCHAR(256) | 是 | 案例标题,快速浏览 |
|
||
| root_cause | TEXT | 是 | 根因分析,核心内容 |
|
||
| solution | TEXT | 是 | 解决方案,核心内容 |
|
||
| reference_count | INT | 是 | 引用次数,用于简单排序(引用多的排前面)|
|
||
|
||
#### 核心设计决策
|
||
|
||
**1. 案例来源**
|
||
|
||
```
|
||
来源1:自动生成(source_type=AUTO)
|
||
├─ 触发条件:诊断成功 + 用户反馈"有用"
|
||
├─ 关联诊断:diagnosis_id 不为空
|
||
└─ 质量保证:用户验证过
|
||
|
||
来源2:人工录入(source_type=MANUAL)
|
||
├─ 运维团队总结的经典案例
|
||
├─ diagnosis_id 为空
|
||
└─ 质量最高
|
||
|
||
注意:诊断失败或用户反馈"无用"的不自动生成案例
|
||
```
|
||
|
||
**2. 简化的评分机制(MVP)**
|
||
|
||
```
|
||
MVP版本:只按 reference_count 排序
|
||
- 引用次数多的排前面
|
||
- 简单有效
|
||
|
||
Phase 2 可增强:
|
||
- 增加 useful_count(用户反馈有用次数)
|
||
- 增加 score(综合评分:引用加分 + 有用率加分 - 时间衰减)
|
||
- 增加 is_featured(人工标记的经典案例)
|
||
```
|
||
|
||
**3. 不做版本管理(MVP)**
|
||
|
||
```
|
||
当前:直接更新案例
|
||
UPDATE case_library
|
||
SET root_cause = '修正后的根因',
|
||
solution = '修正后的方案'
|
||
WHERE case_id = 'xxx';
|
||
|
||
优点:简单,保持单一案例
|
||
缺点:历史版本丢失
|
||
|
||
Phase 2 如需版本管理:
|
||
- 方案A:增加 version 字段
|
||
- 方案B:建 case_history 表
|
||
```
|
||
|
||
**4. 与 diagnosis_record 的关系**
|
||
|
||
```
|
||
关系:一对一(可选)
|
||
- 一次诊断 → 可以生成一个案例
|
||
- 通过 diagnosis_id 关联
|
||
- diagnosis_id 可为空(人工录入案例)
|
||
|
||
流程:
|
||
diagnosis_record(成功)
|
||
↓
|
||
用户反馈"有用"
|
||
↓
|
||
自动生成 case_library
|
||
↓
|
||
后续可人工修正、合并相似案例
|
||
```
|
||
|
||
#### 数据流
|
||
|
||
**场景1:自动生成案例**
|
||
```
|
||
诊断完成 + 用户反馈"有用"
|
||
↓
|
||
INSERT INTO case_library
|
||
- diagnosis_id: diag-001
|
||
- source_type: AUTO
|
||
- fault_category: INTERNAL_ERROR
|
||
- error_code: NullPointerException
|
||
- title: "订单服务创建订单空指针异常"
|
||
- root_cause: "OrderController.createOrder()方法中user对象为null,未做空判断"
|
||
- solution: "在第45行添加空判断:if (user == null) throw new BizException(...)"
|
||
- reference_count: 0
|
||
```
|
||
|
||
**场景2:人工录入案例**
|
||
```
|
||
运维团队总结经验
|
||
↓
|
||
INSERT INTO case_library
|
||
- diagnosis_id: NULL
|
||
- source_type: MANUAL
|
||
- fault_category: EXTERNAL_API
|
||
- error_code: 40003
|
||
- title: "广东省社保接口参数缺失通用处理"
|
||
- root_cause: "前端表单未做必填校验,导致请求报文缺少关键参数"
|
||
- solution: "前端增加必填校验;后端返回明确的字段缺失提示"
|
||
- reference_count: 0
|
||
```
|
||
|
||
**场景3:推荐案例并更新引用次数**
|
||
```
|
||
诊断时查询相似案例
|
||
↓
|
||
SELECT * FROM case_library
|
||
WHERE error_code = '40003'
|
||
AND fault_category = 'EXTERNAL_API'
|
||
ORDER BY reference_count DESC
|
||
LIMIT 3;
|
||
↓
|
||
返回 Top 3 案例
|
||
↓
|
||
UPDATE case_library
|
||
SET reference_count = reference_count + 1
|
||
WHERE case_id IN ('case-001', 'case-005', 'case-012');
|
||
```
|
||
|
||
#### 典型查询场景
|
||
|
||
**1. 精确匹配查询(优先)**
|
||
```sql
|
||
-- 按错误码查询
|
||
SELECT * FROM case_library
|
||
WHERE error_code = '40003'
|
||
ORDER BY reference_count DESC
|
||
LIMIT 5;
|
||
|
||
-- 按故障类别 + 错误码查询
|
||
SELECT * FROM case_library
|
||
WHERE fault_category = 'INTERNAL_ERROR'
|
||
AND error_code = 'NullPointerException'
|
||
ORDER BY reference_count DESC
|
||
LIMIT 5;
|
||
|
||
-- 按故障源查询(如省份、服务名)
|
||
SELECT * FROM case_library
|
||
WHERE fault_source = '广东'
|
||
AND fault_category = 'EXTERNAL_API'
|
||
ORDER BY reference_count DESC
|
||
LIMIT 5;
|
||
```
|
||
|
||
**2. 统计分析**
|
||
```sql
|
||
-- 统计案例分布
|
||
SELECT
|
||
fault_category,
|
||
COUNT(*) as count,
|
||
AVG(reference_count) as avg_reference
|
||
FROM case_library
|
||
GROUP BY fault_category
|
||
ORDER BY count DESC;
|
||
|
||
-- Top 引用案例
|
||
SELECT title, reference_count, created_at
|
||
FROM case_library
|
||
ORDER BY reference_count DESC
|
||
LIMIT 10;
|
||
|
||
-- 人工录入的案例
|
||
SELECT * FROM case_library
|
||
WHERE source_type = 'MANUAL'
|
||
ORDER BY created_at DESC;
|
||
```
|
||
|
||
**3. 案例查重(避免重复)**
|
||
```sql
|
||
-- 检查是否已有相同错误码的案例
|
||
SELECT * FROM case_library
|
||
WHERE error_code = '40003'
|
||
AND fault_source = '广东'
|
||
AND fault_category = 'EXTERNAL_API';
|
||
```
|
||
|
||
#### 数据示例
|
||
|
||
```sql
|
||
-- 外部接口故障案例
|
||
INSERT INTO case_library VALUES
|
||
(1, 'case-001', 'diag-001', 'AUTO', 'EXTERNAL_API', '广东', '40003',
|
||
'广东社保查询idCard字段缺失',
|
||
'请求报文中未传入idCard字段,导致参数校验失败',
|
||
'前端表单增加idCard必填校验;后端增加参数校验提示',
|
||
15, 'system', NOW(), NOW());
|
||
|
||
-- 内部错误案例
|
||
INSERT INTO case_library VALUES
|
||
(2, 'case-002', 'diag-045', 'AUTO', 'INTERNAL_ERROR', 'order-service', 'NullPointerException',
|
||
'订单服务创建订单空指针异常',
|
||
'OrderController.createOrder()方法中user对象为null,未做空判断',
|
||
'在第45行添加空判断:if (user == null) throw new BizException("用户信息不存在")',
|
||
8, 'system', NOW(), NOW());
|
||
|
||
-- 人工录入的经典案例
|
||
INSERT INTO case_library VALUES
|
||
(3, 'case-003', NULL, 'MANUAL', 'DATABASE', 'mysql-master-01', '1213',
|
||
'订单库存更新死锁通用处理',
|
||
'两个事务互相等待对方释放锁:事务A持有订单锁等待库存锁,事务B持有库存锁等待订单锁',
|
||
'调整事务加锁顺序:统一先锁订单,再锁库存;或使用乐观锁方案',
|
||
3, 'admin', NOW(), NOW());
|
||
|
||
-- 缓存问题案例
|
||
INSERT INTO case_library VALUES
|
||
(4, 'case-004', 'diag-078', 'AUTO', 'CACHE', 'redis-cluster', 'CACHE_MISS',
|
||
'用户信息缓存穿透',
|
||
'恶意查询不存在的用户ID,缓存未命中,每次都打到数据库',
|
||
'使用布隆过滤器拦截无效查询;或缓存空结果(TTL 5分钟)',
|
||
5, 'system', NOW(), NOW());
|
||
```
|
||
|
||
#### 与 Milvus 向量库的配合
|
||
|
||
```
|
||
案例检索策略(混合检索):
|
||
|
||
1. 精确匹配(MySQL)
|
||
- 按 error_code 精确查询
|
||
- 按 fault_category + fault_source 组合查询
|
||
- 优点:快速、准确
|
||
- 缺点:只能匹配相同错误码
|
||
|
||
2. 语义检索(Milvus)
|
||
- 将案例内容(title + root_cause + solution)向量化
|
||
- 存储到 Milvus 的 case_library_collection
|
||
- 按语义相似度查询
|
||
- 优点:能找到相似但不同错误码的案例
|
||
- 缺点:召回可能不精确
|
||
|
||
3. 混合策略(推荐)
|
||
Step 1: 先精确匹配(MySQL)
|
||
Step 2: 如果结果 < 3 个,补充语义检索(Milvus)
|
||
Step 3: 合并去重,按 reference_count 排序
|
||
Step 4: 返回 Top 5
|
||
|
||
Milvus Collection 设计:
|
||
{
|
||
"collection_name": "case_library_collection",
|
||
"fields": [
|
||
{"name": "case_id", "type": "VARCHAR"},
|
||
{"name": "embedding", "type": "FLOAT_VECTOR", "dim": 1536},
|
||
{"name": "reference_count", "type": "INT32"}
|
||
],
|
||
"metric_type": "COSINE"
|
||
}
|
||
```
|
||
|
||
#### MVP 版本的简化
|
||
|
||
```
|
||
Phase 1(当前):
|
||
✅ 基础字段和表结构
|
||
✅ 自动生成案例(诊断成功 + 用户反馈)
|
||
✅ 人工录入案例
|
||
✅ 按 reference_count 简单排序
|
||
✅ 精确匹配查询
|
||
|
||
Phase 2(未来增强):
|
||
❌ 复杂评分机制(useful_count + score)
|
||
❌ 版本管理(case_history 表)
|
||
❌ 标签分类(tags 字段)
|
||
❌ 案例合并功能(多个相似案例 → 1个综合案例)
|
||
❌ 人工标注(is_featured 字段)
|
||
❌ 时间衰减(score 计算中考虑时间因素)
|
||
```
|
||
|
||
---
|
||
|
||
### 2.4 api_document(文档元数据表)
|
||
|
||
#### 设计理念:文档管理,不是文档检索
|
||
|
||
**核心定位**:
|
||
- MySQL 负责文档元数据管理(状态、版本、去重)
|
||
- Milvus 负责文档内容存储和检索
|
||
- 通过 doc_id 关联两者
|
||
|
||
**MVP版本原则**:
|
||
- ✅ 最简字段,满足基本管理需求
|
||
- ✅ 文件去重(基于 file_hash)
|
||
- ✅ 状态追踪(索引进度)
|
||
- ✅ 硬删除(同步删除 Milvus 数据)
|
||
- ❌ 暂不支持:软删除、启用开关、版本管理(Phase 2)
|
||
|
||
---
|
||
|
||
#### 表结构(MVP版)
|
||
|
||
```sql
|
||
CREATE TABLE api_document (
|
||
-- 主键
|
||
id BIGINT PRIMARY KEY AUTO_INCREMENT,
|
||
doc_id VARCHAR(64) UNIQUE NOT NULL COMMENT '文档唯一ID(UUID),关联Milvus',
|
||
|
||
-- 文档分类
|
||
fault_category VARCHAR(32) DEFAULT 'EXTERNAL_API' COMMENT '文档类别(EXTERNAL_API/INTERNAL_ERROR...)',
|
||
fault_source VARCHAR(128) COMMENT '文档归属(省份/服务名,如"广东"/"order-service")',
|
||
api_name VARCHAR(128) COMMENT '接口名称(如"社保查询")',
|
||
version VARCHAR(32) DEFAULT 'v1.0' COMMENT '文档版本',
|
||
|
||
-- 文件信息
|
||
file_name VARCHAR(256) NOT NULL COMMENT '原始文件名',
|
||
file_path VARCHAR(512) COMMENT '文件存储路径',
|
||
file_hash VARCHAR(64) COMMENT '文件MD5 hash(用于去重)',
|
||
file_size BIGINT COMMENT '文件大小(字节)',
|
||
|
||
-- 索引状态
|
||
status VARCHAR(16) DEFAULT 'PENDING' COMMENT '索引状态(PENDING/PROCESSING/INDEXED/FAILED)',
|
||
chunk_count INT DEFAULT 0 COMMENT '分块数量',
|
||
error_message TEXT COMMENT '失败原因',
|
||
|
||
-- 时间字段
|
||
indexed_at DATETIME COMMENT '索引完成时间',
|
||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||
|
||
-- 索引
|
||
UNIQUE INDEX uk_file_hash (file_hash),
|
||
INDEX idx_doc_id (doc_id),
|
||
INDEX idx_fault_source (fault_source),
|
||
INDEX idx_status (status),
|
||
INDEX idx_created_at (created_at)
|
||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='文档元数据表(MVP版)';
|
||
```
|
||
|
||
#### 字段说明
|
||
|
||
| 字段 | 类型 | 必填 | 说明 |
|
||
|------|------|------|------|
|
||
| doc_id | VARCHAR(64) | 是 | **核心**:文档唯一ID,关联 Milvus 中的所有分块 |
|
||
| fault_category | VARCHAR(32) | 否 | 文档类别,与 diagnosis_record 一致 |
|
||
| fault_source | VARCHAR(128) | 否 | 文档归属(省份/服务名)|
|
||
| api_name | VARCHAR(128) | 否 | 接口名称 |
|
||
| version | VARCHAR(32) | 否 | 文档版本 |
|
||
| file_name | VARCHAR(256) | 是 | 原始文件名 |
|
||
| file_path | VARCHAR(512) | 否 | 文件存储路径 |
|
||
| file_hash | VARCHAR(64) | 否 | **去重关键**:文件MD5,唯一约束 |
|
||
| file_size | BIGINT | 否 | 文件大小 |
|
||
| status | VARCHAR(16) | 是 | **状态追踪**:PENDING/PROCESSING/INDEXED/FAILED |
|
||
| chunk_count | INT | 否 | 分块数量 |
|
||
| error_message | TEXT | 否 | 失败原因 |
|
||
| indexed_at | DATETIME | 否 | 索引完成时间 |
|
||
|
||
#### 核心设计决策
|
||
|
||
**1. doc_id:MySQL 与 Milvus 的桥梁**
|
||
|
||
```
|
||
作用:
|
||
- MySQL:通过 doc_id 管理文档元数据
|
||
- Milvus:每个 chunk 的 metadata 中携带 doc_id
|
||
|
||
关联关系:
|
||
api_document (MySQL)
|
||
doc_id: doc-001
|
||
↓ 1:N
|
||
Milvus chunks
|
||
chunk_1: {doc_id: 'doc-001', text: '...', vector: [...]}
|
||
chunk_2: {doc_id: 'doc-001', text: '...', vector: [...]}
|
||
|
||
管理操作:
|
||
- 删除文档:
|
||
DELETE FROM milvus_collection WHERE metadata["doc_id"] == 'doc-001';
|
||
DELETE FROM api_document WHERE doc_id = 'doc-001';
|
||
|
||
- 重新索引:
|
||
先删除旧数据,再重新导入
|
||
```
|
||
|
||
---
|
||
|
||
**2. file_hash:文件去重**
|
||
|
||
```
|
||
去重流程:
|
||
1. 用户上传文件
|
||
↓
|
||
2. 计算文件 MD5
|
||
file_hash = md5(file_content)
|
||
↓
|
||
3. 检查是否已存在
|
||
SELECT * FROM api_document WHERE file_hash = 'abc123...';
|
||
↓
|
||
4a. 如果存在 → 提示"文档已存在"
|
||
4b. 如果不存在 → 继续导入
|
||
|
||
唯一约束:
|
||
UNIQUE INDEX uk_file_hash (file_hash)
|
||
```
|
||
|
||
---
|
||
|
||
**3. status:状态追踪**
|
||
|
||
```
|
||
状态流转:
|
||
PENDING (待处理)
|
||
↓
|
||
PROCESSING (处理中)
|
||
↓ 成功
|
||
INDEXED (已索引)
|
||
|
||
↓ 失败
|
||
FAILED (失败)
|
||
|
||
用途:
|
||
- 批量导入时监控进度
|
||
- 失败重试
|
||
- 统计索引成功率
|
||
```
|
||
|
||
---
|
||
|
||
**4. 硬删除策略(MVP)**
|
||
|
||
```
|
||
删除文档时:
|
||
1. 删除 Milvus 中的所有分块
|
||
DELETE FROM milvus_collection WHERE metadata["doc_id"] == 'xxx';
|
||
|
||
2. 删除 MySQL 元数据
|
||
DELETE FROM api_document WHERE doc_id = 'xxx';
|
||
|
||
3. 可选:删除原始文件
|
||
Files.delete(file_path);
|
||
|
||
特点:
|
||
- 简单直接
|
||
- 数据彻底删除
|
||
- 不可恢复(需谨慎)
|
||
|
||
Phase 2 可增强:
|
||
- 软删除(archived_at)
|
||
- 启用开关(enabled)
|
||
```
|
||
|
||
---
|
||
|
||
#### 数据流
|
||
|
||
**场景1:导入新文档**
|
||
|
||
```
|
||
1. 用户上传文件
|
||
file: "广东社保查询v2.1.docx"
|
||
↓
|
||
2. 计算 hash
|
||
file_hash = md5(file)
|
||
↓
|
||
3. 检查去重(MySQL)
|
||
SELECT * FROM api_document WHERE file_hash = 'abc123';
|
||
→ 不存在
|
||
↓
|
||
4. 插入元数据(MySQL)
|
||
INSERT INTO api_document VALUES (
|
||
NULL, 'doc-001', 'EXTERNAL_API', '广东', '社保查询', 'v2.1',
|
||
'广东社保查询v2.1.docx', '/docs/guangdong/social-v2.1.docx',
|
||
'abc123...', 1048576,
|
||
'PROCESSING', 0, NULL, NULL, NOW(), NOW()
|
||
);
|
||
↓
|
||
5. 后台任务处理
|
||
- 解析 → Markdown
|
||
- 分块(15个chunk)
|
||
- 向量化
|
||
- 存入 Milvus(每个chunk的metadata中携带doc_id='doc-001')
|
||
↓
|
||
6. 更新状态(MySQL)
|
||
UPDATE api_document
|
||
SET status = 'INDEXED',
|
||
chunk_count = 15,
|
||
indexed_at = NOW()
|
||
WHERE doc_id = 'doc-001';
|
||
```
|
||
|
||
---
|
||
|
||
**场景2:删除文档**
|
||
|
||
```
|
||
1. 用户删除文档
|
||
doc_id = 'doc-001'
|
||
↓
|
||
2. 删除 Milvus 中的所有分块
|
||
DELETE FROM milvus_collection
|
||
WHERE metadata["doc_id"] == 'doc-001';
|
||
↓
|
||
3. 删除 MySQL 元数据
|
||
DELETE FROM api_document WHERE doc_id = 'doc-001';
|
||
↓
|
||
4. 可选:删除原始文件
|
||
rm /docs/guangdong/social-v2.1.docx
|
||
```
|
||
|
||
---
|
||
|
||
**场景3:重新索引文档**
|
||
|
||
```
|
||
1. 文档内容更新,需要重新索引
|
||
doc_id = 'doc-001'
|
||
↓
|
||
2. 删除旧数据
|
||
- Milvus: DELETE WHERE metadata["doc_id"] == 'doc-001'
|
||
- MySQL: DELETE FROM api_document WHERE doc_id = 'doc-001'
|
||
↓
|
||
3. 重新导入(同场景1)
|
||
```
|
||
|
||
---
|
||
|
||
#### 典型查询
|
||
|
||
**1. 查看文档列表**
|
||
```sql
|
||
-- 按省份查询
|
||
SELECT doc_id, file_name, version, status, chunk_count, indexed_at
|
||
FROM api_document
|
||
WHERE fault_source = '广东'
|
||
AND status = 'INDEXED'
|
||
ORDER BY indexed_at DESC;
|
||
|
||
-- 查询失败的文档
|
||
SELECT doc_id, file_name, error_message
|
||
FROM api_document
|
||
WHERE status = 'FAILED';
|
||
```
|
||
|
||
**2. 文档去重检查**
|
||
```sql
|
||
-- 导入前检查
|
||
SELECT doc_id, file_name
|
||
FROM api_document
|
||
WHERE file_hash = 'abc123...';
|
||
```
|
||
|
||
**3. 统计分析**
|
||
```sql
|
||
-- 统计各状态文档数量
|
||
SELECT status, COUNT(*) as count
|
||
FROM api_document
|
||
GROUP BY status;
|
||
|
||
-- 统计各省份文档数量
|
||
SELECT fault_source, COUNT(*) as count
|
||
FROM api_document
|
||
WHERE status = 'INDEXED'
|
||
GROUP BY fault_source
|
||
ORDER BY count DESC;
|
||
```
|
||
|
||
---
|
||
|
||
#### 与 Milvus 的协作
|
||
|
||
**Milvus Collection Schema**
|
||
|
||
```python
|
||
{
|
||
"collection_name": "api_doc_collection",
|
||
"fields": [
|
||
{"name": "id", "type": "VARCHAR", "max_length": 64, "is_primary": true},
|
||
{"name": "content", "type": "VARCHAR", "max_length": 2000},
|
||
{"name": "vector", "type": "FLOAT_VECTOR", "dim": 1536},
|
||
{"name": "metadata", "type": "JSON"}
|
||
]
|
||
}
|
||
|
||
# metadata 结构
|
||
{
|
||
"doc_id": "doc-001", # 关联 MySQL
|
||
"_source": "/path/to/file",
|
||
"_file_name": "xxx.docx",
|
||
"chunkIndex": 0,
|
||
"totalChunks": 15,
|
||
"fault_category": "EXTERNAL_API",
|
||
"fault_source": "广东",
|
||
"error_code": "40003"
|
||
}
|
||
```
|
||
|
||
**Java 代码示例**
|
||
|
||
```java
|
||
// 插入时携带 doc_id
|
||
Map<String, Object> metadata = new HashMap<>();
|
||
metadata.put("doc_id", docId); // 关联 MySQL
|
||
metadata.put("_source", filePath);
|
||
metadata.put("chunkIndex", chunkIndex);
|
||
metadata.put("fault_source", province);
|
||
|
||
// 删除文档的所有分块
|
||
String expr = String.format("metadata[\"doc_id\"] == \"%s\"", docId);
|
||
DeleteParam deleteParam = DeleteParam.newBuilder()
|
||
.withCollectionName(COLLECTION_NAME)
|
||
.withExpr(expr)
|
||
.build();
|
||
milvusClient.delete(deleteParam);
|
||
```
|
||
|
||
---
|
||
|
||
#### 数据示例
|
||
|
||
```sql
|
||
-- 外部接口文档
|
||
INSERT INTO api_document VALUES
|
||
(1, 'doc-001', 'EXTERNAL_API', '广东', '社保查询', 'v2.1',
|
||
'广东社保查询v2.1.docx', '/docs/guangdong/social-v2.1.docx',
|
||
'abc123...', 1048576,
|
||
'INDEXED', 15, NULL, '2024-06-15 10:30:00', NOW(), NOW());
|
||
|
||
-- 内部服务文档
|
||
INSERT INTO api_document VALUES
|
||
(2, 'doc-002', 'INTERNAL_ERROR', 'order-service', '订单服务API', 'v1.0',
|
||
'订单服务API文档.pdf', '/docs/internal/order-service-api.pdf',
|
||
'def456...', 2097152,
|
||
'INDEXED', 20, NULL, '2024-06-14 15:20:00', NOW(), NOW());
|
||
|
||
-- 处理失败的文档
|
||
INSERT INTO api_document VALUES
|
||
(3, 'doc-003', 'EXTERNAL_API', '江苏', '公积金查询', 'v1.5',
|
||
'江苏公积金查询.html', '/docs/jiangsu/fund-v1.5.html',
|
||
'ghi789...', 512000,
|
||
'FAILED', 0, '不支持HTML格式,请转换为Word或PDF', NULL, NOW(), NOW());
|
||
```
|
||
|
||
---
|
||
|
||
#### MVP 版本的简化
|
||
|
||
```
|
||
Phase 1(当前):
|
||
✅ 基础字段和表结构
|
||
✅ 文件去重(file_hash)
|
||
✅ 状态追踪(status)
|
||
✅ 硬删除(彻底删除)
|
||
✅ 通过 doc_id 关联 Milvus
|
||
|
||
Phase 2(未来增强):
|
||
❌ enabled(启用开关)
|
||
❌ archived_at(软删除)
|
||
❌ batch_id(批次管理)
|
||
❌ status 细化(PARSING/SPLITTING/INDEXING...)
|
||
❌ tags(标签分类)
|
||
❌ is_latest(版本标记)
|
||
```
|
||
|
||
---
|
||
|
||
## 三、会话管理设计
|
||
|
||
#### 表结构
|
||
|
||
```sql
|
||
CREATE TABLE conversation_history (
|
||
-- 主键
|
||
id BIGINT PRIMARY KEY AUTO_INCREMENT,
|
||
conversation_id VARCHAR(64) NOT NULL COMMENT '对话ID(UUID)',
|
||
|
||
-- 关联信息
|
||
session_id VARCHAR(64) NOT NULL COMMENT '会话ID',
|
||
diagnosis_id VARCHAR(64) COMMENT '关联诊断ID(追问时可为空)',
|
||
|
||
-- 对话内容
|
||
round_number INT NOT NULL COMMENT '对话轮次(1, 2, 3...)',
|
||
role VARCHAR(16) NOT NULL COMMENT '角色(user/assistant)',
|
||
content TEXT NOT NULL COMMENT '对话内容',
|
||
|
||
-- 调试字段(可选)
|
||
tool_calls JSON COMMENT '工具调用记录',
|
||
|
||
-- 元数据
|
||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
|
||
|
||
-- 索引
|
||
INDEX idx_session_id (session_id),
|
||
INDEX idx_diagnosis_id (diagnosis_id),
|
||
INDEX idx_created_at (created_at)
|
||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='对话历史表(可选,用于分析)';
|
||
```
|
||
|
||
#### 字段说明
|
||
|
||
| 字段 | 类型 | 必填 | 说明 |
|
||
|------|------|------|------|
|
||
| conversation_id | VARCHAR(64) | 是 | 对话唯一标识 |
|
||
| session_id | VARCHAR(64) | 是 | 会话ID,关联多轮对话 |
|
||
| diagnosis_id | VARCHAR(64) | 否 | 关联诊断ID,初次诊断时填写,追问时为空 |
|
||
| round_number | INT | 是 | 对话轮次,从1开始递增 |
|
||
| role | VARCHAR(16) | 是 | 角色:user(用户)/assistant(AI) |
|
||
| content | TEXT | 是 | 对话内容 |
|
||
|
||
#### 核心设计决策
|
||
|
||
**1. 用途定位**
|
||
```
|
||
主要用途:
|
||
- BadCase分析(用户追问什么?)
|
||
- 功能优化(哪些问题常被追问?)
|
||
- 审计追溯(完整对话记录)
|
||
|
||
不是:
|
||
- 主要业务表(诊断记录才是)
|
||
- 实时查询(对话上下文在Redis)
|
||
|
||
结论:辅助表,Phase 2 再加
|
||
```
|
||
|
||
**2. diagnosis_id 可为空**
|
||
```
|
||
场景1:初次诊断
|
||
- diagnosis_id: diag-001
|
||
- round 1: user → "诊断订单 A"
|
||
- round 2: assistant → "完整报告..."
|
||
|
||
场景2:追问(不创建新诊断)
|
||
- diagnosis_id: NULL
|
||
- round 3: user → "为什么会缺失字段?"
|
||
- round 4: assistant → "因为前端表单未校验..."
|
||
|
||
场景3:新诊断
|
||
- diagnosis_id: diag-002
|
||
- round 5: user → "诊断订单 B"
|
||
- round 6: assistant → "完整报告..."
|
||
```
|
||
|
||
#### 典型查询场景
|
||
|
||
**1. 查询会话的所有对话**
|
||
```sql
|
||
-- 按轮次排序
|
||
SELECT * FROM conversation_history
|
||
WHERE session_id = 'sess-abc'
|
||
ORDER BY round_number;
|
||
```
|
||
|
||
**2. 查询某次诊断的对话**
|
||
```sql
|
||
-- 包括诊断前后的追问
|
||
SELECT * FROM conversation_history
|
||
WHERE diagnosis_id = 'diag-001'
|
||
OR (session_id IN (
|
||
SELECT session_id FROM conversation_history WHERE diagnosis_id = 'diag-001'
|
||
))
|
||
ORDER BY round_number;
|
||
```
|
||
|
||
**3. 统计追问频率**
|
||
```sql
|
||
-- 统计有多少诊断被追问
|
||
SELECT
|
||
COUNT(DISTINCT diagnosis_id) as total_diagnosis,
|
||
COUNT(DISTINCT CASE WHEN round_number > 2 THEN diagnosis_id END) as with_followup,
|
||
ROUND(COUNT(DISTINCT CASE WHEN round_number > 2 THEN diagnosis_id END) * 100.0 / COUNT(DISTINCT diagnosis_id), 2) as followup_rate
|
||
FROM conversation_history
|
||
WHERE diagnosis_id IS NOT NULL;
|
||
```
|
||
|
||
---
|
||
|
||
## 三、会话管理设计
|
||
|
||
### 3.1 会话存储策略
|
||
|
||
#### Redis(主)
|
||
```
|
||
数据结构:
|
||
key: session:{session_id}
|
||
value: {
|
||
"sessionId": "sess-abc",
|
||
"userId": "user-123",
|
||
"currentDiagnosisId": "diag-001",
|
||
"messages": [
|
||
{"role": "user", "content": "诊断订单 A"},
|
||
{"role": "assistant", "content": "完整报告..."}
|
||
],
|
||
"context": {
|
||
"province": "广东",
|
||
"apiName": "社保查询",
|
||
"errorCode": "40003"
|
||
},
|
||
"createdAt": "2024-06-15T14:30:00Z",
|
||
"lastActiveAt": "2024-06-15T14:35:00Z"
|
||
}
|
||
ttl: 1800秒(30分钟)
|
||
|
||
优势:
|
||
- 快速读写
|
||
- 自动过期
|
||
- 支持追问(保存上下文)
|
||
```
|
||
|
||
#### MySQL(辅助,可选)
|
||
```
|
||
同步策略:
|
||
1. 重要会话同步
|
||
- 有用户反馈的会话
|
||
- 诊断失败的会话(BadCase)
|
||
- 多轮对话 > 3 轮的会话
|
||
|
||
2. 同步时机
|
||
- 会话结束时(30分钟过期)
|
||
- 用户反馈时(实时)
|
||
- 定时任务(每小时,可选)
|
||
|
||
3. 同步目标
|
||
- conversation_history 表
|
||
- 用于长期分析和审计
|
||
```
|
||
|
||
### 3.2 数据流设计
|
||
|
||
#### 场景1:单次诊断(主流 80%)
|
||
|
||
```
|
||
1. 用户发起诊断
|
||
POST /api/diagnosis/start
|
||
{
|
||
"orderId": "202406150001"
|
||
}
|
||
|
||
2. 创建会话(Redis)
|
||
key: session:sess-abc
|
||
ttl: 1800秒
|
||
|
||
3. 创建诊断记录(MySQL)
|
||
INSERT INTO diagnosis_record
|
||
- diagnosis_id: diag-001
|
||
- session_id: sess-abc
|
||
- status: RUNNING
|
||
|
||
4. Agent 执行诊断
|
||
- 调用工具(queryOrder, queryLogs, searchDoc...)
|
||
- 生成报告
|
||
|
||
5. 更新诊断记录(MySQL)
|
||
UPDATE diagnosis_record
|
||
- status: SUCCESS
|
||
- root_cause: "idCard字段缺失"
|
||
- report_markdown: "完整报告..."
|
||
|
||
6. 返回报告
|
||
→ 大部分用户到此结束
|
||
```
|
||
|
||
#### 场景2:追问(少数 20%)
|
||
|
||
```
|
||
1. 用户追问
|
||
POST /api/chat
|
||
{
|
||
"sessionId": "sess-abc",
|
||
"message": "为什么会缺失字段?"
|
||
}
|
||
|
||
2. 从 Redis 获取上下文
|
||
GET session:sess-abc
|
||
- 有之前的诊断结果
|
||
- 有对话历史
|
||
|
||
3. Agent 基于上下文回答
|
||
- 不创建新的 diagnosis_record
|
||
- 只是普通对话
|
||
|
||
4. 更新 Redis 会话
|
||
- 追加对话历史
|
||
- 刷新 TTL(重新计时30分钟)
|
||
|
||
5. 可选:保存到 conversation_history(MySQL)
|
||
- 如果需要长期分析
|
||
- 异步存储
|
||
```
|
||
|
||
#### 场景3:同一会话多次诊断
|
||
|
||
```
|
||
1. 用户第一次诊断
|
||
"诊断订单 A"
|
||
→ diagnosis_record(diag-001, session_id=sess-abc)
|
||
|
||
2. 用户第二次诊断
|
||
"再诊断订单 B"
|
||
→ diagnosis_record(diag-002, session_id=sess-abc)
|
||
|
||
3. 会话关联
|
||
- 同一个 session_id
|
||
- 两条 diagnosis_record
|
||
- Redis 中保存完整对话历史
|
||
```
|
||
|
||
---
|
||
|
||
## 四、实施规划
|
||
|
||
### 4.1 Phase 1:核心功能(第1周)
|
||
|
||
**实现内容**:
|
||
```
|
||
✅ diagnosis_record 表
|
||
✅ Redis 会话管理
|
||
✅ 单次诊断流程
|
||
|
||
不实现:
|
||
❌ conversation_history 表(先不加)
|
||
❌ 会话同步(先不做)
|
||
❌ 追问功能(先不支持)
|
||
```
|
||
|
||
**验收标准**:
|
||
```
|
||
- 用户输入订单号 → 返回诊断报告
|
||
- 诊断记录持久化到 MySQL
|
||
- 可以查询历史诊断
|
||
- 可以统计诊断成功率
|
||
```
|
||
|
||
### 4.2 Phase 2:追问功能(第2周)
|
||
|
||
**实现内容**:
|
||
```
|
||
✅ 支持多轮对话(基于 Redis 上下文)
|
||
✅ conversation_history 表(可选)
|
||
✅ 会话上下文管理
|
||
```
|
||
|
||
**验收标准**:
|
||
```
|
||
- 用户可以追问细节
|
||
- Agent 能基于上下文回答
|
||
- 追问不创建新的诊断记录
|
||
```
|
||
|
||
### 4.3 Phase 3:优化分析(第3周)
|
||
|
||
**实现内容**:
|
||
```
|
||
✅ 会话同步(Redis → MySQL)
|
||
✅ BadCase 分析
|
||
✅ 追问频率统计
|
||
```
|
||
|
||
**验收标准**:
|
||
```
|
||
- 重要会话自动同步到 MySQL
|
||
- 可以分析用户追问模式
|
||
- 可以优化 Prompt 和功能
|
||
```
|
||
|
||
---
|
||
|
||
## 五、关键设计决策总结
|
||
|
||
### 5.1 单次诊断 vs 多轮对话
|
||
|
||
**决策**:主要是单次诊断,辅助支持追问
|
||
|
||
**理由**:
|
||
- 系统定位是"自动化诊断",不是聊天机器人
|
||
- 大部分用户需求:输入 → 报告 → 结束
|
||
- 追问是少数场景,不应主导设计
|
||
|
||
**实现**:
|
||
- diagnosis_record 只记录诊断任务
|
||
- conversation_history 记录追问对话(可选)
|
||
|
||
### 5.2 会话存储:Redis vs MySQL
|
||
|
||
**决策**:Redis 主存储,MySQL 辅助备份
|
||
|
||
**理由**:
|
||
- 会话是临时数据,30分钟过期
|
||
- Redis 读写快,适合实时交互
|
||
- MySQL 用于长期分析,不是主路径
|
||
|
||
**实现**:
|
||
- Redis 存所有会话(自动过期)
|
||
- MySQL 只存重要会话(按需同步)
|
||
|
||
### 5.3 report_markdown 是否存储
|
||
|
||
**决策**:存储完整报告
|
||
|
||
**理由**:
|
||
- 报告是最终产物,需要固化
|
||
- Prompt 可能变化,历史报告不应变
|
||
- 查询历史时直接展示,不重新生成
|
||
|
||
**成本**:
|
||
- TEXT 字段较大
|
||
- 适度冗余可接受
|
||
|
||
### 5.4 conversation_history 是否必需
|
||
|
||
**决策**:Phase 2 再加,不是必需
|
||
|
||
**理由**:
|
||
- 核心功能不依赖对话历史
|
||
- 主要用于分析和优化
|
||
- 可以后期扩展
|
||
|
||
---
|
||
|
||
## 六、数据量预估
|
||
|
||
### 6.1 diagnosis_record
|
||
|
||
```
|
||
场景:中型企业运维团队
|
||
- 日均诊断:100 次
|
||
- 月均诊断:3000 次
|
||
- 年均诊断:36000 次
|
||
|
||
存储预估:
|
||
- 单条记录:约 5KB(含报告)
|
||
- 年存储量:36000 × 5KB = 180MB
|
||
- 三年存储:540MB
|
||
|
||
结论:数据量不大,可以全量保留
|
||
```
|
||
|
||
### 6.2 conversation_history
|
||
|
||
```
|
||
场景:20% 用户会追问
|
||
- 日均追问:20 次
|
||
- 平均追问轮次:3 轮
|
||
- 日均对话记录:20 × 3 × 2(user+assistant)= 120 条
|
||
|
||
存储预估:
|
||
- 单条记录:约 1KB
|
||
- 年存储量:120 × 365 × 1KB = 44MB
|
||
|
||
结论:数据量很小,可以全量保留
|
||
```
|
||
|
||
---
|
||
|
||
## 七、索引设计说明
|
||
|
||
### 7.1 diagnosis_record 索引
|
||
|
||
```sql
|
||
-- 业务查询索引
|
||
INDEX idx_business_id (business_id) -- 按业务标识查询(订单号/请求ID/线程ID)
|
||
INDEX idx_trace_id (trace_id) -- 按链路ID查询
|
||
INDEX idx_session_id (session_id) -- 按会话查询
|
||
|
||
-- 故障分类索引
|
||
INDEX idx_fault_category (fault_category) -- 按故障类别过滤
|
||
INDEX idx_fault_source_target (fault_source, fault_target(100)) -- 按故障源+目标统计
|
||
INDEX idx_error_code (error_code) -- 按错误码统计
|
||
|
||
-- 通用索引
|
||
INDEX idx_created_at (created_at) -- 时间范围查询
|
||
INDEX idx_status (status) -- 状态过滤
|
||
```
|
||
|
||
### 7.2 索引使用场景
|
||
|
||
```sql
|
||
-- 场景1:查询业务标识历史(使用 idx_business_id)
|
||
SELECT * FROM diagnosis_record
|
||
WHERE business_id = '202406150001';
|
||
|
||
-- 场景2:统计故障类别(使用 idx_fault_category)
|
||
SELECT fault_category, COUNT(*)
|
||
FROM diagnosis_record
|
||
WHERE fault_category = 'INTERNAL_ERROR'
|
||
GROUP BY fault_category;
|
||
|
||
-- 场景3:统计特定服务的异常(使用 idx_fault_source_target)
|
||
SELECT fault_target, COUNT(*)
|
||
FROM diagnosis_record
|
||
WHERE fault_source = 'order-service'
|
||
AND fault_category = 'INTERNAL_ERROR'
|
||
GROUP BY fault_target;
|
||
|
||
-- 场景4:时间范围统计(使用 idx_created_at)
|
||
SELECT DATE(created_at) as date, COUNT(*)
|
||
FROM diagnosis_record
|
||
WHERE created_at >= '2024-06-01'
|
||
GROUP BY DATE(created_at);
|
||
```
|
||
|
||
---
|
||
|
||
## 八、数据安全考虑
|
||
|
||
### 8.1 敏感数据处理
|
||
|
||
```
|
||
敏感字段:
|
||
- 请求报文中的手机号、身份证
|
||
- 响应报文中的个人信息
|
||
|
||
脱敏策略:
|
||
- 存储时脱敏(在 tool_calls JSON 中)
|
||
- 手机号:138****5678
|
||
- 身份证:110101********1234
|
||
|
||
实现:
|
||
- 报文查询工具自动脱敏
|
||
- 存入数据库前已脱敏
|
||
- 降低泄露风险
|
||
```
|
||
|
||
### 8.2 数据保留策略
|
||
|
||
```
|
||
diagnosis_record:
|
||
- 保留周期:3 年
|
||
- 清理策略:定时任务(每月)
|
||
- 归档:超过 3 年的数据导出后删除
|
||
|
||
conversation_history:
|
||
- 保留周期:1 年
|
||
- 清理策略:定时任务(每月)
|
||
- 可选:关联诊断被删除时级联删除
|
||
```
|
||
|
||
---
|
||
|
||
## 九、扩展性考虑
|
||
|
||
### 9.1 预留扩展字段
|
||
|
||
```sql
|
||
-- diagnosis_record 可选扩展字段
|
||
├─ tags VARCHAR(256) -- 标签(用于分类)
|
||
├─ severity VARCHAR(16) -- 严重级别(LOW/MEDIUM/HIGH/CRITICAL)
|
||
├─ affected_users INT -- 影响用户数
|
||
├─ resolved_at DATETIME -- 解决时间
|
||
└─ resolver VARCHAR(64) -- 解决人
|
||
|
||
-- 添加方式(不影响现有功能)
|
||
ALTER TABLE diagnosis_record ADD COLUMN tags VARCHAR(256);
|
||
```
|
||
|
||
### 9.2 分表策略(未来)
|
||
|
||
```
|
||
场景:数据量达到千万级别
|
||
|
||
方案1:按时间分表
|
||
- diagnosis_record_2024_06
|
||
- diagnosis_record_2024_07
|
||
- ...
|
||
|
||
方案2:按省份分表
|
||
- diagnosis_record_guangdong
|
||
- diagnosis_record_jiangsu
|
||
- ...
|
||
|
||
当前:不分表,单表够用(年均 36000 条)
|
||
```
|
||
|
||
---
|
||
|
||
## 十、变更日志
|
||
|
||
| 版本 | 日期 | 变更内容 | 变更人 |
|
||
|------|------|---------|--------|
|
||
| v1.0 | 2024-06-15 | 初版,定义核心表结构 | - |
|
||
| v1.1 | 2024-06-15 | 添加会话管理设计 | - |
|
||
| v1.2 | 2024-06-15 | 补充实施规划和数据流 | - |
|
||
| v2.0 | 2024-06-15 | **重大更新**:字段泛化,支持多种故障类型(外部接口+内部错误) | - |
|
||
|
||
### v2.0 主要变更
|
||
|
||
**字段泛化**:
|
||
- `order_id` → `business_id`(订单号→业务标识)
|
||
- `province` → `fault_source`(省份→故障源)
|
||
- `api_name` → 移除(信息合并到fault_target)
|
||
- `api_url` → `fault_target`(接口URL→故障目标)
|
||
- `error_code`:扩展支持(业务错误码→通用错误码)
|
||
|
||
**新增字段**:
|
||
- `fault_category`:故障类别(EXTERNAL_API/INTERNAL_ERROR/DATABASE/CACHE/NETWORK/THREAD/MEMORY/CONFIG)
|
||
- `error_message`:错误消息(通用描述)
|
||
- `stack_trace`:堆栈信息(内部错误专用)
|
||
|
||
**设计理念**:
|
||
- 从"只支持外部接口故障"扩展到"支持所有故障类型"
|
||
- 字段语义更通用,根据故障类型灵活填写
|
||
- 保持向下兼容,可通过数据迁移支持旧数据
|
||
|
||
**影响范围**:
|
||
- SQL建表语句
|
||
- 索引设计
|
||
- 查询示例
|
||
- 数据示例
|
||
|
||
---
|
||
|
||
**文档维护说明**:
|
||
- 本文档随系统演进持续更新
|
||
- 任何表结构变更需同步更新此文档
|
||
- 重大设计调整需记录决策理由
|