refactor(knowledge): KnowledgeIndexService 从数据库加载索引
## 改动内容
### 修改前:从文件系统扫描
```java
@Value("${knowledge.base-path}")
private String knowledgeBasePath;
@Autowired
private FrontmatterParser frontmatterParser;
@PostConstruct
public void loadIndex() {
// 1. 扫描 knowledge_base 目录
// 2. 读取每个 .md 文件
// 3. 解析 frontmatter
// 4. 构建内存索引
}
```
**问题**:
- 依赖文件系统,无法利用数据库已有数据
- 启动时需要重新扫描和解析所有文件
- 文件和数据库可能不一致
---
### 修改后:从数据库加载
```java
@Autowired
private ApiDocumentRepository apiDocumentRepository;
@PostConstruct
public void loadIndex() {
// 1. 从数据库读取所有文档
List<ApiDocument> documents = apiDocumentRepository.findAll();
// 2. 解析 metadata JSON
// 3. 构建内存索引
}
```
**优势**:
- ✅ 数据源统一:数据库是唯一真实数据源
- ✅ 启动更快:无需重新扫描文件和解析 frontmatter
- ✅ 数据一致:L0 索引与数据库完全同步
- ✅ 支持动态更新:初始化接口更新数据库后,L0 索引也会更新
---
## 核心方法
### 1. parseDocumentToEntry
从 `ApiDocument` 转换为 `KnowledgeEntry`:
```java
private KnowledgeEntry parseDocumentToEntry(ApiDocument doc) {
String metadata = doc.getMetadata();
String title = extractJsonValue(metadata, "title");
String summary = extractJsonValue(metadata, "summary");
String category = extractJsonValue(metadata, "category");
List<String> keywords = extractJsonArray(metadata, "keywords");
return KnowledgeEntry.builder()
.filePath(doc.getFilePath())
.title(title)
.keywords(keywords)
.summary(summary)
.category(category)
.build();
}
```
### 2. 简单的 JSON 解析
```java
private String extractJsonValue(String json, String key) {
// 提取 "key":"value" 格式
}
private List<String> extractJsonArray(String json, String key) {
// 提取 "key":["v1","v2"] 格式
}
```
**注意**:使用简单的字符串解析,避免引入 JSON 库依赖
---
## 数据流
```
应用启动
↓
KnowledgeIndexService.loadIndex()
↓
apiDocumentRepository.findAll()
↓
读取所有 api_document 记录
↓
解析每条记录的 metadata JSON
↓
构建 KnowledgeEntry
↓
加入内存索引(CopyOnWriteArrayList)
↓
L0 索引就绪
```
---
## 启动日志
```
[INFO] 开始从数据库加载知识库索引
[INFO] 知识库索引加载完成,共 6 个文档
```
---
## 与初始化接口的配合
### 流程 1:首次启动(数据库为空)
```
1. 应用启动
2. KnowledgeIndexService.loadIndex() → 0 个文档
3. 调用 POST /api/knowledge/init
4. 写入数据库 + 调用 knowledgeIndexService.addToIndex()
5. L0 索引更新为 6 个文档
```
### 流程 2:重启应用(数据库有数据)
```
1. 应用启动
2. KnowledgeIndexService.loadIndex() → 从数据库加载 6 个文档
3. L0 索引已就绪,无需再调用初始化接口
```
---
## 兼容性
### metadata JSON 示例
```json
{
"title": "支付网关错误码定义",
"summary": "记录了支付网关所有核心错误码的含义及排查方向",
"category": "api",
"keywords": ["ERR_TIMEOUT","超时","支付网关"]
}
```
### 字段映射
| metadata | KnowledgeEntry | 说明 |
|----------|---------------|------|
| title | title | 文档标题 |
| summary | summary | 文档摘要 |
| category | category | 文档分类 |
| keywords | keywords | 关键词列表 |
---
## 删除的代码
- ❌ `@Value("${knowledge.base-path}")`:不再需要文件路径配置
- ❌ `FrontmatterParser` 依赖:不再扫描文件
- ❌ `indexFile()` 方法:不再读取文件
- ❌ `extractCategoryFromPath()` 方法:从 metadata 获取
---
## 验证
```bash
# 1. 清空数据库
# DELETE FROM api_document;
# 2. 启动应用
mvn spring-boot:run
# 3. 查看日志
# [INFO] 知识库索引加载完成,共 0 个文档
# 4. 初始化
curl -X POST http://localhost:9900/api/knowledge/init
# 5. 重启应用
# [INFO] 知识库索引加载完成,共 6 个文档
```
This commit is contained in:
@@ -29,11 +29,8 @@ import java.util.stream.Stream;
|
|||||||
@Service
|
@Service
|
||||||
public class KnowledgeIndexService {
|
public class KnowledgeIndexService {
|
||||||
|
|
||||||
@Value("${knowledge.base-path}")
|
|
||||||
private String knowledgeBasePath;
|
|
||||||
|
|
||||||
@Autowired
|
@Autowired
|
||||||
private FrontmatterParser frontmatterParser;
|
private ApiDocumentRepository apiDocumentRepository;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 内存索引(线程安全)
|
* 内存索引(线程安全)
|
||||||
@@ -41,88 +38,108 @@ public class KnowledgeIndexService {
|
|||||||
private final List<KnowledgeEntry> knowledgeIndex = new CopyOnWriteArrayList<>();
|
private final List<KnowledgeEntry> knowledgeIndex = new CopyOnWriteArrayList<>();
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 启动时扫描知识库目录,构建索引
|
* 启动时从数据库加载索引
|
||||||
*/
|
*/
|
||||||
@PostConstruct
|
@PostConstruct
|
||||||
public void loadIndex() {
|
public void loadIndex() {
|
||||||
log.info("开始扫描知识库目录: {}", knowledgeBasePath);
|
log.info("开始从数据库加载知识库索引");
|
||||||
|
|
||||||
try {
|
try {
|
||||||
Path basePath = Paths.get(knowledgeBasePath);
|
// 从数据库读取所有已索引的文档
|
||||||
|
List<ApiDocument> documents = apiDocumentRepository.findAll();
|
||||||
|
|
||||||
// 目录不存在时自动创建
|
int loaded = 0;
|
||||||
if (!Files.exists(basePath)) {
|
for (ApiDocument doc : documents) {
|
||||||
Files.createDirectories(basePath);
|
try {
|
||||||
log.info("知识库目录已创建: {}", basePath.toAbsolutePath());
|
// 从 metadata JSON 中提取信息
|
||||||
|
KnowledgeEntry entry = parseDocumentToEntry(doc);
|
||||||
|
if (entry != null) {
|
||||||
|
knowledgeIndex.add(entry);
|
||||||
|
loaded++;
|
||||||
|
}
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.warn("解析文档失败: docId={}, error={}", doc.getDocId(), e.getMessage());
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// 递归扫描 .md 文件
|
log.info("知识库索引加载完成,共 {} 个文档", loaded);
|
||||||
try (Stream<Path> paths = Files.walk(basePath)) {
|
|
||||||
paths.filter(p -> p.toString().endsWith(".md"))
|
|
||||||
.forEach(this::indexFile);
|
|
||||||
}
|
|
||||||
|
|
||||||
log.info("知识库索引加载完成,共 {} 个文档", knowledgeIndex.size());
|
} catch (Exception e) {
|
||||||
|
|
||||||
} catch (IOException e) {
|
|
||||||
log.error("知识库索引加载失败", e);
|
log.error("知识库索引加载失败", e);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 索引单个文件
|
* 将 ApiDocument 转换为 KnowledgeEntry
|
||||||
*
|
|
||||||
* @param filePath 文件路径
|
|
||||||
*/
|
*/
|
||||||
private void indexFile(Path filePath) {
|
private KnowledgeEntry parseDocumentToEntry(ApiDocument doc) {
|
||||||
|
if (doc.getMetadata() == null || doc.getMetadata().isEmpty()) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
// 读取文件内容
|
// 简单的 JSON 解析
|
||||||
String content = Files.readString(filePath);
|
String metadata = doc.getMetadata();
|
||||||
|
|
||||||
// 解析 frontmatter
|
String title = extractJsonValue(metadata, "title");
|
||||||
Frontmatter frontmatter = frontmatterParser.parse(content);
|
String summary = extractJsonValue(metadata, "summary");
|
||||||
if (frontmatter == null) {
|
String category = extractJsonValue(metadata, "category");
|
||||||
log.debug("跳过文件(无有效 frontmatter): {}", filePath);
|
List<String> keywords = extractJsonArray(metadata, "keywords");
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 提取 category(从路径中获取)
|
return KnowledgeEntry.builder()
|
||||||
String category = extractCategoryFromPath(filePath.toString());
|
.filePath(doc.getFilePath())
|
||||||
|
.title(title != null ? title : doc.getApiName())
|
||||||
// 构建索引条目
|
.keywords(keywords)
|
||||||
KnowledgeEntry entry = KnowledgeEntry.builder()
|
.summary(summary)
|
||||||
.filePath(filePath.toString())
|
|
||||||
.title(frontmatter.getTitle())
|
|
||||||
.keywords(frontmatter.getKeywords())
|
|
||||||
.summary(frontmatter.getSummary())
|
|
||||||
.category(category)
|
.category(category)
|
||||||
.sections(frontmatter.getSections())
|
|
||||||
.build();
|
.build();
|
||||||
|
|
||||||
knowledgeIndex.add(entry);
|
} catch (Exception e) {
|
||||||
log.debug("文档已加入索引: title={}, filePath={}", entry.getTitle(), filePath);
|
log.warn("解析 metadata 失败: {}", doc.getDocId(), e);
|
||||||
|
return null;
|
||||||
} catch (IOException e) {
|
|
||||||
log.warn("读取文件失败: {}", filePath, e);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 从文件路径中提取 category
|
* 从 JSON 字符串中提取值
|
||||||
* 例如:knowledge_base/api/test.md -> api
|
|
||||||
*/
|
*/
|
||||||
private String extractCategoryFromPath(String filePath) {
|
private String extractJsonValue(String json, String key) {
|
||||||
String normalized = filePath.replace("\\", "/");
|
String pattern = "\"" + key + "\":\"";
|
||||||
String[] parts = normalized.split("/");
|
int startIndex = json.indexOf(pattern);
|
||||||
|
if (startIndex == -1) {
|
||||||
// 查找 knowledge_base 后的第一个目录
|
return null;
|
||||||
for (int i = 0; i < parts.length - 1; i++) {
|
|
||||||
if (parts[i].equals("knowledge_base") && i + 1 < parts.length) {
|
|
||||||
return parts[i + 1];
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return "default";
|
startIndex += pattern.length();
|
||||||
|
int endIndex = json.indexOf("\"", startIndex);
|
||||||
|
if (endIndex == -1) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
return json.substring(startIndex, endIndex);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 从 JSON 字符串中提取数组
|
||||||
|
*/
|
||||||
|
private List<String> extractJsonArray(String json, String key) {
|
||||||
|
String pattern = "\"" + key + "\":[";
|
||||||
|
int startIndex = json.indexOf(pattern);
|
||||||
|
if (startIndex == -1) {
|
||||||
|
return Collections.emptyList();
|
||||||
|
}
|
||||||
|
|
||||||
|
startIndex += pattern.length();
|
||||||
|
int endIndex = json.indexOf("]", startIndex);
|
||||||
|
if (endIndex == -1) {
|
||||||
|
return Collections.emptyList();
|
||||||
|
}
|
||||||
|
|
||||||
|
String arrayContent = json.substring(startIndex, endIndex);
|
||||||
|
return Arrays.stream(arrayContent.split(","))
|
||||||
|
.map(s -> s.trim().replaceAll("^\"|\"$", ""))
|
||||||
|
.filter(s -> !s.isEmpty())
|
||||||
|
.collect(Collectors.toList());
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
Reference in New Issue
Block a user