osiman/SuperBizAgent-java
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: reorganize MVP interview documentation

Browse Source
This commit is contained in:
aruo
2026-07-05 15:29:28 +08:00
parent b22f2d22c8
commit 88e0a6c944
51 changed files with 4352 additions and 1318 deletions
Download Patch File Download Diff File Expand all files Collapse all files
interview/rag-retrieval-quality-report.md
+63 -126
View File
@@ -1,71 +1,69 @@
# RAG Retrieval Quality Report
# RAG 检索质量报告
## Purpose
## 1. 目的
This report compares the live retrieval behavior of the original Milvus SDK path and the new Spring AI VectorStore path.
这份报告回答一个面试关键问题：
The goal is to answer an interview-critical question:
```text
迁移到 Spring AI VectorStore 后，怎么证明检索质量没有退化？
```
> After moving retrieval to Spring AI VectorStore, how do we know retrieval quality did not regress?
这不是完整 benchmark，而是针对当前 Milvus/Zilliz collection 的代表性 live smoke comparison。
This is not a full benchmark yet. It is a focused live smoke comparison using representative RAG queries against the current Milvus/Zilliz collection.
## 2. 验证设置
## Setup
Service endpoint:
服务端点：
```text
GET http://127.0.0.1:9900/api/search/similar
```
Collection:
collection：
```text
biz
```
Compared modes:
对比模式：
```text
retrieval.vector-store.mode=sdk
retrieval.vector-store.mode=spring-ai
```
Each case used:
每个 case：
```text
topK=3
```
The application was restarted once per mode using command-line configuration so no repository config file had to be changed.
## 3. 测试案例
## Cases
| Case | Query | 目的 |
|---|---|---|
| `err-timeout` | `ERR_TIMEOUT` | 精确错误码检索 |
| `payment-service-timeout` | `payment-service timeout` | 服务超时排障 |
| `mysql-connection-pool` | `MySQL connection pool is exhausted. How should I diagnose it?` | 数据库排障 |
| `high-cpu-payment` | `HighCPUUsage payment-service` | AIOps 告警式检索 |
| `rag-l0-l1` | `Should L0 keyword matching decide the final retrieval result?` | 抽象 RAG 设计问题 |
| `database-filter` | `mysql timeout`, category=`database` | metadata filter 行为 |
| Case | Query | Purpose |
| --- | --- | --- |
| `err-timeout` | `ERR_TIMEOUT` | Exact error-code retrieval |
| `payment-service-timeout` | `payment-service timeout` | Service timeout troubleshooting |
| `mysql-connection-pool` | `MySQL connection pool is exhausted. How should I diagnose it?` | Database troubleshooting |
| `high-cpu-payment` | `HighCPUUsage payment-service` | AIOps alert-style retrieval |
| `rag-l0-l1` | `Should L0 keyword matching decide the final retrieval result?` | Abstract RAG design query |
| `database-filter` | `mysql timeout`, category=`database` | Metadata filter behavior |
## 4. 对比摘要
## Summary
| Case | SDK 数量 | VectorStore 数量 | Top1 一致 | TopK 重叠 | 结论 |
|---|---:|---:|---|---:|---|
| `err-timeout` | 3 | 3 | 是 | 3/3 | 文档和顺序一致 |
| `payment-service-timeout` | 3 | 3 | 是 | 3/3 | 文档和顺序一致 |
| `mysql-connection-pool` | 3 | 3 | 是 | 3/3 | 文档和顺序一致 |
| `high-cpu-payment` | 3 | 3 | 是 | 3/3 | AIOps 核心 query 一致 |
| `rag-l0-l1` | 3 | 1 | 是 | 1/3 | VectorStore 尾部结果更少 |
| `database-filter` | 0 | 0 | 不适用 | 不适用 | filter 行为一致，taxonomy 有问题 |
| Case | SDK Count | VectorStore Count | Top1 Same | TopK Overlap | Notes |
| --- | ---: | ---: | --- | ---: | --- |
| `err-timeout` | 3 | 3 | Yes | 3/3 | Same ordering and same documents |
| `payment-service-timeout` | 3 | 3 | Yes | 3/3 | Same ordering and same documents |
| `mysql-connection-pool` | 3 | 3 | Yes | 3/3 | Same ordering and same documents |
| `high-cpu-payment` | 3 | 3 | Yes | 3/3 | Same ordering and same documents |
| `rag-l0-l1` | 3 | 1 | Yes | 1/3 | VectorStore returned only the strongest candidate |
| `database-filter` | 0 | 0 | N/A | N/A | Both paths applied the filter consistently; no live docs matched `category=database` |
## 5. 代表性结果
## Representative Results
### ERR_TIMEOUT
### `ERR_TIMEOUT`
SDK:
SDK：
```text
1. ERR_TIMEOUT score=0.5659486 label=l2_distance
@@ -73,7 +71,7 @@ SDK:
3. Error handling score=0.7735061 label=l2_distance
```
VectorStore:
VectorStore：
```text
1. ERR_TIMEOUT score=0.5659486 rawScore=0.4340513 label=similarity
@@ -81,15 +79,15 @@ VectorStore:
3. Error handling score=0.7735061 rawScore=0.2264938 label=similarity
```
Interpretation:
解释：
- Document ordering is identical.
- Compatibility `score` is identical to SDK L2 distance.
- VectorStore `rawScore` exposes Spring AI similarity separately.
- 排序一致。
- 兼容 `score` 与 SDK L2 distance 一致。
- `rawScore` 暴露 Spring AI similarity。
### `MySQL connection pool`
### MySQL connection pool
Both paths returned:
两条路径都返回：
```text
1. MySQL connection pool config
@@ -97,58 +95,23 @@ Both paths returned:
3. idle-timeout
```
Interpretation:
说明迁移保留了核心基础设施排障检索能力。
- The migration preserves a precise infrastructure troubleshooting retrieval case.
- Metadata fields such as title, category, and source remain available.
### HighCPUUsage payment-service
### `HighCPUUsage payment-service`
两条路径都返回 payment-service 高 CPU 相关排障文档，说明 AIOps 告警式 query 没有退化。
Both paths returned:
### rag-l0-l1
```text
1. 3. HighCPUUsage / payment-service troubleshooting steps
2. evidence mapping table row for HighCPUUsage/payment-service
3. 3.1 Symptom confirmation
```
VectorStore 只返回一个候选，但 Top1 与 SDK 一致。这说明抽象设计类 query 需要后续 query rewrite、补充索引或 threshold 调整。
Interpretation:
### database-filter
- AIOps-style alert terms still retrieve the expected troubleshooting document.
- This is important because AIOps diagnosis depends on knowledge retrieval plus metrics/log evidence.
两条路径都返回 0，因为相关 MySQL 文档当前分类是 `infrastructure`，不是 `database`。这是 metadata taxonomy 问题，不是 VectorStore 回归。
### `rag-l0-l1`
## 6. 分数兼容结论
SDK returned three results, while VectorStore returned one:
```text
Top1: Return error information
```
Interpretation:
- Top1 did not regress.
- VectorStore appears stricter for low-similarity tail results because the Spring AI path uses `similarityThresholdAll()`.
- This is acceptable for current read-path migration, but it is worth tracking because abstract design questions may need query rewriting, better indexed docs, or adjusted threshold behavior.
### `database-filter`
Both paths returned zero results for:
```text
query=mysql timeout
category=database
```
Interpretation:
- The filter path is consistent.
- The live indexed MySQL docs are categorized as `infrastructure`, not `database`.
- This highlights a metadata taxonomy issue rather than a VectorStore migration regression.
## Score Compatibility
The comparison validates the score design:
对比验证了当前分数设计：
```text
SDK:
@@ -162,50 +125,24 @@ VectorStore:
scoreLabel = similarity
```
This keeps `lookup_knowledge` relevance normalization stable while still exposing the VectorStore score semantics for trace/debugging.
这样既保持 `lookup_knowledge` 原有归一化逻辑，又能暴露 VectorStore 语义。
## Findings
## 7. 验收结论
### Finding 1: Main live cases are equivalent
Spring AI VectorStore 读路径可以接受用于当前 MVP/面试：
For exact error code, service timeout, MySQL troubleshooting, and AIOps alert-style retrieval, SDK and VectorStore returned identical top3 documents in identical order.
- 核心排障和 AIOps case 与 SDK top3 一致。
- 分数兼容性保留。
- VectorStore 语义通过 `rawScore` 和 `scoreLabel` 可观察。
- SDK fallback 仍保留运行安全。
This is strong evidence that the read-path migration did not regress the most important demo and troubleshooting cases.
后续检索质量工作不阻塞这次迁移，应作为独立优化继续推进。
### Finding 2: Abstract RAG design queries need better retrieval support
## 8. 下一步
The `rag-l0-l1` query only returned one VectorStore candidate. The top result matched SDK top1, but the tail differed.
- 增加自动 live comparison 脚本。
- 在 offline evaluator 中加入 topK overlap、top1 hit、MRR。
- 规范 metadata category，例如 `database` 与 `infrastructure`。
- 为抽象设计类 query 增加 query rewriting。
- 后续再评估是否迁移写入路径到 `VectorStore.add(...)`。
This suggests the next quality work should focus on:
- Query transformation for abstract design questions.
- Better indexing of interview/devflow RAG design docs.
- Context expansion around same-section chunks.
- Possibly tuning VectorStore threshold behavior.
### Finding 3: Metadata taxonomy matters
The category filter case returned zero results in both modes because the relevant MySQL docs are categorized as `infrastructure`, not `database`.
This supports a previous RAG issue: category/domain metadata should be normalized before it is used as a hard filter.
## Acceptance Decision
The Spring AI VectorStore read path is accepted for current MVP/interview use:
- Core troubleshooting cases match SDK behavior.
- Score compatibility is preserved.
- The VectorStore path exposes better score semantics without changing the `lookup_knowledge` API.
- SDK fallback remains available for runtime safety.
The next retrieval-quality improvements should not block this migration. They should be handled as separate RAG quality work.
## Next Work
Recommended next steps:
- Add a small automated live comparison script if repeated validation becomes common.
- Add topK overlap and top1 hit metrics to the offline evaluator.
- Normalize metadata categories such as `database` vs `infrastructure`.
- Add query rewriting for abstract RAG questions.
- Decide later whether to migrate indexing writes to Spring AI `VectorStore.add(...)`.