feat: add rag post-reindex acceptance

This commit is contained in:
aruo
2026-07-05 12:29:24 +08:00
parent c7e2fc2ee2
commit 674dd27a48
8 changed files with 528 additions and 0 deletions
@@ -0,0 +1,66 @@
# RAG Breadcrumb Embedding Acceptance
## What Changed
The indexing path now builds embedding text from chunk structure plus content:
```text
Title: {title}
Path: {breadcrumb}
Content:
{content}
```
The stored Milvus `content` field remains the original chunk content. This keeps display and evidence output clean while allowing the vector to carry section-level semantics.
## Why Reindex Is Required
Embeddings are materialized at index time. Existing vectors were generated from the previous content-only text, so they cannot benefit from `title` and `breadcrumb` until the knowledge base is reindexed.
This is the key acceptance point:
```text
code change alone != live retrieval changed
code change + reindex + live query report = accepted behavior
```
## How To Validate
1. Start the Spring Boot application.
2. Reindex the knowledge base through the existing indexing path.
3. Run:
```bash
python scripts/eval_rag_live_acceptance.py
```
The script writes:
```text
eval/rag-retrieval/reports/live-post-reindex.json
eval/rag-retrieval/reports/live-post-reindex.md
```
The default cases cover:
- RAG chunk context questions where breadcrumb matters.
- Diagnosis flow questions where section path matters.
- `ERR_TIMEOUT` exact error-code retrieval.
- MySQL connection pool troubleshooting.
- AIOps payment-service latency alert retrieval.
## What To Look For
For breadcrumb-sensitive cases, inspect whether top candidates expose expected `title` and `breadcrumb` values in the report.
For core troubleshooting cases, check that result counts and top candidates remain stable. The goal is not to prove a full benchmark; it is to prove that reindexing did not obviously break important demo retrieval paths.
## Interview Answer
If asked how I verified the breadcrumb embedding change:
> I separated deterministic regression from live acceptance. The offline fixture baseline still runs without services. But because embedding changes only affect newly indexed vectors, I added a live post-reindex acceptance script. It calls the real `/api/search/similar` endpoint against representative breadcrumb-sensitive, troubleshooting, and AIOps queries, then writes JSON and Markdown reports. This lets me prove both that the code changed and that the live vector collection was refreshed.
If asked why the script does not reindex automatically:
> Reindexing mutates the vector store and depends on environment-specific data. I kept mutation explicit and made the script validation-only. That makes failures easier to diagnose: if retrieval does not improve, I can distinguish code changes, reindex state, and runtime retrieval behavior.