feat: add rag post-reindex acceptance
This commit is contained in:
@@ -0,0 +1,66 @@
|
||||
# RAG Breadcrumb Embedding Acceptance
|
||||
|
||||
## What Changed
|
||||
|
||||
The indexing path now builds embedding text from chunk structure plus content:
|
||||
|
||||
```text
|
||||
Title: {title}
|
||||
Path: {breadcrumb}
|
||||
Content:
|
||||
{content}
|
||||
```
|
||||
|
||||
The stored Milvus `content` field remains the original chunk content. This keeps display and evidence output clean while allowing the vector to carry section-level semantics.
|
||||
|
||||
## Why Reindex Is Required
|
||||
|
||||
Embeddings are materialized at index time. Existing vectors were generated from the previous content-only text, so they cannot benefit from `title` and `breadcrumb` until the knowledge base is reindexed.
|
||||
|
||||
This is the key acceptance point:
|
||||
|
||||
```text
|
||||
code change alone != live retrieval changed
|
||||
code change + reindex + live query report = accepted behavior
|
||||
```
|
||||
|
||||
## How To Validate
|
||||
|
||||
1. Start the Spring Boot application.
|
||||
2. Reindex the knowledge base through the existing indexing path.
|
||||
3. Run:
|
||||
|
||||
```bash
|
||||
python scripts/eval_rag_live_acceptance.py
|
||||
```
|
||||
|
||||
The script writes:
|
||||
|
||||
```text
|
||||
eval/rag-retrieval/reports/live-post-reindex.json
|
||||
eval/rag-retrieval/reports/live-post-reindex.md
|
||||
```
|
||||
|
||||
The default cases cover:
|
||||
|
||||
- RAG chunk context questions where breadcrumb matters.
|
||||
- Diagnosis flow questions where section path matters.
|
||||
- `ERR_TIMEOUT` exact error-code retrieval.
|
||||
- MySQL connection pool troubleshooting.
|
||||
- AIOps payment-service latency alert retrieval.
|
||||
|
||||
## What To Look For
|
||||
|
||||
For breadcrumb-sensitive cases, inspect whether top candidates expose expected `title` and `breadcrumb` values in the report.
|
||||
|
||||
For core troubleshooting cases, check that result counts and top candidates remain stable. The goal is not to prove a full benchmark; it is to prove that reindexing did not obviously break important demo retrieval paths.
|
||||
|
||||
## Interview Answer
|
||||
|
||||
If asked how I verified the breadcrumb embedding change:
|
||||
|
||||
> I separated deterministic regression from live acceptance. The offline fixture baseline still runs without services. But because embedding changes only affect newly indexed vectors, I added a live post-reindex acceptance script. It calls the real `/api/search/similar` endpoint against representative breadcrumb-sensitive, troubleshooting, and AIOps queries, then writes JSON and Markdown reports. This lets me prove both that the code changed and that the live vector collection was refreshed.
|
||||
|
||||
If asked why the script does not reindex automatically:
|
||||
|
||||
> Reindexing mutates the vector store and depends on environment-specific data. I kept mutation explicit and made the script validation-only. That makes failures easier to diagnose: if retrieval does not improve, I can distinguish code changes, reindex state, and runtime retrieval behavior.
|
||||
Reference in New Issue
Block a user