67 lines
2.6 KiB
Markdown
67 lines
2.6 KiB
Markdown
# RAG Breadcrumb Embedding Acceptance
|
|
|
|
## What Changed
|
|
|
|
The indexing path now builds embedding text from chunk structure plus content:
|
|
|
|
```text
|
|
Title: {title}
|
|
Path: {breadcrumb}
|
|
Content:
|
|
{content}
|
|
```
|
|
|
|
The stored Milvus `content` field remains the original chunk content. This keeps display and evidence output clean while allowing the vector to carry section-level semantics.
|
|
|
|
## Why Reindex Is Required
|
|
|
|
Embeddings are materialized at index time. Existing vectors were generated from the previous content-only text, so they cannot benefit from `title` and `breadcrumb` until the knowledge base is reindexed.
|
|
|
|
This is the key acceptance point:
|
|
|
|
```text
|
|
code change alone != live retrieval changed
|
|
code change + reindex + live query report = accepted behavior
|
|
```
|
|
|
|
## How To Validate
|
|
|
|
1. Start the Spring Boot application.
|
|
2. Reindex the knowledge base through the existing indexing path.
|
|
3. Run:
|
|
|
|
```bash
|
|
python scripts/eval_rag_live_acceptance.py
|
|
```
|
|
|
|
The script writes:
|
|
|
|
```text
|
|
eval/rag-retrieval/reports/live-post-reindex.json
|
|
eval/rag-retrieval/reports/live-post-reindex.md
|
|
```
|
|
|
|
The default cases cover:
|
|
|
|
- RAG chunk context questions where breadcrumb matters.
|
|
- Diagnosis flow questions where section path matters.
|
|
- `ERR_TIMEOUT` exact error-code retrieval.
|
|
- MySQL connection pool troubleshooting.
|
|
- AIOps payment-service latency alert retrieval.
|
|
|
|
## What To Look For
|
|
|
|
For breadcrumb-sensitive cases, inspect whether top candidates expose expected `title` and `breadcrumb` values in the report.
|
|
|
|
For core troubleshooting cases, check that result counts and top candidates remain stable. The goal is not to prove a full benchmark; it is to prove that reindexing did not obviously break important demo retrieval paths.
|
|
|
|
## Interview Answer
|
|
|
|
If asked how I verified the breadcrumb embedding change:
|
|
|
|
> I separated deterministic regression from live acceptance. The offline fixture baseline still runs without services. But because embedding changes only affect newly indexed vectors, I added a live post-reindex acceptance script. It calls the real `/api/search/similar` endpoint against representative breadcrumb-sensitive, troubleshooting, and AIOps queries, then writes JSON and Markdown reports. This lets me prove both that the code changed and that the live vector collection was refreshed.
|
|
|
|
If asked why the script does not reindex automatically:
|
|
|
|
> Reindexing mutates the vector store and depends on environment-specific data. I kept mutation explicit and made the script validation-only. That makes failures easier to diagnose: if retrieval does not improve, I can distinguish code changes, reindex state, and runtime retrieval behavior.
|