feat(rag): modularize knowledge retrieval pipeline
This commit is contained in:
zhuyongxin
@@ -0,0 +1,172 @@
|
||||
# Modular RAG Pipeline Proposal
|
||||
|
||||
## Problem
|
||||
|
||||
`lookup_knowledge` already exposes structured retrieval evidence, but the runtime flow is still concentrated inside `LookupKnowledgeTool`. Query understanding, vector retrieval, relevance normalization, evidence block construction, session deduplication, and trace recording are tightly coupled. This makes the RAG path harder to explain, test, evolve, and present as a modular Agent engineering design.
|
||||
|
||||
The current result contract also still carries the old `primary` / `supplement` model, where `primary` means L0 exact match and `supplement` means L1 semantic retrieval. That contract no longer matches the intended architecture: L0 should be a query understanding and retrieval-control signal, while L1 vector retrieval should be the main evidence source.
|
||||
|
||||
## Proposed Change
|
||||
|
||||
Refactor `lookup_knowledge` into a modular RAG pipeline while preserving the explicit Agent tool boundary and `tool_invocation` evidence trace.
|
||||
|
||||
Target pipeline:
|
||||
|
||||
```text
|
||||
LookupKnowledgeTool
|
||||
-> KnowledgeQueryTransformer
|
||||
-> KnowledgeDocumentRetriever
|
||||
-> KnowledgeEvidencePostProcessor
|
||||
-> KnowledgeContextPacker
|
||||
-> LookupResultAssembler
|
||||
-> ToolInvocationRecorder
|
||||
```
|
||||
|
||||
### Query Transformation
|
||||
|
||||
Introduce a query transformer around the current L0 analysis.
|
||||
|
||||
L0 SHALL provide:
|
||||
|
||||
- domain/category hints
|
||||
- matched keywords
|
||||
- extracted entities
|
||||
- optional metadata filter candidate
|
||||
- traceable query understanding data
|
||||
|
||||
L0 SHALL NOT act as a main evidence retrieval path in the normal flow.
|
||||
|
||||
### Retrieval
|
||||
|
||||
L1 vector retrieval remains the main document retrieval path through `VectorSearchService`, which already supports Spring AI `VectorStore` as the preferred path and Milvus SDK fallback.
|
||||
|
||||
MVP fallback strategy:
|
||||
|
||||
```text
|
||||
1. Run filtered vector retrieval with the L0-derived category filter when unambiguous.
|
||||
2. If filtered retrieval returns no usable evidence or low-quality evidence, retry raw query through unfiltered vector retrieval.
|
||||
3. If unfiltered retrieval also fails, return no_evidence.
|
||||
```
|
||||
|
||||
The fallback SHALL skip the L0-derived filter rather than returning L0 documents as fact evidence.
|
||||
|
||||
### Post-Retrieval Processing
|
||||
|
||||
Move evidence post-processing out of `LookupKnowledgeTool`.
|
||||
|
||||
The post-processor SHALL handle:
|
||||
|
||||
- relevance normalization
|
||||
- evidence block creation
|
||||
- source-level deduplication
|
||||
- rule-based lightweight rerank
|
||||
- retrieval trace and rerank trace generation
|
||||
|
||||
The first rerank implementation should be rule-based, using available signals such as vector score, domain match, entity match, keyword match, source type, and whether evidence aligns with query hints.
|
||||
|
||||
### Context Packing
|
||||
|
||||
Add a context packing step that converts final evidence blocks into an Agent-facing context package.
|
||||
|
||||
The packer SHALL:
|
||||
|
||||
- keep source/title/breadcrumb visible
|
||||
- obey a configurable character budget in the MVP
|
||||
- prioritize reranked evidence order
|
||||
- avoid duplicate source content
|
||||
- produce a compact summary of included and omitted evidence
|
||||
|
||||
### Result Contract
|
||||
|
||||
This change intentionally updates the `lookup_knowledge` return contract.
|
||||
|
||||
New preferred contract:
|
||||
|
||||
```text
|
||||
found
|
||||
evidenceBlocks
|
||||
contextPack
|
||||
retrievalTrace
|
||||
rerankTrace
|
||||
relevanceLevel
|
||||
completenessHint
|
||||
retrievedDomainsThisSession
|
||||
message
|
||||
```
|
||||
|
||||
The old `primary` and `supplement` fields may be removed as part of this change, because they encode the outdated assumption that L0 is the primary evidence source and L1 is supplemental evidence.
|
||||
|
||||
## Scope
|
||||
|
||||
In scope:
|
||||
|
||||
- Refactor `LookupKnowledgeTool` into a thin tool boundary and pipeline orchestrator.
|
||||
- Add local pipeline classes and DTOs for query transformation, retrieval result normalization, post-processing, context packing, and traces.
|
||||
- Update `LookupResult` to prefer `evidenceBlocks`, `contextPack`, `retrievalTrace`, and `rerankTrace`.
|
||||
- Remove or deprecate `primary` / `supplement` according to the final spec.
|
||||
- Update `ToolInvocationRecorder` to persist compact summaries for evidence blocks, context pack, retrieval trace, fallback reason, and rerank trace.
|
||||
- Update tool description / prompt wording so Agent behavior matches the new contract.
|
||||
- Update tests for filtered retrieval, unfiltered retry, rerank, context packing, evidence persistence, and result contract changes.
|
||||
- Update `rag-knowledge-retrieval` spec to remove L0 primary fallback and old compatibility-field requirements.
|
||||
|
||||
Out of scope:
|
||||
|
||||
- Replacing the explicit `lookup_knowledge` tool with an implicit Advisor.
|
||||
- Introducing a model-based reranker or cross-encoder.
|
||||
- Introducing BM25, RRF, Elasticsearch, or OpenSearch.
|
||||
- Migrating document upload, chunking, embedding writes, or Milvus schema.
|
||||
- Changing the Agent decision of when to call `lookup_knowledge`.
|
||||
|
||||
## Interface Impact
|
||||
|
||||
Impact level: L4 breaking interface change.
|
||||
|
||||
Reason:
|
||||
|
||||
- `LookupResult.primary` and `LookupResult.supplement` may be removed.
|
||||
- The JSON returned by the `lookup_knowledge` Agent tool changes shape.
|
||||
- Tests and internal consumers that read `primary` / `supplement` must migrate to `evidenceBlocks` and `contextPack`.
|
||||
|
||||
Known affected areas:
|
||||
|
||||
- `LookupKnowledgeTool`
|
||||
- `LookupResult`
|
||||
- `PrimaryResult` / `SupplementResult`
|
||||
- `ToolInvocationRecorder`
|
||||
- `LookupKnowledgeToolTest`
|
||||
- `ToolInvocationRecorderTest`
|
||||
- Agent tool prompt / executor prompt references
|
||||
- `rag-knowledge-retrieval` OpenSpec requirements
|
||||
- RAG docs under `mvp/architecture/`
|
||||
|
||||
Migration approach:
|
||||
|
||||
- Update all in-repo consumers in the same change.
|
||||
- Keep `lookup_knowledge` tool name and input argument unchanged.
|
||||
- Keep `tool_invocation` persistence compatible at table level while enriching `retrieval_details`.
|
||||
- Record fallback and no-evidence semantics explicitly so Verifier and Eval do not treat hint-only data as fact evidence.
|
||||
|
||||
## Context Constraints
|
||||
|
||||
Relevant project decisions:
|
||||
|
||||
- `lookup_knowledge` remains an explicit Agent evidence tool.
|
||||
- L0 is already documented as a hint layer, not a final decision layer.
|
||||
- Spring AI `VectorStore` is already the preferred retrieval path behind `VectorSearchService`.
|
||||
- Milvus SDK fallback remains valuable for MVP resilience.
|
||||
- `tool_invocation` is the stable evidence trace used by trace inspection, Verifier, and Eval.
|
||||
- Existing `rag-knowledge-retrieval` spec still contains legacy fallback and compatibility-field requirements that must be changed.
|
||||
|
||||
## Risks
|
||||
|
||||
- Breaking result contract may affect prompt behavior because the tool JSON changes.
|
||||
- Removing `primary` / `supplement` requires updating tests and recorder preview logic.
|
||||
- Rule-based rerank may create unexpected ordering changes if score semantics are not handled carefully.
|
||||
- Context packing can hide useful evidence if the budget is too small.
|
||||
- Existing OpenSpec requirements conflict with the new L0 fallback policy and must be updated before implementation.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- What exact minimum fields should `ContextPack`, `RetrievalTrace`, and `RerankTrace` expose in the committed spec?
|
||||
- Should `PrimaryResult` and `SupplementResult` classes be deleted immediately or left deprecated for one change cycle?
|
||||
- What threshold defines "filtered retrieval low quality" for triggering unfiltered retry?
|
||||
Reference in New Issue
Block a user