feat: treat l0 retrieval as domain hint
This commit is contained in:
aruo
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-04
|
||||
@@ -0,0 +1,62 @@
|
||||
## Context
|
||||
|
||||
`LookupKnowledgeTool` currently performs L0 keyword matching first. If L0 returns exactly one document, the tool treats it as high confidence, skips L1 semantic retrieval, and returns the L0-derived primary result. This was useful for the MVP but conflicts with the RAG refactor direction: L0 should constrain and explain retrieval, not decide final evidence by itself.
|
||||
|
||||
The refactor plan keeps L0 and metadata as valuable business signals. This change narrows L0 to a domain/entity hint provider while keeping `lookup_knowledge` as the explicit Agent tool entry point and preserving `tool_invocation` observability.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
|
||||
- Produce structured L0 hints from current keyword/frontmatter matches.
|
||||
- Include matched keywords, domains, entities, and titles in the trace.
|
||||
- Run L1 retrieval by default even for unique L0 hits.
|
||||
- Use a single clear L0 domain as a category filter for L1.
|
||||
- Preserve existing result shape as much as possible.
|
||||
|
||||
**Non-Goals:**
|
||||
|
||||
- Do not migrate to Spring AI VectorStore.
|
||||
- Do not implement BM25, RRF, rerank, or evidence packing.
|
||||
- Do not change document upload, chunking, or Milvus schema.
|
||||
- Do not remove L0.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Decision: Add a structured L0 hint result beside existing exact matches
|
||||
|
||||
`KnowledgeIndexService` will expose an `analyzeQuery` style method that returns:
|
||||
|
||||
- matched entries
|
||||
- matched keywords
|
||||
- domains/categories
|
||||
- entity terms
|
||||
|
||||
The existing `exactMatch` method can remain for compatibility.
|
||||
|
||||
Rationale: this avoids rewriting all callers while giving `LookupKnowledgeTool` richer data for tracing and filtering.
|
||||
|
||||
### Decision: Treat L0 unique hit as a hint, not a short circuit
|
||||
|
||||
`LookupKnowledgeTool` will no longer skip L1 solely because L0 matched one document. L1 will be called using the query and an optional category filter when L0 provides exactly one clear domain.
|
||||
|
||||
Rationale: the upcoming Spring AI retriever and evidence post-processing pipeline needs L0 and L1 to cooperate rather than use early return semantics.
|
||||
|
||||
### Decision: Keep `PRECISE` only when L0 and L1 both support the result
|
||||
|
||||
The relevance assessment should not mark `PRECISE` just because L0 matched once. It may mark `PRECISE` when L0 has one match and L1 returns evidence above the configured high relevance threshold, or when L0 has one match and L1 cannot run but the L0 result is still available.
|
||||
|
||||
Rationale: this preserves a graceful fallback while reducing overconfidence when semantic evidence disagrees.
|
||||
|
||||
### Decision: Persist L0 hints in retrieval details
|
||||
|
||||
`ToolInvocationRecorder.LookupKnowledgeRecord` will include fields for L0 matched keywords, domains, and entities. These will be serialized into `retrieval_details`.
|
||||
|
||||
Rationale: evidence trace and later evaluation need to explain why metadata filters or query augmentation happened.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- Increased latency because L1 is called more often -> keep topK small and allow category filter to reduce search scope.
|
||||
- L0 domain filter may be too narrow -> only apply it when there is exactly one nonblank domain; otherwise search without filter.
|
||||
- Existing tests may assume `L0` retrieval layer for unique hits -> update expectations to `L0+L1` when L1 participates.
|
||||
- If L1 fails, the tool should still return L0 evidence rather than fail the entire knowledge lookup.
|
||||
@@ -0,0 +1,30 @@
|
||||
## Why
|
||||
|
||||
The current `lookup_knowledge` implementation treats a unique L0 keyword hit as high confidence and skips L1 semantic retrieval. That makes L0 too authoritative for the RAG refactor target: L0 should provide domain/entity hints, metadata-filter intent, and explainability while final evidence still comes from the retrieval pipeline.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Change L0 from final retrieval decision maker to domain/entity hint provider.
|
||||
- Add structured L0 hint output that includes matched keywords, domains, entities, and matched titles.
|
||||
- Make `lookup_knowledge` run L1 semantic retrieval by default even when L0 has a unique hit.
|
||||
- Use L0 domain hints to pass category metadata filters into L1 when a single clear domain is detected.
|
||||
- Persist L0 hint details in `tool_invocation.retrieval_details`.
|
||||
- Keep `lookup_knowledge` as the explicit Agent tool entry point.
|
||||
- No Spring AI VectorStore migration in this change.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `rag-knowledge-retrieval`: Defines runtime behavior for the explicit RAG knowledge retrieval tool, including L0 hinting and L1 retrieval cooperation.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- None.
|
||||
|
||||
## Impact
|
||||
|
||||
- Affects `KnowledgeIndexService`, `LookupKnowledgeTool`, and `ToolInvocationRecorder`.
|
||||
- May affect retrieval latency because L1 is no longer skipped for unique L0 hits.
|
||||
- Improves traceability by recording L0 matched keywords/entities/domains in retrieval details.
|
||||
- Does not change document upload, chunking, Milvus schema, or Agent flow.
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Knowledge retrieval SHALL keep L0 as a hint provider
|
||||
The `lookup_knowledge` retrieval flow SHALL retain L0 keyword/frontmatter matching but use it as domain, entity, and explainability hint data rather than as the sole final retrieval decision.
|
||||
|
||||
#### Scenario: L0 produces traceable hint data
|
||||
- **WHEN** L0 matches one or more indexed knowledge entries
|
||||
- **THEN** the retrieval flow SHALL expose matched titles, matched keywords, domains or categories, and entity terms as structured hint data
|
||||
|
||||
#### Scenario: L0 does not bypass semantic retrieval by default
|
||||
- **WHEN** L0 returns exactly one match
|
||||
- **THEN** the retrieval flow SHALL still attempt semantic L1 retrieval unless L1 is unavailable or explicitly disabled by configuration
|
||||
|
||||
### Requirement: Knowledge retrieval SHALL use L0 domain as optional L1 filter
|
||||
The retrieval flow SHALL use L0 domain/category information as an optional metadata filter for L1 retrieval when the domain is unambiguous.
|
||||
|
||||
#### Scenario: Single domain filter
|
||||
- **WHEN** L0 hint data contains exactly one nonblank domain or category
|
||||
- **THEN** the L1 retrieval request SHALL include that category as a metadata filter
|
||||
|
||||
#### Scenario: Ambiguous domain fallback
|
||||
- **WHEN** L0 hint data contains zero domains or multiple domains
|
||||
- **THEN** the L1 retrieval request SHALL run without an L0-derived category filter
|
||||
|
||||
### Requirement: Knowledge retrieval SHALL preserve fallback evidence
|
||||
The retrieval flow SHALL still return useful L0 evidence when L1 produces no usable result.
|
||||
|
||||
#### Scenario: L1 has no results
|
||||
- **WHEN** L0 has at least one match and L1 returns no candidates
|
||||
- **THEN** the tool SHALL return an L0-based primary result
|
||||
- **AND** the relevance assessment SHALL not claim semantic support from L1
|
||||
|
||||
#### Scenario: L1 fails
|
||||
- **WHEN** L0 has at least one match and L1 retrieval throws or fails
|
||||
- **THEN** the tool SHALL return an L0-based primary result
|
||||
- **AND** the tool invocation record SHALL preserve the L0 hint details
|
||||
|
||||
### Requirement: Knowledge retrieval SHALL persist L0 hints
|
||||
The system SHALL persist L0 hint details in `tool_invocation.retrieval_details` for `lookup_knowledge` calls.
|
||||
|
||||
#### Scenario: Retrieval details include L0 hints
|
||||
- **WHEN** a `lookup_knowledge` call records a tool invocation
|
||||
- **THEN** `retrieval_details` SHALL include L0 matched keywords, domains, entities, and titles when available
|
||||
|
||||
#### Scenario: Retrieval layer reflects cooperating retrieval
|
||||
- **WHEN** both L0 hint data and L1 candidates participate in a lookup
|
||||
- **THEN** the recorded retrieval layer SHALL be `L0+L1`
|
||||
@@ -0,0 +1,27 @@
|
||||
## 1. L0 Hint Model
|
||||
|
||||
- [x] 1.1 Add structured L0 hint analysis in `KnowledgeIndexService`.
|
||||
- [x] 1.2 Preserve `exactMatch` compatibility for existing callers.
|
||||
|
||||
## 2. Retrieval Flow
|
||||
|
||||
- [x] 2.1 Update `LookupKnowledgeTool` so unique L0 hits no longer skip L1 by default.
|
||||
- [x] 2.2 Apply a category filter to L1 only when L0 hint data has one clear domain.
|
||||
- [x] 2.3 Preserve L0 fallback evidence when L1 is empty or fails.
|
||||
- [x] 2.4 Adjust relevance assessment so `PRECISE` no longer depends only on unique L0.
|
||||
|
||||
## 3. Trace Recording
|
||||
|
||||
- [x] 3.1 Extend `ToolInvocationRecorder.LookupKnowledgeRecord` with L0 matched keywords, domains, and entities.
|
||||
- [x] 3.2 Persist L0 hint fields in `retrieval_details`.
|
||||
|
||||
## 4. Tests
|
||||
|
||||
- [x] 4.1 Add or update unit tests for L0 hint extraction.
|
||||
- [x] 4.2 Add or update tests for `lookup_knowledge` unique-L0 plus L1 participation.
|
||||
- [x] 4.3 Run targeted tests and the RAG retrieval baseline evaluator.
|
||||
|
||||
## 5. Validation
|
||||
|
||||
- [x] 5.1 Run OpenSpec validation for the change.
|
||||
- [x] 5.2 Review git diff to confirm only expected code/spec/test files changed.
|
||||
Reference in New Issue
Block a user