# rag-knowledge-retrieval Specification

## Purpose
Define the runtime contract for the explicit `lookup_knowledge` Agent tool, including how L0 keyword/frontmatter hints cooperate with L1 semantic retrieval while preserving metadata filters, fallback evidence, and traceable retrieval details.
## Requirements
### Requirement: Knowledge retrieval SHALL keep L0 as a hint provider
The `lookup_knowledge` retrieval flow SHALL retain L0 keyword/frontmatter matching but use it only as query understanding, filtering, rerank, and explainability hint data rather than as a final evidence retrieval decision.

#### Scenario: L0 produces traceable hint data
- **WHEN** L0 matches one or more indexed knowledge entries
- **THEN** the retrieval flow SHALL expose matched titles, matched keywords, domains or categories, and entity terms as structured hint data

#### Scenario: L0 does not bypass semantic retrieval
- **WHEN** L0 returns exactly one match
- **THEN** the retrieval flow SHALL still attempt semantic L1 retrieval unless L1 is explicitly disabled by configuration

#### Scenario: L0 hints do not become normal evidence
- **WHEN** L1 retrieval returns no usable evidence
- **THEN** L0 matched documents SHALL NOT be returned as fact evidence blocks
- **AND** L0 hint data MAY still be recorded in retrieval trace details

### Requirement: Knowledge retrieval SHALL use L0 domain as optional L1 filter
The retrieval flow SHALL use L0 domain/category information as an optional metadata filter for L1 retrieval when the domain is unambiguous.

#### Scenario: Single domain filter
- **WHEN** L0 hint data contains exactly one nonblank domain or category
- **THEN** the L1 retrieval request SHALL include that category as a metadata filter

#### Scenario: Ambiguous domain fallback
- **WHEN** L0 hint data contains zero domains or multiple domains
- **THEN** the L1 retrieval request SHALL run without an L0-derived category filter

### Requirement: Knowledge retrieval SHALL persist L0 hints
The system SHALL persist L0 hint details in `tool_invocation.retrieval_details` for `lookup_knowledge` calls.

#### Scenario: Retrieval details include L0 hints
- **WHEN** a `lookup_knowledge` call records a tool invocation
- **THEN** `retrieval_details` SHALL include L0 matched keywords, domains, entities, and titles when available

#### Scenario: Retrieval layer reflects cooperating retrieval
- **WHEN** both L0 hint data and L1 candidates participate in a lookup
- **THEN** the recorded retrieval layer SHALL be `L0+L1`

### Requirement: Knowledge retrieval SHALL return structured evidence blocks
The `lookup_knowledge` retrieval flow SHALL expose retrieved evidence as structured evidence blocks.

#### Scenario: Evidence block contains source metadata
- **WHEN** a `lookup_knowledge` call returns evidence
- **THEN** each evidence block SHALL include source, title when available, breadcrumb when available, retrieval layer, content, and hit reasons

#### Scenario: Evidence blocks are the primary evidence contract
- **WHEN** evidence blocks are returned
- **THEN** Agent-facing knowledge content SHALL be derived from evidence blocks and context pack
- **AND** the result SHALL NOT rely on legacy `primary` or `supplement` fields for L0/L1 meaning

### Requirement: Knowledge retrieval SHALL deduplicate evidence blocks

The retrieval flow SHALL remove duplicate evidence blocks before returning them to the Agent. Duplicates are defined by chunk-level `evidenceKey` identity, not by document source alone.

#### Scenario: Chunk-level deduplication keeps distinct chunks

- **WHEN** L1 produces multiple candidates with the same source but different chunk identities
- **THEN** the retrieval flow SHALL keep separate evidence blocks for those chunk identities subject to per-document caps
- **AND** SHALL NOT collapse them solely because source is equal

#### Scenario: Same evidenceKey collapses

- **WHEN** two candidates share the same evidenceKey
- **THEN** the retrieval flow SHALL keep a single evidence block for that identity
- **AND** the evidence block SHALL preserve hit reasons from both paths when available

#### Scenario: Postprocess count tracking

- **WHEN** evidence post-processing completes
- **THEN** the tool invocation details SHALL record candidate count and final evidence block count

### Requirement: Knowledge retrieval SHALL persist evidence block summaries
The system SHALL persist compact evidence block summaries in `tool_invocation.retrieval_details`.

#### Scenario: Evidence summaries are persisted
- **WHEN** a `lookup_knowledge` call records a tool invocation
- **THEN** `retrieval_details` SHALL include evidence block summaries containing source, title, retrieval layer, score when available, and hit reasons

#### Scenario: Full content is not duplicated into retrieval details
- **WHEN** evidence block summaries are persisted
- **THEN** full evidence content SHALL be omitted or truncated so the trace record remains compact

### Requirement: Knowledge retrieval SHALL support a disabled-by-default Spring AI sidecar
The retrieval system SHALL allow a Spring AI VectorStore retrieval path to be wired as a sidecar without changing the default `lookup_knowledge` runtime path.

#### Scenario: Sidecar disabled by default
- **WHEN** the application starts without explicit sidecar enablement
- **THEN** `lookup_knowledge` SHALL continue using the existing retrieval path
- **AND** Chat and AIOps runtime behavior SHALL not depend on the sidecar

#### Scenario: Sidecar failure does not break main retrieval
- **WHEN** the Spring AI sidecar is enabled but cannot initialize or query successfully
- **THEN** the existing retrieval path SHALL remain usable
- **AND** the failure SHALL be reported as sidecar status rather than as a main retrieval failure

### Requirement: Knowledge retrieval SHALL normalize sidecar results for comparison
The sidecar retrieval path SHALL expose results in a comparable structure aligned with the current retrieval result shape.

#### Scenario: Comparable result metadata
- **WHEN** sidecar retrieval returns candidates
- **THEN** each comparable result SHALL include source or doc id, title when available, breadcrumb when available, category when available, rank, content preview, and the sidecar score label/value

#### Scenario: Score semantics are explicit
- **WHEN** current retrieval and sidecar retrieval scores are compared
- **THEN** the report SHALL label score semantics by path instead of assuming direct numeric equivalence

### Requirement: Knowledge retrieval SHALL prefer Spring AI VectorStore when configured
The retrieval service SHALL support Spring AI VectorStore as the preferred vector retrieval abstraction without changing the `lookup_knowledge` tool contract.

#### Scenario: VectorStore mode uses Spring AI
- **WHEN** retrieval vector store mode is configured as `spring-ai`
- **THEN** semantic retrieval SHALL query through Spring AI `VectorStore`
- **AND** the returned candidates SHALL be normalized into the existing vector search result shape

#### Scenario: Auto mode prefers VectorStore
- **WHEN** retrieval vector store mode is configured as `auto`
- **AND** a Spring AI `VectorStore` bean is available
- **THEN** semantic retrieval SHALL attempt Spring AI `VectorStore` before the SDK path

### Requirement: Knowledge retrieval SHALL preserve SDK fallback
The retrieval service SHALL keep the existing Milvus SDK retrieval implementation available.

#### Scenario: SDK mode bypasses VectorStore
- **WHEN** retrieval vector store mode is configured as `sdk`
- **THEN** semantic retrieval SHALL use the existing Milvus SDK path

#### Scenario: Auto fallback uses SDK
- **WHEN** retrieval vector store mode is `auto`
- **AND** Spring AI `VectorStore` is unavailable or fails
- **THEN** semantic retrieval SHALL fall back to the existing Milvus SDK path

### Requirement: Knowledge retrieval SHALL keep score semantics explicit
The retrieval service SHALL preserve score semantics when results come from different retrieval implementations.

#### Scenario: SDK score remains L2 distance
- **WHEN** a candidate is returned by the SDK path
- **THEN** its score semantics SHALL remain compatible with existing L2 distance normalization

#### Scenario: VectorStore score is mapped without changing tool contract
- **WHEN** a candidate is returned by Spring AI `VectorStore`
- **THEN** it SHALL be mapped into the existing result shape
- **AND** trace or comparison code SHALL be able to distinguish it as a VectorStore similarity score when needed

### Requirement: Knowledge retrieval SHALL reuse the existing Milvus collection
Spring AI Milvus integration SHALL be configured to use the existing collection schema unless explicitly changed.

#### Scenario: Existing field mapping
- **WHEN** Spring AI Milvus VectorStore is configured
- **THEN** it SHALL use the existing id, content, vector, and metadata field names
- **AND** it SHALL use the configured embedding dimension and metric type compatible with existing vectors

### Requirement: Knowledge retrieval SHALL use a modular RAG pipeline
The `lookup_knowledge` tool SHALL route each request through explicit query transformation, vector retrieval, post-retrieval processing, context packing, result assembly, and trace recording components.

#### Scenario: Pipeline components execute in order
- **WHEN** `lookup_knowledge` receives a query
- **THEN** the system SHALL transform the query before retrieval
- **AND** it SHALL retrieve vector candidates before post-processing
- **AND** it SHALL build evidence blocks before context packing
- **AND** it SHALL record trace details after result assembly

#### Scenario: Tool boundary remains explicit
- **WHEN** the modular pipeline is used
- **THEN** the Agent SHALL still call the explicit `lookup_knowledge` tool with the same query argument
- **AND** the implementation SHALL NOT require an implicit Advisor to inject knowledge into every chat response

### Requirement: Knowledge retrieval SHALL retry without L0 filter when filtered L1 is low quality
The retrieval flow SHALL treat L0-derived category filtering as an optimization, not as a hard dependency for final recall.

#### Scenario: Filtered retrieval succeeds
- **WHEN** L0 provides an unambiguous category filter
- **AND** filtered L1 retrieval returns usable evidence at or above the configured reference threshold
- **THEN** the tool SHALL use the filtered L1 candidates without running an unfiltered retry

#### Scenario: Filtered retrieval returns no evidence
- **WHEN** L0 provides a category filter
- **AND** filtered L1 retrieval returns no candidates or no final evidence blocks
- **THEN** the tool SHALL retry L1 retrieval with the raw query and no L0-derived category filter
- **AND** the retrieval trace SHALL record fallback reason `filtered_vector_no_evidence`

#### Scenario: Filtered retrieval is below reference quality
- **WHEN** L0 provides a category filter
- **AND** filtered L1 retrieval returns candidates whose top normalized similarity is below the configured reference threshold
- **THEN** the tool SHALL retry L1 retrieval with the raw query and no L0-derived category filter
- **AND** the retrieval trace SHALL record fallback reason `filtered_vector_low_quality`

#### Scenario: Both retrieval attempts fail
- **WHEN** filtered L1 retrieval and unfiltered L1 retry both produce no usable evidence
- **THEN** the tool SHALL return `found=false`
- **AND** the tool SHALL set evidence status to `no_evidence`
- **AND** the tool SHALL NOT return L0 documents as fact evidence

### Requirement: Knowledge retrieval SHALL return an evidence-first result contract
The `lookup_knowledge` result SHALL expose structured evidence and packed context as the preferred contract.

#### Scenario: Evidence result contains context and traces
- **WHEN** `lookup_knowledge` returns usable evidence
- **THEN** the result SHALL include `evidenceBlocks`
- **AND** it SHALL include `contextPack`
- **AND** it SHALL include `retrievalTrace`
- **AND** it SHALL include `rerankTrace`
- **AND** it SHALL include `relevanceLevel` and `completenessHint`

#### Scenario: No-evidence result keeps traceability
- **WHEN** `lookup_knowledge` returns no usable evidence
- **THEN** the result SHALL include `found=false`
- **AND** it SHALL include a message explaining that no knowledge evidence was found
- **AND** it SHALL include retrieval trace details for attempted retrieval paths

### Requirement: Knowledge retrieval SHALL pack evidence context for Agent consumption
The post-retrieval flow SHALL convert final evidence blocks into a compact context package for the Agent.

#### Scenario: Context pack preserves source metadata
- **WHEN** evidence blocks are packed
- **THEN** the packed context SHALL preserve source, title when available, breadcrumb when available, and hit reasons for included evidence

#### Scenario: Context pack respects budget
- **WHEN** final evidence content exceeds the configured context budget
- **THEN** the packer SHALL truncate content rather than source metadata
- **AND** it SHALL record included and omitted sources in the context pack summary

### Requirement: Knowledge retrieval SHALL rerank evidence with traceable rule signals
The post-retrieval flow SHALL rerank vector candidates using deterministic rule-based signals and expose the explanation.

#### Scenario: Rerank trace records score contributions
- **WHEN** candidates are reranked
- **THEN** the rerank trace SHALL record final rank, source, base retrieval score when available, and major boost reasons for top evidence blocks

#### Scenario: Query hints influence rerank without becoming evidence
- **WHEN** L0 query hints match candidate metadata or content
- **THEN** the reranker MAY boost the candidate
- **AND** the evidence block SHALL record the hint as a hit reason
- **AND** the system SHALL NOT treat the L0 hint itself as fact evidence

### Requirement: Lookup knowledge SHALL persist modular RAG trace details
The `lookup_knowledge` tool SHALL persist modular RAG pipeline details in `tool_invocation.retrieval_details` for every new lookup invocation.

#### Scenario: Evidence lookup persists modular detail keys
- **WHEN** `lookup_knowledge` returns usable evidence
- **THEN** `retrieval_details` SHALL include `query_transform`
- **AND** it SHALL include `retrieval_trace`
- **AND** it SHALL include `context_pack_summary`
- **AND** it SHALL include `rerank_trace`
- **AND** it SHALL include `evidence_blocks`

#### Scenario: Fallback lookup persists fallback reason
- **WHEN** `lookup_knowledge` performs an unfiltered retry after filtered retrieval fails or is low quality
- **THEN** `retrieval_details.retrieval_trace` SHALL include the selected attempt
- **AND** `retrieval_details.fallback_reason` SHALL preserve the fallback reason

#### Scenario: No-evidence lookup still preserves trace
- **WHEN** `lookup_knowledge` returns no usable evidence
- **THEN** `retrieval_details` SHALL still include retrieval trace information for attempted retrieval paths
- **AND** it SHALL not include full evidence content as duplicated trace data