feat(rag): modularize knowledge retrieval pipeline

openspec/specs/rag-knowledge-retrieval/spec.md

@@ -4,15 +4,20 @@
 Define the runtime contract for the explicit `lookup_knowledge` Agent tool, including how L0 keyword/frontmatter hints cooperate with L1 semantic retrieval while preserving metadata filters, fallback evidence, and traceable retrieval details.
 ## Requirements
 ### Requirement: Knowledge retrieval SHALL keep L0 as a hint provider
-The `lookup_knowledge` retrieval flow SHALL retain L0 keyword/frontmatter matching but use it as domain, entity, and explainability hint data rather than as the sole final retrieval decision.
+The `lookup_knowledge` retrieval flow SHALL retain L0 keyword/frontmatter matching but use it only as query understanding, filtering, rerank, and explainability hint data rather than as a final evidence retrieval decision.
 
 #### Scenario: L0 produces traceable hint data
 - **WHEN** L0 matches one or more indexed knowledge entries
 - **THEN** the retrieval flow SHALL expose matched titles, matched keywords, domains or categories, and entity terms as structured hint data
 
-#### Scenario: L0 does not bypass semantic retrieval by default
+#### Scenario: L0 does not bypass semantic retrieval
 - **WHEN** L0 returns exactly one match
-- **THEN** the retrieval flow SHALL still attempt semantic L1 retrieval unless L1 is unavailable or explicitly disabled by configuration
+- **THEN** the retrieval flow SHALL still attempt semantic L1 retrieval unless L1 is explicitly disabled by configuration
+
+#### Scenario: L0 hints do not become normal evidence
+- **WHEN** L1 retrieval returns no usable evidence
+- **THEN** L0 matched documents SHALL NOT be returned as fact evidence blocks
+- **AND** L0 hint data MAY still be recorded in retrieval trace details
 
 ### Requirement: Knowledge retrieval SHALL use L0 domain as optional L1 filter
 The retrieval flow SHALL use L0 domain/category information as an optional metadata filter for L1 retrieval when the domain is unambiguous.
@@ -25,19 +30,6 @@ The retrieval flow SHALL use L0 domain/category information as an optional metad
 - **WHEN** L0 hint data contains zero domains or multiple domains
 - **THEN** the L1 retrieval request SHALL run without an L0-derived category filter
 
-### Requirement: Knowledge retrieval SHALL preserve fallback evidence
-The retrieval flow SHALL still return useful L0 evidence when L1 produces no usable result.
-
-#### Scenario: L1 has no results
-- **WHEN** L0 has at least one match and L1 returns no candidates
-- **THEN** the tool SHALL return an L0-based primary result
-- **AND** the relevance assessment SHALL not claim semantic support from L1
-
-#### Scenario: L1 fails
-- **WHEN** L0 has at least one match and L1 retrieval throws or fails
-- **THEN** the tool SHALL return an L0-based primary result
-- **AND** the tool invocation record SHALL preserve the L0 hint details
-
 ### Requirement: Knowledge retrieval SHALL persist L0 hints
 The system SHALL persist L0 hint details in `tool_invocation.retrieval_details` for `lookup_knowledge` calls.
 
@@ -50,15 +42,16 @@ The system SHALL persist L0 hint details in `tool_invocation.retrieval_details`
 - **THEN** the recorded retrieval layer SHALL be `L0+L1`
 
 ### Requirement: Knowledge retrieval SHALL return structured evidence blocks
-The `lookup_knowledge` retrieval flow SHALL expose retrieved evidence as structured evidence blocks in addition to the existing compatibility fields.
+The `lookup_knowledge` retrieval flow SHALL expose retrieved evidence as structured evidence blocks.
 
 #### Scenario: Evidence block contains source metadata
 - **WHEN** a `lookup_knowledge` call returns evidence
 - **THEN** each evidence block SHALL include source, title when available, breadcrumb when available, retrieval layer, content, and hit reasons
 
-#### Scenario: Compatibility fields remain available
+#### Scenario: Evidence blocks are the primary evidence contract
 - **WHEN** evidence blocks are returned
-- **THEN** the existing `primary` and `supplement` result fields SHALL remain available when their source evidence exists
+- **THEN** Agent-facing knowledge content SHALL be derived from evidence blocks and context pack
+- **AND** the result SHALL NOT rely on legacy `primary` or `supplement` fields for L0/L1 meaning
 
 ### Requirement: Knowledge retrieval SHALL deduplicate evidence blocks
 The retrieval flow SHALL remove duplicate evidence blocks before returning them to the Agent.
@@ -151,3 +144,86 @@ Spring AI Milvus integration SHALL be configured to use the existing collection
 - **WHEN** Spring AI Milvus VectorStore is configured
 - **THEN** it SHALL use the existing id, content, vector, and metadata field names
 - **AND** it SHALL use the configured embedding dimension and metric type compatible with existing vectors
+
+### Requirement: Knowledge retrieval SHALL use a modular RAG pipeline
+The `lookup_knowledge` tool SHALL route each request through explicit query transformation, vector retrieval, post-retrieval processing, context packing, result assembly, and trace recording components.
+
+#### Scenario: Pipeline components execute in order
+- **WHEN** `lookup_knowledge` receives a query
+- **THEN** the system SHALL transform the query before retrieval
+- **AND** it SHALL retrieve vector candidates before post-processing
+- **AND** it SHALL build evidence blocks before context packing
+- **AND** it SHALL record trace details after result assembly
+
+#### Scenario: Tool boundary remains explicit
+- **WHEN** the modular pipeline is used
+- **THEN** the Agent SHALL still call the explicit `lookup_knowledge` tool with the same query argument
+- **AND** the implementation SHALL NOT require an implicit Advisor to inject knowledge into every chat response
+
+### Requirement: Knowledge retrieval SHALL retry without L0 filter when filtered L1 is low quality
+The retrieval flow SHALL treat L0-derived category filtering as an optimization, not as a hard dependency for final recall.
+
+#### Scenario: Filtered retrieval succeeds
+- **WHEN** L0 provides an unambiguous category filter
+- **AND** filtered L1 retrieval returns usable evidence at or above the configured reference threshold
+- **THEN** the tool SHALL use the filtered L1 candidates without running an unfiltered retry
+
+#### Scenario: Filtered retrieval returns no evidence
+- **WHEN** L0 provides a category filter
+- **AND** filtered L1 retrieval returns no candidates or no final evidence blocks
+- **THEN** the tool SHALL retry L1 retrieval with the raw query and no L0-derived category filter
+- **AND** the retrieval trace SHALL record fallback reason `filtered_vector_no_evidence`
+
+#### Scenario: Filtered retrieval is below reference quality
+- **WHEN** L0 provides a category filter
+- **AND** filtered L1 retrieval returns candidates whose top normalized similarity is below the configured reference threshold
+- **THEN** the tool SHALL retry L1 retrieval with the raw query and no L0-derived category filter
+- **AND** the retrieval trace SHALL record fallback reason `filtered_vector_low_quality`
+
+#### Scenario: Both retrieval attempts fail
+- **WHEN** filtered L1 retrieval and unfiltered L1 retry both produce no usable evidence
+- **THEN** the tool SHALL return `found=false`
+- **AND** the tool SHALL set evidence status to `no_evidence`
+- **AND** the tool SHALL NOT return L0 documents as fact evidence
+
+### Requirement: Knowledge retrieval SHALL return an evidence-first result contract
+The `lookup_knowledge` result SHALL expose structured evidence and packed context as the preferred contract.
+
+#### Scenario: Evidence result contains context and traces
+- **WHEN** `lookup_knowledge` returns usable evidence
+- **THEN** the result SHALL include `evidenceBlocks`
+- **AND** it SHALL include `contextPack`
+- **AND** it SHALL include `retrievalTrace`
+- **AND** it SHALL include `rerankTrace`
+- **AND** it SHALL include `relevanceLevel` and `completenessHint`
+
+#### Scenario: No-evidence result keeps traceability
+- **WHEN** `lookup_knowledge` returns no usable evidence
+- **THEN** the result SHALL include `found=false`
+- **AND** it SHALL include a message explaining that no knowledge evidence was found
+- **AND** it SHALL include retrieval trace details for attempted retrieval paths
+
+### Requirement: Knowledge retrieval SHALL pack evidence context for Agent consumption
+The post-retrieval flow SHALL convert final evidence blocks into a compact context package for the Agent.
+
+#### Scenario: Context pack preserves source metadata
+- **WHEN** evidence blocks are packed
+- **THEN** the packed context SHALL preserve source, title when available, breadcrumb when available, and hit reasons for included evidence
+
+#### Scenario: Context pack respects budget
+- **WHEN** final evidence content exceeds the configured context budget
+- **THEN** the packer SHALL truncate content rather than source metadata
+- **AND** it SHALL record included and omitted sources in the context pack summary
+
+### Requirement: Knowledge retrieval SHALL rerank evidence with traceable rule signals
+The post-retrieval flow SHALL rerank vector candidates using deterministic rule-based signals and expose the explanation.
+
+#### Scenario: Rerank trace records score contributions
+- **WHEN** candidates are reranked
+- **THEN** the rerank trace SHALL record final rank, source, base retrieval score when available, and major boost reasons for top evidence blocks
+
+#### Scenario: Query hints influence rerank without becoming evidence
+- **WHEN** L0 query hints match candidate metadata or content
+- **THEN** the reranker MAY boost the candidate
+- **AND** the evidence block SHALL record the hint as a hit reason
+- **AND** the system SHALL NOT treat the L0 hint itself as fact evidence