feat: add rag evidence postprocess blocks

2026-07-05 03:03:18 +08:00
parent 4a94c14feb
commit 5197712719
12 changed files with 497 additions and 1 deletions
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-07-04
@@ -0,0 +1,53 @@
+## Context
+
+The previous change converted L0 into a hint provider and made L0/L1 cooperate. The next step is to stop treating the tool output as an unstructured primary/supplement pair. The Agent can keep receiving compatible fields, but retrieval internals and traces should have structured evidence blocks.
+
+This change is a bridge toward later DocumentPostProcessor-style behavior. It should be small enough to archive independently and should not introduce Spring AI dependencies.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Add `EvidenceBlock` DTOs to `LookupResult`.
+- Create evidence blocks from L0 matches and L1 results.
+- Deduplicate evidence by stable source key.
+- Capture source, title, breadcrumb, score, retrieval layer, hit reasons, and content preview.
+- Persist evidence blocks and postprocess counts in `tool_invocation.retrieval_details`.
+
+**Non-Goals:**
+
+- Do not implement neighbor chunk expansion yet.
+- Do not replace primary/supplement output.
+- Do not add cross-encoder or LLM rerank.
+- Do not migrate to Spring AI DocumentPostProcessor yet.
+
+## Decisions
+
+### Decision: Add evidence blocks while preserving existing result fields
+
+`LookupResult` will gain `List<EvidenceBlock> evidenceBlocks`. Existing `primary`, `supplement`, `found`, `relevanceLevel`, and `completenessHint` remain compatible.
+
+Rationale: this lets the Agent continue using the current shape while tests and traces begin validating the new evidence model.
+
+### Decision: Keep postprocess rule-based
+
+The evidence builder will use deterministic rules:
+
+- L0 entries become `L0` evidence.
+- L1 candidates become `L1` evidence.
+- Same source key is deduplicated.
+- Hit reasons are collected from L0 hints, L1 rank, category filters, and fallback state.
+
+Rationale: this is explainable, cheap, and suitable before introducing framework postprocessors.
+
+### Decision: Persist compact evidence summaries
+
+`ToolInvocationRecorder` will store compact evidence block metadata, not full content, inside `retrieval_details`.
+
+Rationale: `tool_invocation` should remain useful for trace review without duplicating large chunks.
+
+## Risks / Trade-offs
+
+- Evidence source keys may be imperfect before full metadata normalization -> fall back to file path, metadata, title, then rank.
+- Agent prompts may ignore `evidenceBlocks` initially -> keep primary/supplement compatibility.
+- Adding content previews increases tool output size -> cap evidence content length.
@@ -0,0 +1,28 @@
+## Why
+
+The current `lookup_knowledge` result is still shaped as one L0 primary result plus one L1 supplement. That makes retrieval evidence hard to inspect, hard to deduplicate, and hard to reuse later by verifier/evaluator code. The RAG refactor needs a structured evidence layer before Spring AI retriever migration.
+
+## What Changes
+
+- Add structured evidence blocks to `LookupResult`.
+- Build evidence blocks from L0 hint matches and L1 candidates.
+- Deduplicate evidence by source identity where possible.
+- Add hit reasons such as L0 matched keywords, L1 semantic rank, category filter, and fallback.
+- Persist evidence block summaries and postprocess counts in `tool_invocation.retrieval_details`.
+- Keep existing `primary` and `supplement` fields for compatibility.
+
+## Capabilities
+
+### New Capabilities
+
+- None.
+
+### Modified Capabilities
+
+- `rag-knowledge-retrieval`: Add evidence block post-processing requirements for explicit RAG knowledge retrieval.
+
+## Impact
+
+- Affects `LookupResult`, `LookupKnowledgeTool`, and `ToolInvocationRecorder`.
+- Updates tool tests and trace recorder tests.
+- Does not change Milvus schema, document upload, chunking, or Spring AI integration.
@@ -0,0 +1,35 @@
+## ADDED Requirements
+
+### Requirement: Knowledge retrieval SHALL return structured evidence blocks
+The `lookup_knowledge` retrieval flow SHALL expose retrieved evidence as structured evidence blocks in addition to the existing compatibility fields.
+
+#### Scenario: Evidence block contains source metadata
+- **WHEN** a `lookup_knowledge` call returns evidence
+- **THEN** each evidence block SHALL include source, title when available, breadcrumb when available, retrieval layer, content, and hit reasons
+
+#### Scenario: Compatibility fields remain available
+- **WHEN** evidence blocks are returned
+- **THEN** the existing `primary` and `supplement` result fields SHALL remain available when their source evidence exists
+
+### Requirement: Knowledge retrieval SHALL deduplicate evidence blocks
+The retrieval flow SHALL remove duplicate evidence blocks before returning them to the Agent.
+
+#### Scenario: Duplicate source deduplication
+- **WHEN** L0 and L1 produce evidence with the same source identity
+- **THEN** the retrieval flow SHALL keep a single evidence block for that source
+- **AND** the evidence block SHALL preserve hit reasons from both retrieval paths when available
+
+#### Scenario: Postprocess count tracking
+- **WHEN** evidence post-processing completes
+- **THEN** the tool invocation details SHALL record candidate count and final evidence block count
+
+### Requirement: Knowledge retrieval SHALL persist evidence block summaries
+The system SHALL persist compact evidence block summaries in `tool_invocation.retrieval_details`.
+
+#### Scenario: Evidence summaries are persisted
+- **WHEN** a `lookup_knowledge` call records a tool invocation
+- **THEN** `retrieval_details` SHALL include evidence block summaries containing source, title, retrieval layer, score when available, and hit reasons
+
+#### Scenario: Full content is not duplicated into retrieval details
+- **WHEN** evidence block summaries are persisted
+- **THEN** full evidence content SHALL be omitted or truncated so the trace record remains compact
@@ -0,0 +1,26 @@
+## 1. Evidence Model
+
+- [x] 1.1 Add an `EvidenceBlock` DTO.
+- [x] 1.2 Add evidence block list and postprocess count fields to `LookupResult`.
+
+## 2. Evidence Postprocess
+
+- [x] 2.1 Build evidence blocks from L0 matches and L1 candidates in `LookupKnowledgeTool`.
+- [x] 2.2 Deduplicate evidence by stable source key.
+- [x] 2.3 Preserve existing primary/supplement compatibility behavior.
+
+## 3. Trace Recording
+
+- [x] 3.1 Extend `ToolInvocationRecorder.LookupKnowledgeRecord` with evidence block summaries and postprocess counts.
+- [x] 3.2 Persist evidence block summaries in `retrieval_details`.
+
+## 4. Tests
+
+- [x] 4.1 Add or update tests for evidence block creation and deduplication.
+- [x] 4.2 Add or update tests for persisted evidence block summaries.
+- [x] 4.3 Run targeted tests and the RAG retrieval baseline evaluator.
+
+## 5. Validation
+
+- [x] 5.1 Run OpenSpec validation for the change.
+- [x] 5.2 Review git diff to confirm only expected code/spec/test files changed.
@@ -48,3 +48,37 @@ The system SHALL persist L0 hint details in `tool_invocation.retrieval_details`
 #### Scenario: Retrieval layer reflects cooperating retrieval
 - **WHEN** both L0 hint data and L1 candidates participate in a lookup
 - **THEN** the recorded retrieval layer SHALL be `L0+L1`
+
+### Requirement: Knowledge retrieval SHALL return structured evidence blocks
+The `lookup_knowledge` retrieval flow SHALL expose retrieved evidence as structured evidence blocks in addition to the existing compatibility fields.
+
+#### Scenario: Evidence block contains source metadata
+- **WHEN** a `lookup_knowledge` call returns evidence
+- **THEN** each evidence block SHALL include source, title when available, breadcrumb when available, retrieval layer, content, and hit reasons
+
+#### Scenario: Compatibility fields remain available
+- **WHEN** evidence blocks are returned
+- **THEN** the existing `primary` and `supplement` result fields SHALL remain available when their source evidence exists
+
+### Requirement: Knowledge retrieval SHALL deduplicate evidence blocks
+The retrieval flow SHALL remove duplicate evidence blocks before returning them to the Agent.
+
+#### Scenario: Duplicate source deduplication
+- **WHEN** L0 and L1 produce evidence with the same source identity
+- **THEN** the retrieval flow SHALL keep a single evidence block for that source
+- **AND** the evidence block SHALL preserve hit reasons from both retrieval paths when available
+
+#### Scenario: Postprocess count tracking
+- **WHEN** evidence post-processing completes
+- **THEN** the tool invocation details SHALL record candidate count and final evidence block count
+
+### Requirement: Knowledge retrieval SHALL persist evidence block summaries
+The system SHALL persist compact evidence block summaries in `tool_invocation.retrieval_details`.
+
+#### Scenario: Evidence summaries are persisted
+- **WHEN** a `lookup_knowledge` call records a tool invocation
+- **THEN** `retrieval_details` SHALL include evidence block summaries containing source, title, retrieval layer, score when available, and hit reasons
+
+#### Scenario: Full content is not duplicated into retrieval details
+- **WHEN** evidence block summaries are persisted
+- **THEN** full evidence content SHALL be omitted or truncated so the trace record remains compact