feat(harness,rag): dual LLM audit fields, run conclusion, and hybrid quality

Persist provider reasoning and assistant text separately on agent_reasoning_audit (DeepSeekAssistantMessage path), extract diagnosis_run.conclusion, enrich RAG tool audit (step_id/query/qualityScore), gate empty mysql tools, drop devtools, and align MVP docs after live E2E verification.
2026-07-28 19:43:13 +08:00
parent 2f40536248
commit 7ae9707a3b
116 changed files with 8364 additions and 1141 deletions
@@ -0,0 +1,50 @@
+# rag-eval-offline-baseline Specification
+
+## Purpose
+
+Keep the offline RAG retrieval baseline aligned with the hybrid search main path and fixture metadata needed for reproducible regression.
+
+## ADDED Requirements
+
+### Requirement: Snapshot generation SHALL use retrieval search mode
+
+RAG lookup fixture generation SHALL configure `retrieval.search.mode` and SHALL NOT require `retrieval.vector-store.mode` for knowledge snapshot generation.
+
+#### Scenario: Default hybrid generation
+
+- **WHEN** the snapshot generator is invoked with default parameters
+- **THEN** it SHALL run with `retrieval.search.mode=hybrid` (or equivalent default)
+- **AND** it SHALL NOT pass `retrieval.vector-store.mode` as a required generation setting
+
+#### Scenario: Dense mode override for comparison runs
+
+- **WHEN** the operator sets search mode to `dense`
+- **THEN** fixture generation SHALL use dense retrieval for that run
+
+### Requirement: Generated fixtures SHALL record search meta
+
+Each generated fixture file SHALL include stable identity and generation meta in addition to the lookup payload.
+
+#### Scenario: Meta fields present
+
+- **WHEN** a fixture is written for a golden case
+- **THEN** the fixture SHALL contain `caseId`, `query`, `retrievedAt`, and `searchMode`
+- **AND** when kb scope is configured non-empty, the fixture SHOULD contain `kbScope`
+
+### Requirement: Offline evaluation SHALL remain dependency-free
+
+The offline baseline checker SHALL evaluate golden cases against fixture files without calling Milvus, embedding APIs, or starting the full application.
+
+#### Scenario: Offline eval without live stack
+
+- **WHEN** `eval_rag_retrieval.py` (or successor) runs against cases and fixtures
+- **THEN** it SHALL produce pass/fail results using fixture contents only
+
+### Requirement: Eval documentation SHALL describe the hybrid-era loop
+
+Project eval README SHALL document seed import, snapshot generation with `search.mode`, offline eval, and baseline diff, without presenting Spring vector-store mode as the knowledge main path.
+
+#### Scenario: README main path
+
+- **WHEN** an engineer follows the eval README happy path
+- **THEN** the documented default generation mode SHALL be hybrid search mode