docs: archive rag and aiops query changes

2026-07-05 13:12:47 +08:00
parent 72a3dbf8c5
commit 2658742119
12 changed files with 37 additions and 0 deletions
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-07-05
@@ -0,0 +1,53 @@
+## Context
+
+`AiOpsService.buildTaskPrompt(...)` already distinguishes two modes:
+
+- `PAYLOAD_TARGETED`: diagnose the supplied alert payload.
+- `AUTO_DISCOVERY`: discover active alerts first.
+
+In payload-targeted mode, the prompt includes alert fields, but it does not provide a normalized retrieval query for `lookup_knowledge`. The Agent may still call the tool, but the exact query is left to model behavior.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Build a deterministic retrieval query from AIOps payload fields.
+- Preserve the original payload fields in the prompt.
+- Make the recommended knowledge query visible in prompt text for trace/debugging.
+- Keep the Agent responsible for deciding when to call `lookup_knowledge`.
+
+**Non-Goals:**
+
+- Do not add automatic pre-Agent retrieval.
+- Do not add verifier logic.
+- Do not change tool invocation schema.
+- Do not change L0/L1 retrieval internals.
+
+## Decisions
+
+### Decision 1: Prompt-Level Query Augmentation
+
+Add a recommended knowledge query to the payload-targeted prompt instead of calling `lookup_knowledge` directly.
+
+Rationale:
+
+- The current AIOps flow is Agent-driven; tools remain explicit.
+- Prompt-level augmentation is low risk and easy to inspect.
+- It avoids introducing another hidden retrieval path that would complicate trace semantics.
+
+Alternative considered: automatically call `lookup_knowledge` before invoking the Supervisor. This was rejected because it changes execution behavior and may create evidence that the Agent did not request.
+
+### Decision 2: Preserve Original Query Terms
+
+The generated query includes raw alert/service/symptom terms rather than replacing them with broad domains.
+
+Rationale:
+
+- Alert name, service name, severity, and symptom are high-value retrieval terms.
+- Broad categories such as `infrastructure` are useful hints but should not replace concrete terms.
+
+## Risks / Trade-offs
+
+- [Risk] Prompt grows slightly longer. -> Mitigation: keep the query compact and skip blank fields.
+- [Risk] The model may ignore the recommendation. -> Mitigation: make the instruction explicit and test prompt inclusion.
+- [Risk] Query construction duplicates some summary fields. -> Mitigation: treat the retrieval query as a compact, tool-oriented view of the payload.
@@ -0,0 +1,25 @@
+## Why
+
+AIOps payload-targeted diagnosis already scopes the Agent to the supplied alert, but the prompt does not provide a deterministic knowledge-retrieval query. This leaves the Agent to invent lookup terms from the full prompt, which can omit high-value alert fields such as alert name, service, severity, and symptom.
+
+## What Changes
+
+- Build a stable knowledge retrieval query from AIOps payload fields.
+- Include the generated retrieval query in payload-targeted prompts as the recommended `lookup_knowledge` query.
+- Keep retrieval explicit through the Agent tool; do not automatically call `lookup_knowledge` before the Agent runs.
+- Add focused tests for query construction and prompt inclusion.
+
+## Capabilities
+
+### New Capabilities
+
+None.
+
+### Modified Capabilities
+
+- `aiops-traceable-diagnosis-entry`: Payload-targeted AIOps prompts include a deterministic knowledge retrieval query derived from alert payload fields.
+
+## Impact
+
+- Affects `AiOpsService` prompt construction only.
+- Does not change the `lookup_knowledge` tool signature, VectorStore retrieval, AIOps API contract, or trace schema.
@@ -0,0 +1,18 @@
+## ADDED Requirements
+
+### Requirement: AIOps payload prompts SHALL include a recommended knowledge query
+When an AIOps request includes alert payload fields, the system SHALL include a deterministic recommended knowledge retrieval query in the prompt sent to the Agent flow.
+
+#### Scenario: Payload-targeted prompt includes knowledge query
+- **WHEN** an AIOps request contains alert name, service, severity, or description
+- **THEN** the generated task prompt SHALL include a recommended `lookup_knowledge` query derived from the supplied payload fields
+
+#### Scenario: Query skips blank fields
+- **WHEN** some AIOps payload fields are blank
+- **THEN** the recommended knowledge query SHALL omit those blank fields
+- **AND** it SHALL preserve the non-blank alert-specific terms
+
+#### Scenario: Auto-discovery prompt does not invent payload query
+- **WHEN** an AIOps request does not include alert payload fields
+- **THEN** the generated task prompt SHALL remain in auto-discovery mode
+- **AND** it SHALL not include a payload-derived recommended knowledge query
@@ -0,0 +1,14 @@
+## 1. Prompt Query Construction
+
+- [x] 1.1 Add a deterministic AIOps knowledge query builder from payload fields.
+- [x] 1.2 Include the recommended query in payload-targeted task prompts.
+
+## 2. Tests And Docs
+
+- [x] 2.1 Add unit tests for query construction and prompt inclusion.
+- [x] 2.2 Update interview/RAG notes to reflect AIOps payload query augmentation.
+
+## 3. Verification
+
+- [x] 3.1 Run focused AIOps service tests.
+- [x] 3.2 Validate the OpenSpec change and review git scope.
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-07-05
@@ -0,0 +1,72 @@
+## Context
+
+The indexing path now builds embeddings from structured text:
+
+```text
+Title: {title}
+Path: {breadcrumb}
+Content:
+{content}
+```
+
+The persisted Milvus `content` field remains the raw chunk content. This improves semantic recall for section-aware questions, but only after documents are reindexed. Existing vectors were generated from the previous content-only input and cannot reflect the new breadcrumb signal.
+
+The repository already has an offline fixture-based retrieval baseline. That baseline is useful for deterministic regression checks, but it does not prove that the live Milvus/Zilliz collection has been reindexed or that the running service returns breadcrumb-aware results.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Provide an explicit post-reindex live acceptance flow.
+- Make the reindex prerequisite visible in documentation.
+- Add a small script that calls the live retrieval endpoint with representative queries and writes reviewable reports.
+- Keep the live flow optional so unit tests and offline evaluation remain service-free.
+
+**Non-Goals:**
+
+- Do not add a new reindex API in this change.
+- Do not automatically mutate live Milvus/Zilliz data from the acceptance script.
+- Do not change `lookup_knowledge`, VectorStore retrieval, or Milvus schema.
+- Do not commit environment-specific live results unless they were intentionally captured for interview evidence.
+
+## Decisions
+
+### Decision 1: Keep Reindex Manual And Explicit
+
+The acceptance flow documents that reindexing must happen before live validation, but it does not perform the reindex itself.
+
+Rationale:
+
+- Reindexing is a data mutation and can be slow or environment-specific.
+- The existing project already has indexing paths through upload, document management, and knowledge-base initialization.
+- Keeping mutation separate from validation makes failures easier to diagnose.
+
+Alternative considered: add a script that triggers reindex and then validates. This was rejected for now because it would need environment-specific credentials, source selection, and safety controls.
+
+### Decision 2: Use HTTP Endpoint Validation
+
+The script calls `/api/search/similar` instead of invoking Java services directly.
+
+Rationale:
+
+- It validates the same runtime path used in demos.
+- It works across SDK, Spring AI, and auto retrieval modes.
+- It produces a simple artifact that can be shown in interview material.
+
+Alternative considered: add a Java integration test. This was rejected because live Milvus and Spring Boot availability should remain optional.
+
+### Decision 3: Preserve Offline Baseline Separately
+
+The existing fixture-based evaluator remains the deterministic baseline. The new live acceptance flow is a smoke/regression companion, not a replacement.
+
+Rationale:
+
+- Offline reports are stable and CI-friendly.
+- Live reports prove environment readiness and post-reindex behavior.
+- Keeping both avoids mixing deterministic fixture checks with external-service validation.
+
+## Risks / Trade-offs
+
+- [Risk] Live results vary by environment, indexed documents, and retrieval mode. -> Mitigation: report the base URL, query set, result count, top candidates, score labels, and timestamp.
+- [Risk] A developer may run live validation before reindexing. -> Mitigation: document the prerequisite clearly and include a report note.
+- [Risk] The script could be mistaken for a benchmark. -> Mitigation: position it as acceptance smoke coverage; keep offline baseline for deterministic metrics.
@@ -0,0 +1,26 @@
+## Why
+
+`title` and `breadcrumb` now participate in embedding text, but that improvement only affects newly indexed vectors. We need a repeatable acceptance path that tells us how to reindex the knowledge base and verify live retrieval after the embedding input changes.
+
+## What Changes
+
+- Add a live RAG retrieval acceptance flow for breadcrumb-aware embedding changes.
+- Document the reindex prerequisite so reviewers understand old vectors do not change automatically.
+- Provide a small repeatable script for calling live retrieval cases and writing JSON/Markdown reports.
+- Add interview-facing acceptance notes that explain what was verified and what remains manual or environment-dependent.
+
+## Capabilities
+
+### New Capabilities
+
+None.
+
+### Modified Capabilities
+
+- `rag-retrieval-evaluation`: Extend retrieval evaluation with an opt-in live acceptance flow for post-reindex verification.
+
+## Impact
+
+- Adds scripts and documentation under the retrieval evaluation/interview areas.
+- Does not change the Agent runtime path, `lookup_knowledge`, VectorStore search logic, or Milvus schema.
+- Live verification depends on a running Spring Boot service and a reindexed Milvus/Zilliz collection.
@@ -0,0 +1,21 @@
+## ADDED Requirements
+
+### Requirement: Retrieval evaluation SHALL provide live post-reindex acceptance
+The retrieval evaluation system SHALL provide an opt-in live acceptance flow for validating retrieval behavior after embedding input changes require a knowledge-base reindex.
+
+#### Scenario: Live acceptance requires a running service
+- **WHEN** live retrieval acceptance is run
+- **THEN** it SHALL call the configured Spring Boot retrieval endpoint
+- **AND** it SHALL not be required by the offline fixture baseline
+
+#### Scenario: Live acceptance records retrieval evidence
+- **WHEN** a live retrieval case is executed
+- **THEN** the report SHALL include the query, requested topK, result count, top candidate titles or sources, score labels, and raw response fields needed for review
+
+#### Scenario: Reindex prerequisite is documented
+- **WHEN** a developer prepares to validate breadcrumb-aware embedding behavior
+- **THEN** the repository SHALL explain that existing vectors must be reindexed before live validation can reflect the new embedding text
+
+#### Scenario: Live report is reviewable
+- **WHEN** the live acceptance script completes
+- **THEN** it SHALL write JSON and Markdown outputs that can be inspected or attached to interview evidence
@@ -0,0 +1,14 @@
+## 1. Live Acceptance Tooling
+
+- [x] 1.1 Add a script that runs representative live `/api/search/similar` queries and writes JSON/Markdown reports.
+- [x] 1.2 Include breadcrumb-sensitive and core troubleshooting cases in the default live query set.
+
+## 2. Documentation
+
+- [x] 2.1 Document the post-reindex validation flow under `eval/rag-retrieval`.
+- [x] 2.2 Add interview-facing acceptance notes for breadcrumb-aware embedding validation.
+
+## 3. Verification
+
+- [x] 3.1 Run targeted tests or syntax checks for the new script.
+- [x] 3.2 Validate the OpenSpec change and confirm the working tree only contains expected files.