commit
This commit is contained in:
zhuyongxin
@@ -0,0 +1,83 @@
|
||||
---
|
||||
name: gitnexus-cli
|
||||
description: "Use when the user needs to run GitNexus CLI commands like analyze/index a repo, check status, clean the index, generate a wiki, or list indexed repos. Examples: \"Index this repo\", \"Reanalyze the codebase\", \"Generate a wiki\""
|
||||
---
|
||||
|
||||
# GitNexus CLI Commands
|
||||
|
||||
All commands work via `npx` — no global install required.
|
||||
|
||||
## Commands
|
||||
|
||||
### analyze — Build or refresh the index
|
||||
|
||||
```bash
|
||||
npx gitnexus analyze
|
||||
```
|
||||
|
||||
Run from the project root. This parses all source files, builds the knowledge graph, writes it to `.gitnexus/`, and generates CLAUDE.md / AGENTS.md context files.
|
||||
|
||||
| Flag | Effect |
|
||||
| -------------- | ---------------------------------------------------------------- |
|
||||
| `--force` | Force full re-index even if up to date |
|
||||
| `--embeddings` | Enable embedding generation for semantic search (off by default) |
|
||||
| `--drop-embeddings` | Drop existing embeddings on rebuild. By default, an `analyze` without `--embeddings` preserves them. |
|
||||
|
||||
**When to run:** First time in a project, after major code changes, or when `gitnexus://repo/{name}/context` reports the index is stale. In Claude Code, a PostToolUse hook detects staleness after `git commit` and `git merge` and notifies the agent to run `analyze` — the hook does not run analyze itself, to avoid blocking the agent for up to 120s and risking KuzuDB corruption on timeout.
|
||||
|
||||
### status — Check index freshness
|
||||
|
||||
```bash
|
||||
npx gitnexus status
|
||||
```
|
||||
|
||||
Shows whether the current repo has a GitNexus index, when it was last updated, and symbol/relationship counts. Use this to check if re-indexing is needed.
|
||||
|
||||
### clean — Delete the index
|
||||
|
||||
```bash
|
||||
npx gitnexus clean
|
||||
```
|
||||
|
||||
Deletes the `.gitnexus/` directory and unregisters the repo from the global registry. Use before re-indexing if the index is corrupt or after removing GitNexus from a project.
|
||||
|
||||
| Flag | Effect |
|
||||
| --------- | ------------------------------------------------- |
|
||||
| `--force` | Skip confirmation prompt |
|
||||
| `--all` | Clean all indexed repos, not just the current one |
|
||||
|
||||
### wiki — Generate documentation from the graph
|
||||
|
||||
```bash
|
||||
npx gitnexus wiki
|
||||
```
|
||||
|
||||
Generates repository documentation from the knowledge graph using an LLM. Requires an API key (saved to `~/.gitnexus/config.json` on first use).
|
||||
|
||||
| Flag | Effect |
|
||||
| ------------------- | ----------------------------------------- |
|
||||
| `--force` | Force full regeneration |
|
||||
| `--model <model>` | LLM model (default: minimax/minimax-m2.5) |
|
||||
| `--base-url <url>` | LLM API base URL |
|
||||
| `--api-key <key>` | LLM API key |
|
||||
| `--concurrency <n>` | Parallel LLM calls (default: 3) |
|
||||
| `--gist` | Publish wiki as a public GitHub Gist |
|
||||
|
||||
### list — Show all indexed repos
|
||||
|
||||
```bash
|
||||
npx gitnexus list
|
||||
```
|
||||
|
||||
Lists all repositories registered in `~/.gitnexus/registry.json`. The MCP `list_repos` tool provides the same information.
|
||||
|
||||
## After Indexing
|
||||
|
||||
1. **Read `gitnexus://repo/{name}/context`** to verify the index loaded
|
||||
2. Use the other GitNexus skills (`exploring`, `debugging`, `impact-analysis`, `refactoring`) for your task
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **"Not inside a git repository"**: Run from a directory inside a git repo
|
||||
- **Index is stale after re-analyzing**: Restart Claude Code to reload the MCP server
|
||||
- **Embeddings slow**: Omit `--embeddings` (it's off by default) or set `OPENAI_API_KEY` for faster API-based embedding
|
||||
@@ -0,0 +1,89 @@
|
||||
---
|
||||
name: gitnexus-debugging
|
||||
description: "Use when the user is debugging a bug, tracing an error, or asking why something fails. Examples: \"Why is X failing?\", \"Where does this error come from?\", \"Trace this bug\""
|
||||
---
|
||||
|
||||
# Debugging with GitNexus
|
||||
|
||||
## When to Use
|
||||
|
||||
- "Why is this function failing?"
|
||||
- "Trace where this error comes from"
|
||||
- "Who calls this method?"
|
||||
- "This endpoint returns 500"
|
||||
- Investigating bugs, errors, or unexpected behavior
|
||||
|
||||
## Workflow
|
||||
|
||||
```
|
||||
1. gitnexus_query({query: "<error or symptom>"}) → Find related execution flows
|
||||
2. gitnexus_context({name: "<suspect>"}) → See callers/callees/processes
|
||||
3. READ gitnexus://repo/{name}/process/{name} → Trace execution flow
|
||||
4. gitnexus_cypher({query: "MATCH path..."}) → Custom traces if needed
|
||||
```
|
||||
|
||||
> If "Index is stale" → run `npx gitnexus analyze` in terminal.
|
||||
|
||||
## Checklist
|
||||
|
||||
```
|
||||
- [ ] Understand the symptom (error message, unexpected behavior)
|
||||
- [ ] gitnexus_query for error text or related code
|
||||
- [ ] Identify the suspect function from returned processes
|
||||
- [ ] gitnexus_context to see callers and callees
|
||||
- [ ] Trace execution flow via process resource if applicable
|
||||
- [ ] gitnexus_cypher for custom call chain traces if needed
|
||||
- [ ] Read source files to confirm root cause
|
||||
```
|
||||
|
||||
## Debugging Patterns
|
||||
|
||||
| Symptom | GitNexus Approach |
|
||||
| -------------------- | ---------------------------------------------------------- |
|
||||
| Error message | `gitnexus_query` for error text → `context` on throw sites |
|
||||
| Wrong return value | `context` on the function → trace callees for data flow |
|
||||
| Intermittent failure | `context` → look for external calls, async deps |
|
||||
| Performance issue | `context` → find symbols with many callers (hot paths) |
|
||||
| Recent regression | `detect_changes` to see what your changes affect |
|
||||
|
||||
## Tools
|
||||
|
||||
**gitnexus_query** — find code related to error:
|
||||
|
||||
```
|
||||
gitnexus_query({query: "payment validation error"})
|
||||
→ Processes: CheckoutFlow, ErrorHandling
|
||||
→ Symbols: validatePayment, handlePaymentError, PaymentException
|
||||
```
|
||||
|
||||
**gitnexus_context** — full context for a suspect:
|
||||
|
||||
```
|
||||
gitnexus_context({name: "validatePayment"})
|
||||
→ Incoming calls: processCheckout, webhookHandler
|
||||
→ Outgoing calls: verifyCard, fetchRates (external API!)
|
||||
→ Processes: CheckoutFlow (step 3/7)
|
||||
```
|
||||
|
||||
**gitnexus_cypher** — custom call chain traces:
|
||||
|
||||
```cypher
|
||||
MATCH path = (a)-[:CodeRelation {type: 'CALLS'}*1..2]->(b:Function {name: "validatePayment"})
|
||||
RETURN [n IN nodes(path) | n.name] AS chain
|
||||
```
|
||||
|
||||
## Example: "Payment endpoint returns 500 intermittently"
|
||||
|
||||
```
|
||||
1. gitnexus_query({query: "payment error handling"})
|
||||
→ Processes: CheckoutFlow, ErrorHandling
|
||||
→ Symbols: validatePayment, handlePaymentError
|
||||
|
||||
2. gitnexus_context({name: "validatePayment"})
|
||||
→ Outgoing calls: verifyCard, fetchRates (external API!)
|
||||
|
||||
3. READ gitnexus://repo/my-app/process/CheckoutFlow
|
||||
→ Step 3: validatePayment → calls fetchRates (external)
|
||||
|
||||
4. Root cause: fetchRates calls external API without proper timeout
|
||||
```
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
name: gitnexus-exploring
|
||||
description: "Use when the user asks how code works, wants to understand architecture, trace execution flows, or explore unfamiliar parts of the codebase. Examples: \"How does X work?\", \"What calls this function?\", \"Show me the auth flow\""
|
||||
---
|
||||
|
||||
# Exploring Codebases with GitNexus
|
||||
|
||||
## When to Use
|
||||
|
||||
- "How does authentication work?"
|
||||
- "What's the project structure?"
|
||||
- "Show me the main components"
|
||||
- "Where is the database logic?"
|
||||
- Understanding code you haven't seen before
|
||||
|
||||
## Workflow
|
||||
|
||||
```
|
||||
1. READ gitnexus://repos → Discover indexed repos
|
||||
2. READ gitnexus://repo/{name}/context → Codebase overview, check staleness
|
||||
3. gitnexus_query({query: "<what you want to understand>"}) → Find related execution flows
|
||||
4. gitnexus_context({name: "<symbol>"}) → Deep dive on specific symbol
|
||||
5. READ gitnexus://repo/{name}/process/{name} → Trace full execution flow
|
||||
```
|
||||
|
||||
> If step 2 says "Index is stale" → run `npx gitnexus analyze` in terminal.
|
||||
|
||||
## Checklist
|
||||
|
||||
```
|
||||
- [ ] READ gitnexus://repo/{name}/context
|
||||
- [ ] gitnexus_query for the concept you want to understand
|
||||
- [ ] Review returned processes (execution flows)
|
||||
- [ ] gitnexus_context on key symbols for callers/callees
|
||||
- [ ] READ process resource for full execution traces
|
||||
- [ ] Read source files for implementation details
|
||||
```
|
||||
|
||||
## Resources
|
||||
|
||||
| Resource | What you get |
|
||||
| --------------------------------------- | ------------------------------------------------------- |
|
||||
| `gitnexus://repo/{name}/context` | Stats, staleness warning (~150 tokens) |
|
||||
| `gitnexus://repo/{name}/clusters` | All functional areas with cohesion scores (~300 tokens) |
|
||||
| `gitnexus://repo/{name}/cluster/{name}` | Area members with file paths (~500 tokens) |
|
||||
| `gitnexus://repo/{name}/process/{name}` | Step-by-step execution trace (~200 tokens) |
|
||||
|
||||
## Tools
|
||||
|
||||
**gitnexus_query** — find execution flows related to a concept:
|
||||
|
||||
```
|
||||
gitnexus_query({query: "payment processing"})
|
||||
→ Processes: CheckoutFlow, RefundFlow, WebhookHandler
|
||||
→ Symbols grouped by flow with file locations
|
||||
```
|
||||
|
||||
**gitnexus_context** — 360-degree view of a symbol:
|
||||
|
||||
```
|
||||
gitnexus_context({name: "validateUser"})
|
||||
→ Incoming calls: loginHandler, apiMiddleware
|
||||
→ Outgoing calls: checkToken, getUserById
|
||||
→ Processes: LoginFlow (step 2/5), TokenRefresh (step 1/3)
|
||||
```
|
||||
|
||||
## Example: "How does payment processing work?"
|
||||
|
||||
```
|
||||
1. READ gitnexus://repo/my-app/context → 918 symbols, 45 processes
|
||||
2. gitnexus_query({query: "payment processing"})
|
||||
→ CheckoutFlow: processPayment → validateCard → chargeStripe
|
||||
→ RefundFlow: initiateRefund → calculateRefund → processRefund
|
||||
3. gitnexus_context({name: "processPayment"})
|
||||
→ Incoming: checkoutHandler, webhookHandler
|
||||
→ Outgoing: validateCard, chargeStripe, saveTransaction
|
||||
4. Read src/payments/processor.ts for implementation details
|
||||
```
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
name: gitnexus-guide
|
||||
description: "Use when the user asks about GitNexus itself — available tools, how to query the knowledge graph, MCP resources, graph schema, or workflow reference. Examples: \"What GitNexus tools are available?\", \"How do I use GitNexus?\""
|
||||
---
|
||||
|
||||
# GitNexus Guide
|
||||
|
||||
Quick reference for all GitNexus MCP tools, resources, and the knowledge graph schema.
|
||||
|
||||
## Always Start Here
|
||||
|
||||
For any task involving code understanding, debugging, impact analysis, or refactoring:
|
||||
|
||||
1. **Read `gitnexus://repo/{name}/context`** — codebase overview + check index freshness
|
||||
2. **Match your task to a skill below** and **read that skill file**
|
||||
3. **Follow the skill's workflow and checklist**
|
||||
|
||||
> If step 1 warns the index is stale, run `npx gitnexus analyze` in the terminal first.
|
||||
|
||||
## Skills
|
||||
|
||||
| Task | Skill to read |
|
||||
| -------------------------------------------- | ------------------- |
|
||||
| Understand architecture / "How does X work?" | `gitnexus-exploring` |
|
||||
| Blast radius / "What breaks if I change X?" | `gitnexus-impact-analysis` |
|
||||
| Trace bugs / "Why is X failing?" | `gitnexus-debugging` |
|
||||
| Rename / extract / split / refactor | `gitnexus-refactoring` |
|
||||
| Tools, resources, schema reference | `gitnexus-guide` (this file) |
|
||||
| Index, status, clean, wiki CLI commands | `gitnexus-cli` |
|
||||
|
||||
## Tools Reference
|
||||
|
||||
| Tool | What it gives you |
|
||||
| ---------------- | ------------------------------------------------------------------------ |
|
||||
| `query` | Process-grouped code intelligence — execution flows related to a concept |
|
||||
| `context` | 360-degree symbol view — categorized refs, processes it participates in |
|
||||
| `impact` | Symbol blast radius — what breaks at depth 1/2/3 with confidence |
|
||||
| `detect_changes` | Git-diff impact — what do your current changes affect |
|
||||
| `rename` | Multi-file coordinated rename with confidence-tagged edits |
|
||||
| `cypher` | Raw graph queries (read `gitnexus://repo/{name}/schema` first) |
|
||||
| `list_repos` | Discover indexed repos |
|
||||
|
||||
## Resources Reference
|
||||
|
||||
Lightweight reads (~100-500 tokens) for navigation:
|
||||
|
||||
| Resource | Content |
|
||||
| ---------------------------------------------- | ----------------------------------------- |
|
||||
| `gitnexus://repo/{name}/context` | Stats, staleness check |
|
||||
| `gitnexus://repo/{name}/clusters` | All functional areas with cohesion scores |
|
||||
| `gitnexus://repo/{name}/cluster/{clusterName}` | Area members |
|
||||
| `gitnexus://repo/{name}/processes` | All execution flows |
|
||||
| `gitnexus://repo/{name}/process/{processName}` | Step-by-step trace |
|
||||
| `gitnexus://repo/{name}/schema` | Graph schema for Cypher |
|
||||
|
||||
## Graph Schema
|
||||
|
||||
**Nodes:** File, Function, Class, Interface, Method, Community, Process
|
||||
**Edges (via CodeRelation.type):** CALLS, IMPORTS, EXTENDS, IMPLEMENTS, DEFINES, MEMBER_OF, STEP_IN_PROCESS
|
||||
|
||||
```cypher
|
||||
MATCH (caller)-[:CodeRelation {type: 'CALLS'}]->(f:Function {name: "myFunc"})
|
||||
RETURN caller.name, caller.filePath
|
||||
```
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
name: gitnexus-impact-analysis
|
||||
description: "Use when the user wants to know what will break if they change something, or needs safety analysis before editing code. Examples: \"Is it safe to change X?\", \"What depends on this?\", \"What will break?\""
|
||||
---
|
||||
|
||||
# Impact Analysis with GitNexus
|
||||
|
||||
## When to Use
|
||||
|
||||
- "Is it safe to change this function?"
|
||||
- "What will break if I modify X?"
|
||||
- "Show me the blast radius"
|
||||
- "Who uses this code?"
|
||||
- Before making non-trivial code changes
|
||||
- Before committing — to understand what your changes affect
|
||||
|
||||
## Workflow
|
||||
|
||||
```
|
||||
1. gitnexus_impact({target: "X", direction: "upstream"}) → What depends on this
|
||||
2. READ gitnexus://repo/{name}/processes → Check affected execution flows
|
||||
3. gitnexus_detect_changes() → Map current git changes to affected flows
|
||||
4. Assess risk and report to user
|
||||
```
|
||||
|
||||
> If "Index is stale" → run `npx gitnexus analyze` in terminal.
|
||||
|
||||
## Checklist
|
||||
|
||||
```
|
||||
- [ ] gitnexus_impact({target, direction: "upstream"}) to find dependents
|
||||
- [ ] Review d=1 items first (these WILL BREAK)
|
||||
- [ ] Check high-confidence (>0.8) dependencies
|
||||
- [ ] READ processes to check affected execution flows
|
||||
- [ ] gitnexus_detect_changes() for pre-commit check
|
||||
- [ ] Assess risk level and report to user
|
||||
```
|
||||
|
||||
## Understanding Output
|
||||
|
||||
| Depth | Risk Level | Meaning |
|
||||
| ----- | ---------------- | ------------------------ |
|
||||
| d=1 | **WILL BREAK** | Direct callers/importers |
|
||||
| d=2 | LIKELY AFFECTED | Indirect dependencies |
|
||||
| d=3 | MAY NEED TESTING | Transitive effects |
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Affected | Risk |
|
||||
| ------------------------------ | -------- |
|
||||
| <5 symbols, few processes | LOW |
|
||||
| 5-15 symbols, 2-5 processes | MEDIUM |
|
||||
| >15 symbols or many processes | HIGH |
|
||||
| Critical path (auth, payments) | CRITICAL |
|
||||
|
||||
## Tools
|
||||
|
||||
**gitnexus_impact** — the primary tool for symbol blast radius:
|
||||
|
||||
```
|
||||
gitnexus_impact({
|
||||
target: "validateUser",
|
||||
direction: "upstream",
|
||||
minConfidence: 0.8,
|
||||
maxDepth: 3
|
||||
})
|
||||
|
||||
→ d=1 (WILL BREAK):
|
||||
- loginHandler (src/auth/login.ts:42) [CALLS, 100%]
|
||||
- apiMiddleware (src/api/middleware.ts:15) [CALLS, 100%]
|
||||
|
||||
→ d=2 (LIKELY AFFECTED):
|
||||
- authRouter (src/routes/auth.ts:22) [CALLS, 95%]
|
||||
```
|
||||
|
||||
**gitnexus_detect_changes** — git-diff based impact analysis:
|
||||
|
||||
```
|
||||
gitnexus_detect_changes({scope: "staged"})
|
||||
|
||||
→ Changed: 5 symbols in 3 files
|
||||
→ Affected: LoginFlow, TokenRefresh, APIMiddlewarePipeline
|
||||
→ Risk: MEDIUM
|
||||
```
|
||||
|
||||
## Example: "What breaks if I change validateUser?"
|
||||
|
||||
```
|
||||
1. gitnexus_impact({target: "validateUser", direction: "upstream"})
|
||||
→ d=1: loginHandler, apiMiddleware (WILL BREAK)
|
||||
→ d=2: authRouter, sessionManager (LIKELY AFFECTED)
|
||||
|
||||
2. READ gitnexus://repo/my-app/processes
|
||||
→ LoginFlow and TokenRefresh touch validateUser
|
||||
|
||||
3. Risk: 2 direct callers, 2 processes = MEDIUM
|
||||
```
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
name: gitnexus-refactoring
|
||||
description: "Use when the user wants to rename, extract, split, move, or restructure code safely. Examples: \"Rename this function\", \"Extract this into a module\", \"Refactor this class\", \"Move this to a separate file\""
|
||||
---
|
||||
|
||||
# Refactoring with GitNexus
|
||||
|
||||
## When to Use
|
||||
|
||||
- "Rename this function safely"
|
||||
- "Extract this into a module"
|
||||
- "Split this service"
|
||||
- "Move this to a new file"
|
||||
- Any task involving renaming, extracting, splitting, or restructuring code
|
||||
|
||||
## Workflow
|
||||
|
||||
```
|
||||
1. gitnexus_impact({target: "X", direction: "upstream"}) → Map all dependents
|
||||
2. gitnexus_query({query: "X"}) → Find execution flows involving X
|
||||
3. gitnexus_context({name: "X"}) → See all incoming/outgoing refs
|
||||
4. Plan update order: interfaces → implementations → callers → tests
|
||||
```
|
||||
|
||||
> If "Index is stale" → run `npx gitnexus analyze` in terminal.
|
||||
|
||||
## Checklists
|
||||
|
||||
### Rename Symbol
|
||||
|
||||
```
|
||||
- [ ] gitnexus_rename({symbol_name: "oldName", new_name: "newName", dry_run: true}) — preview all edits
|
||||
- [ ] Review graph edits (high confidence) and ast_search edits (review carefully)
|
||||
- [ ] If satisfied: gitnexus_rename({..., dry_run: false}) — apply edits
|
||||
- [ ] gitnexus_detect_changes() — verify only expected files changed
|
||||
- [ ] Run tests for affected processes
|
||||
```
|
||||
|
||||
### Extract Module
|
||||
|
||||
```
|
||||
- [ ] gitnexus_context({name: target}) — see all incoming/outgoing refs
|
||||
- [ ] gitnexus_impact({target, direction: "upstream"}) — find all external callers
|
||||
- [ ] Define new module interface
|
||||
- [ ] Extract code, update imports
|
||||
- [ ] gitnexus_detect_changes() — verify affected scope
|
||||
- [ ] Run tests for affected processes
|
||||
```
|
||||
|
||||
### Split Function/Service
|
||||
|
||||
```
|
||||
- [ ] gitnexus_context({name: target}) — understand all callees
|
||||
- [ ] Group callees by responsibility
|
||||
- [ ] gitnexus_impact({target, direction: "upstream"}) — map callers to update
|
||||
- [ ] Create new functions/services
|
||||
- [ ] Update callers
|
||||
- [ ] gitnexus_detect_changes() — verify affected scope
|
||||
- [ ] Run tests for affected processes
|
||||
```
|
||||
|
||||
## Tools
|
||||
|
||||
**gitnexus_rename** — automated multi-file rename:
|
||||
|
||||
```
|
||||
gitnexus_rename({symbol_name: "validateUser", new_name: "authenticateUser", dry_run: true})
|
||||
→ 12 edits across 8 files
|
||||
→ 10 graph edits (high confidence), 2 ast_search edits (review)
|
||||
→ Changes: [{file_path, edits: [{line, old_text, new_text, confidence}]}]
|
||||
```
|
||||
|
||||
**gitnexus_impact** — map all dependents first:
|
||||
|
||||
```
|
||||
gitnexus_impact({target: "validateUser", direction: "upstream"})
|
||||
→ d=1: loginHandler, apiMiddleware, testUtils
|
||||
→ Affected Processes: LoginFlow, TokenRefresh
|
||||
```
|
||||
|
||||
**gitnexus_detect_changes** — verify your changes after refactoring:
|
||||
|
||||
```
|
||||
gitnexus_detect_changes({scope: "all"})
|
||||
→ Changed: 8 files, 12 symbols
|
||||
→ Affected processes: LoginFlow, TokenRefresh
|
||||
→ Risk: MEDIUM
|
||||
```
|
||||
|
||||
**gitnexus_cypher** — custom reference queries:
|
||||
|
||||
```cypher
|
||||
MATCH (caller)-[:CodeRelation {type: 'CALLS'}]->(f:Function {name: "validateUser"})
|
||||
RETURN caller.name, caller.filePath ORDER BY caller.filePath
|
||||
```
|
||||
|
||||
## Risk Rules
|
||||
|
||||
| Risk Factor | Mitigation |
|
||||
| ------------------- | ----------------------------------------- |
|
||||
| Many callers (>5) | Use gitnexus_rename for automated updates |
|
||||
| Cross-area refs | Use detect_changes after to verify scope |
|
||||
| String/dynamic refs | gitnexus_query to find them |
|
||||
| External/public API | Version and deprecate properly |
|
||||
|
||||
## Example: Rename `validateUser` to `authenticateUser`
|
||||
|
||||
```
|
||||
1. gitnexus_rename({symbol_name: "validateUser", new_name: "authenticateUser", dry_run: true})
|
||||
→ 12 edits: 10 graph (safe), 2 ast_search (review)
|
||||
→ Files: validator.ts, login.ts, middleware.ts, config.json...
|
||||
|
||||
2. Review ast_search edits (config.json: dynamic reference!)
|
||||
|
||||
3. gitnexus_rename({symbol_name: "validateUser", new_name: "authenticateUser", dry_run: false})
|
||||
→ Applied 12 edits across 8 files
|
||||
|
||||
4. gitnexus_detect_changes({scope: "all"})
|
||||
→ Affected: LoginFlow, TokenRefresh
|
||||
→ Risk: MEDIUM — run tests for these flows
|
||||
```
|
||||
@@ -0,0 +1,43 @@
|
||||
<!-- gitnexus:start -->
|
||||
# GitNexus — Code Intelligence
|
||||
|
||||
This project is indexed by GitNexus as **SuperBizAgent-java** (1262 symbols, 2537 relationships, 89 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely.
|
||||
|
||||
> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first.
|
||||
|
||||
## Always Do
|
||||
|
||||
- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `gitnexus_impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user.
|
||||
- **MUST run `gitnexus_detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows.
|
||||
- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits.
|
||||
- When exploring unfamiliar code, use `gitnexus_query({query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance.
|
||||
- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `gitnexus_context({name: "symbolName"})`.
|
||||
|
||||
## Never Do
|
||||
|
||||
- NEVER edit a function, class, or method without first running `gitnexus_impact` on it.
|
||||
- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis.
|
||||
- NEVER rename symbols with find-and-replace — use `gitnexus_rename` which understands the call graph.
|
||||
- NEVER commit changes without running `gitnexus_detect_changes()` to check affected scope.
|
||||
|
||||
## Resources
|
||||
|
||||
| Resource | Use for |
|
||||
|----------|---------|
|
||||
| `gitnexus://repo/SuperBizAgent-java/context` | Codebase overview, check index freshness |
|
||||
| `gitnexus://repo/SuperBizAgent-java/clusters` | All functional areas |
|
||||
| `gitnexus://repo/SuperBizAgent-java/processes` | All execution flows |
|
||||
| `gitnexus://repo/SuperBizAgent-java/process/{name}` | Step-by-step execution trace |
|
||||
|
||||
## CLI
|
||||
|
||||
| Task | Read this skill file |
|
||||
|------|---------------------|
|
||||
| Understand architecture / "How does X work?" | `.claude/skills/gitnexus/gitnexus-exploring/SKILL.md` |
|
||||
| Blast radius / "What breaks if I change X?" | `.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md` |
|
||||
| Trace bugs / "Why is X failing?" | `.claude/skills/gitnexus/gitnexus-debugging/SKILL.md` |
|
||||
| Rename / extract / split / refactor | `.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md` |
|
||||
| Tools, resources, schema reference | `.claude/skills/gitnexus/gitnexus-guide/SKILL.md` |
|
||||
| Index, status, clean, wiki CLI commands | `.claude/skills/gitnexus/gitnexus-cli/SKILL.md` |
|
||||
|
||||
<!-- gitnexus:end -->
|
||||
@@ -0,0 +1,43 @@
|
||||
<!-- gitnexus:start -->
|
||||
# GitNexus — Code Intelligence
|
||||
|
||||
This project is indexed by GitNexus as **SuperBizAgent-java** (1262 symbols, 2537 relationships, 89 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely.
|
||||
|
||||
> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first.
|
||||
|
||||
## Always Do
|
||||
|
||||
- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `gitnexus_impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user.
|
||||
- **MUST run `gitnexus_detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows.
|
||||
- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits.
|
||||
- When exploring unfamiliar code, use `gitnexus_query({query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance.
|
||||
- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `gitnexus_context({name: "symbolName"})`.
|
||||
|
||||
## Never Do
|
||||
|
||||
- NEVER edit a function, class, or method without first running `gitnexus_impact` on it.
|
||||
- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis.
|
||||
- NEVER rename symbols with find-and-replace — use `gitnexus_rename` which understands the call graph.
|
||||
- NEVER commit changes without running `gitnexus_detect_changes()` to check affected scope.
|
||||
|
||||
## Resources
|
||||
|
||||
| Resource | Use for |
|
||||
|----------|---------|
|
||||
| `gitnexus://repo/SuperBizAgent-java/context` | Codebase overview, check index freshness |
|
||||
| `gitnexus://repo/SuperBizAgent-java/clusters` | All functional areas |
|
||||
| `gitnexus://repo/SuperBizAgent-java/processes` | All execution flows |
|
||||
| `gitnexus://repo/SuperBizAgent-java/process/{name}` | Step-by-step execution trace |
|
||||
|
||||
## CLI
|
||||
|
||||
| Task | Read this skill file |
|
||||
|------|---------------------|
|
||||
| Understand architecture / "How does X work?" | `.claude/skills/gitnexus/gitnexus-exploring/SKILL.md` |
|
||||
| Blast radius / "What breaks if I change X?" | `.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md` |
|
||||
| Trace bugs / "Why is X failing?" | `.claude/skills/gitnexus/gitnexus-debugging/SKILL.md` |
|
||||
| Rename / extract / split / refactor | `.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md` |
|
||||
| Tools, resources, schema reference | `.claude/skills/gitnexus/gitnexus-guide/SKILL.md` |
|
||||
| Index, status, clean, wiki CLI commands | `.claude/skills/gitnexus/gitnexus-cli/SKILL.md` |
|
||||
|
||||
<!-- gitnexus:end -->
|
||||
@@ -0,0 +1,29 @@
|
||||
# 上下文词汇表
|
||||
|
||||
## 术语
|
||||
|
||||
### ChatModel
|
||||
- 定义：Spring AI 的聊天模型抽象接口，所有 LLM 提供商（DashScope、OpenAI、Ollama 等）都实现此接口
|
||||
- 使用场景：所有需要 LLM 推理/生成回答的代码应面向此接口编程
|
||||
|
||||
### EmbeddingModel
|
||||
- 定义：Spring AI 的文本向量化抽象接口，将文本转换为向量
|
||||
- 使用场景：RAG 流程中将文档文本转为向量存入 Milvus
|
||||
|
||||
### DashScopeChatModel
|
||||
- 定义：DashScope（阿里云）对 ChatModel 的具体实现
|
||||
- 使用场景：当前项目硬编码使用，需要改为通过 ChatModel 接口引用
|
||||
|
||||
### ReactAgent
|
||||
- 定义：Spring AI Alibaba Agent Framework 的反应式 Agent 实现
|
||||
- 使用场景：Planner-Executor-Replanner 多 Agent 协作
|
||||
|
||||
### Spring AI Alibaba Agent Framework
|
||||
- 定义：基于 Spring AI 的多 Agent 协作框架，提供 ReactAgent、PlannerAgent、ExecutorAgent 等
|
||||
- 使用场景：项目核心 Agent 逻辑，ReactAgent.builder().model() 接受 ChatModel 接口
|
||||
|
||||
## 业务规则
|
||||
|
||||
- ChatModel 是唯一 LLM 调用抽象：替换模型只需更换 Spring Boot starter 和配置
|
||||
- EmbeddingModel 是唯一向量化抽象：替换向量模型只需更换 starter 和配置
|
||||
- ReactAgent 已兼容 ChatModel 接口，不绑定 DashScope
|
||||
@@ -0,0 +1,7 @@
|
||||
# devflow 索引
|
||||
|
||||
## 项目
|
||||
|
||||
| 日期 | slug | 领域 | 关键词 | 状态 |
|
||||
|---|---|---|---|---|
|
||||
| 2026-05-29 | chatmodel-abstraction | 解耦 | ChatModel, EmbeddingModel, DashScope, Spring AI | 进行中 |
|
||||
@@ -0,0 +1,42 @@
|
||||
# ChatModel Abstraction Decisions
|
||||
|
||||
## Question Pool
|
||||
|
||||
| # | 维度 | 问题 | 模式 | 状态 |
|
||||
|---|---|---|---|---|
|
||||
| Q1 | 术语 | ChatModel 注入方式：Spring Boot 自动注入 vs 手动工厂创建 | evidence-driven | 已解决 |
|
||||
| Q2 | 边界 | RagService 流式对话：Spring AI ChatModel.stream() 替代 DashScope Generation | evidence-driven | 已解决 |
|
||||
| Q3 | 验收 | VECTOR_DIM 是否需要动态化 | user-interview | 已解决 |
|
||||
| Q4 | 边界 | VectorEmbeddingService 批量向量化：EmbeddingModel 支持批量调用 | evidence-driven | 已解决 |
|
||||
|
||||
## Evidence-driven
|
||||
|
||||
| 结论 | 证据来源 | 是否已汇报用户 |
|
||||
|---|---|---|
|
||||
| ReactAgent.builder().model() 接受 ChatModel 接口 | javap 反编译 | 已汇报 |
|
||||
| ChatModel 应通过 Spring Boot 自动注入 | DashScope starter 自动注册 ChatModel Bean | 已汇报 |
|
||||
| RagService 可用 ChatModel.stream() 替代 Generation | Spring AI 接口有 stream(Prompt) 返回 Flux | 已汇报 |
|
||||
| EmbeddingModel 支持批量调用 | EmbeddingModel.call(EmbeddingRequest) 接受多条文本 | 已汇报 |
|
||||
|
||||
## User-interview
|
||||
|
||||
| 问题原文 | 用户原话 | 确认状态 | OpenSpec 回写 |
|
||||
|---|---|---|---|
|
||||
| VECTOR_DIM 怎么处理？ | "配置文件动态化" | 已确认 | 已回写 proposal |
|
||||
|
||||
## 关键取舍
|
||||
|
||||
- 决策：本次只解耦不替换实现
|
||||
- 原因：先验证抽象层正确再换模型
|
||||
- 影响：代码改动不改变运行行为
|
||||
- 风险接受：用户同意先只做解耦
|
||||
- 决策：VECTOR_DIM 从配置文件读取
|
||||
- 原因：换模型时改 yml 即可
|
||||
- 影响：MilvusConstants.VECTOR_DIM 改为从 MilvusProperties 读取
|
||||
|
||||
## 架构审计
|
||||
|
||||
- 风险1：RagService 流式适配 — DashScope Generation 和 Spring AI ChatModel.stream() 返回结构不同，需验证 thinking/content 分离逻辑
|
||||
- 风险2：DashScopeConfig 通用性 — 硬编码 dashscope 配置键，换模型后需改为通用键
|
||||
- 风险3：ChatModel Bean 冲突 — 多 starter 并存时需 @Primary 或条件注解
|
||||
- 低风险/无风险：VectorEmbeddingService、MilvusClientFactory 直接替换无问题
|
||||
@@ -0,0 +1,342 @@
|
||||
# Essence Report: SuperBizAgent-java — RAG 切片流程
|
||||
|
||||
> **Lens:** mechanical
|
||||
> **Design analyzed:** 四层递进式文档分块算法——标题→章节→段落→句子边界的逐级切割策略
|
||||
> **Files examined:** 3 (`DocumentChunkService.java`, `DocumentChunkConfig.java`, `DocumentChunk.java`)
|
||||
> **Pattern:** Hierarchical Splitter with Sentence-Boundary-Aware Overlap
|
||||
> **Status:** complete
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Deep Dive
|
||||
|
||||
### 核心文件
|
||||
|
||||
| # | 文件 | 行数 | 角色 |
|
||||
|---|------|------|------|
|
||||
| 1 | `service/DocumentChunkService.java` | 229 | 分块引擎本身 |
|
||||
| 2 | `config/DocumentChunkConfig.java` | 32 | 参数契约 `maxSize=800, overlap=100` |
|
||||
| 3 | `dto/DocumentChunk.java` | 59 | 分块数据载体 |
|
||||
|
||||
### 完整调用链
|
||||
|
||||
```
|
||||
VectorIndexService.indexSingleFile()
|
||||
└─ chunkService.chunkDocument(content, filePath) [L35]
|
||||
│
|
||||
├─ splitByHeadings(content) [L44]
|
||||
│ ├─ 正则: ^(#{1,6})\s+(.+)$ [L65]
|
||||
│ ├─ 迭代 matcher.find() 找到每个标题位置
|
||||
│ ├─ 标题之间的内容 → Section(title, content, startIndex)
|
||||
│ └─ → List<Section>
|
||||
│
|
||||
└─ for each Section:
|
||||
└─ chunkSection(section, globalChunkIndex) [L49]
|
||||
│
|
||||
├─ if content.length() ≤ maxSize (800):
|
||||
│ └─ 直接作为一个分块 [L110-119]
|
||||
│
|
||||
├─ else (需要进一步切割):
|
||||
│ ├─ splitByParagraphs(content) [L124]
|
||||
│ │ └─ content.split("\n\n+") [L178]
|
||||
│ │
|
||||
│ ├─ for each paragraph: [L130-167]
|
||||
│ │ ├─ 当前缓冲区 + 新段落 ≤ maxSize? → 继续追加
|
||||
│ │ └─ 当前缓冲区 + 新段落 > maxSize? → 触发切分:
|
||||
│ │ ├─ 保存当前分块
|
||||
│ │ ├─ getOverlapText(当前分块内容) [L147]
|
||||
│ │ │ ├─ 取末尾 overlap(100) 字符
|
||||
│ │ │ ├─ 在重叠文本中找最后一个句子终止符
|
||||
│ │ │ │ max(lastIndexOf('。'), lastIndexOf('？'), lastIndexOf('！'))
|
||||
│ │ │ ├─ if 句子边界 > overlapSize/2 (50字符):
|
||||
│ │ │ │ └─ 从句子边界后截取（保证新块以完整句开头）
|
||||
│ │ │ └─ else:
|
||||
│ │ │ └─ 直接用 overlap 末尾截取
|
||||
│ │ └─ 新缓冲区 = 重叠文本 + 当前段落
|
||||
│ │
|
||||
│ └─ 最后一个分块: 保存缓冲区剩余内容
|
||||
│
|
||||
└─ → List<DocumentChunk>
|
||||
```
|
||||
|
||||
### 算法的四层递进结构
|
||||
|
||||
```
|
||||
第1层：标题分割
|
||||
输入："# CPU高负载\n内容...\n## 排查步骤\n内容..."
|
||||
输出：Section("CPU高负载", "内容..."), Section("排查步骤", "内容...")
|
||||
作用：保持文档结构，同一主题的内容不被拆散
|
||||
|
||||
第2层：容量判断
|
||||
if section.length() ≤ 800: 整个章节 = 一个分块
|
||||
else: 进入段落级切割
|
||||
作用：短章节保持完整，不破坏语义
|
||||
|
||||
第3层：段落边界切割
|
||||
输入：超长章节的全部段落
|
||||
算法：逐个追加段落到缓冲区，超过 maxSize 时触发一次切分
|
||||
作用：不在段落中间截断
|
||||
|
||||
第4层：重叠窗口 + 句子边界对齐
|
||||
输入：即将被切断的分块末尾
|
||||
算法：取末尾100字符 → 找最近的。？！ → 从该位置之后截取作为下一块的"种子"
|
||||
作用：相邻分块在语义上是"连续"的，检索时召回更完整
|
||||
```
|
||||
|
||||
### 架构图
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
DOC[/"原始文档"/] --> L1{"第1层: splitByHeadings()"}
|
||||
|
||||
L1 --> S1["Section 1<br/>title: CPU高负载<br/>content: ..."]
|
||||
L1 --> S2["Section 2<br/>title: 排查步骤<br/>content: ..."]
|
||||
L1 --> S3["Section N"]
|
||||
|
||||
S1 --> L2{"第2层: 容量判断"}
|
||||
S2 --> L2
|
||||
S3 --> L2
|
||||
|
||||
L2 -->|"≤800字符"| CHUNK["作为1个分块<br/>继承 title"]
|
||||
L2 -->|">800字符"| L3{"第3层: splitByParagraphs()<br/>在段落边界切分"}
|
||||
|
||||
L3 --> BUF["逐段追加到缓冲区"]
|
||||
BUF --> CHECK{"buf + para<br/>> maxSize?"}
|
||||
CHECK -->|否| APPEND["追加段落<br/>继续累积"]
|
||||
CHECK -->|是| L4{"第4层: getOverlapText()<br/>句子边界校准"}
|
||||
|
||||
APPEND --> CHECK
|
||||
|
||||
L4 --> FIND["在重叠区末尾100字符<br/>找最近的 。？！"]
|
||||
FIND --> EVAL{"句子边界位置<br/>> overlapSize/2?"}
|
||||
EVAL -->|是| ALIGN["从句号后截取<br/>保证新块以完整句开头"]
|
||||
EVAL -->|否| RAW["退回原始截取<br/>直接用末尾100字符"]
|
||||
|
||||
ALIGN --> SEED["种子 + 当前段落<br/>→ 新缓冲区"]
|
||||
RAW --> SEED
|
||||
SEED --> CHECK
|
||||
|
||||
CHUNK --> RESULT[/"List<DocumentChunk><br/>每个携带: content + title + startIndex + endIndex + chunkIndex"/]
|
||||
```
|
||||
|
||||
### 关键代码证据
|
||||
|
||||
#### 第1层——标题正则
|
||||
|
||||
```java
|
||||
// DocumentChunkService.java:65
|
||||
Pattern headingPattern = Pattern.compile("^(#{1,6})\\s+(.+)$", Pattern.MULTILINE);
|
||||
```
|
||||
|
||||
支持 H1-H6，`MULTILINE` 模式让 `^` 匹配行首而非仅字符串首。
|
||||
|
||||
#### 第2层——容量判断（短路）
|
||||
|
||||
```java
|
||||
// DocumentChunkService.java:110-119
|
||||
if (content.length() <= chunkConfig.getMaxSize()) {
|
||||
DocumentChunk chunk = new DocumentChunk(content, startIndex, endIndex, chunkIndex);
|
||||
chunk.setTitle(title);
|
||||
chunks.add(chunk);
|
||||
return chunks; // 直接返回，不进入段落切割
|
||||
}
|
||||
```
|
||||
|
||||
#### 第3层——段落级触发切分
|
||||
|
||||
```java
|
||||
// DocumentChunkService.java:132-148
|
||||
if (currentChunk.length() > 0 &&
|
||||
currentChunk.length() + paragraph.length() > chunkConfig.getMaxSize()) {
|
||||
// 触发切分：保存当前块
|
||||
String overlap = getOverlapText(chunkContent); // 提取重叠文本
|
||||
currentChunk = new StringBuilder(overlap); // 新块以重叠文本开头
|
||||
currentStartIndex = currentStartIndex + chunkContent.length() - overlap.length();
|
||||
}
|
||||
currentChunk.append(paragraph).append("\n\n"); // 继续追加
|
||||
```
|
||||
|
||||
#### 第4层——句子边界检测（核心巧思）
|
||||
|
||||
```java
|
||||
// DocumentChunkService.java:193-213
|
||||
private String getOverlapText(String text) {
|
||||
int overlapSize = Math.min(chunkConfig.getOverlap(), text.length());
|
||||
String overlap = text.substring(text.length() - overlapSize);
|
||||
|
||||
// 在重叠文本中找最近的句子终止符
|
||||
int lastSentenceEnd = Math.max(
|
||||
overlap.lastIndexOf('。'),
|
||||
Math.max(overlap.lastIndexOf('？'), overlap.lastIndexOf('！'))
|
||||
);
|
||||
|
||||
// 质量阈值：只有句子边界在重叠区后半段才采用
|
||||
if (lastSentenceEnd > overlapSize / 2) {
|
||||
return overlap.substring(lastSentenceEnd + 1).trim();
|
||||
}
|
||||
return overlap.trim(); // 退回普通重叠
|
||||
}
|
||||
```
|
||||
|
||||
`overlapSize / 2` 条件是一个**质量阈值**。如果最近的句子边界在重叠区的前半段（即离截断点太远），说明分块点本身就接近句子边界，不需要特殊处理。只有句子边界明显位于重叠区后半段时才调整——避免把半个句子作为新块的"种子"。
|
||||
|
||||
### 数据流契约
|
||||
|
||||
```
|
||||
chunkDocument(content, filePath)
|
||||
│
|
||||
│ IN: String content — 原始文档全文
|
||||
│ String filePath — 仅用于日志
|
||||
│
|
||||
│ INNER CLASS: Section
|
||||
│ String title — 所在标题（可为 null）
|
||||
│ String content — 标题下的所有文本
|
||||
│ int startIndex — 在原文档中的字符偏移
|
||||
│
|
||||
│ OUT: List<DocumentChunk>
|
||||
│ String content — 分块文本
|
||||
│ int startIndex — 在原文档中的起始位置
|
||||
│ int endIndex — 在原文档中的结束位置
|
||||
│ int chunkIndex — 分块序号 (0, 1, 2, ...)
|
||||
│ String title — 所属章节标题（继承自 Section）
|
||||
│
|
||||
└─ 消费者: VectorIndexService.indexSingleFile():142
|
||||
→ 遍历 chunks → embeddingService.generateEmbedding(chunk.content)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Extract Pattern
|
||||
|
||||
### 模式名：Hierarchical Splitter with Sentence-Boundary-Aware Overlap
|
||||
|
||||
**一句话：** 从粗到细逐级切割——先按文档结构（标题）分章，再按语义边界（段落）分块，最后在切分点用句子终止符校准重叠窗口。
|
||||
|
||||
### 问题
|
||||
|
||||
固定长度切割的典型失败场景：
|
||||
|
||||
```
|
||||
切在句子中间： "CPU使用率达到 95%，建议" | "立即重启相关服务"
|
||||
↑ 检索"CPU问题"时召回这块——后半句完全脱离上下文，LLM 误判
|
||||
|
||||
切在段落中间："## 排查步骤\n1. 查看监控\n2. 检" | "查日志\n3. 重启服务"
|
||||
↑ 步骤 2 被切断，Agent 拿着残缺的排查步骤执行操作
|
||||
```
|
||||
|
||||
### 替代方案对比
|
||||
|
||||
| 方案 | 切分依据 | 优势 | 劣势 |
|
||||
|------|----------|------|------|
|
||||
| **固定字符切割**（最简陋） | maxSize，不关心内容 | 实现简单 | 句子截断、丢失语义 |
|
||||
| **递归字符切割**（LangChain RecursiveTextSplitter） | `\n\n` → `\n` → ` ` → `` | 通用性好 | 不理解 Markdown 结构 |
|
||||
| **语义切割**（用 LLM 判断切点） | LLM 标注切分位置 | 理论上最优 | 慢、贵、不可预测 |
|
||||
| **本项目：层级式+句子校准** | 标题→段落→句子终止符 | 快速 + 保留文档结构 | 仅支持 Markdown，非标题文档退化为段落切割 |
|
||||
|
||||
### 为什么标题分割放在第一步？
|
||||
|
||||
```java
|
||||
// DocumentChunkService.java:43-44
|
||||
// 1. 首先尝试按标题分割（Markdown格式）
|
||||
List<Section> sections = splitByHeadings(content);
|
||||
```
|
||||
|
||||
看本项目的知识库文档就懂了：
|
||||
|
||||
```markdown
|
||||
# CPU高负载问题排查 ← 一个独立主题
|
||||
## 问题现象
|
||||
...
|
||||
## 排查步骤 ← 这些步骤必须完整才能被 Agent 执行
|
||||
1. 使用 top 命令确认 CPU 使用率最高的进程
|
||||
2. 检查对应服务的日志
|
||||
3. ...
|
||||
## 解决方案
|
||||
...
|
||||
|
||||
# 内存高负载问题排查 ← 另一个独立主题
|
||||
...
|
||||
```
|
||||
|
||||
如果把「CPU 排查步骤」和「内存排查步骤」混在一个分块里，Agent 查询"CPU 高"时会召回包含内存排查步骤的分块——噪声干扰判断。
|
||||
|
||||
标题优先分割 = **用文档作者自己标注的结构来界定语义边界**，比任何算法都准确。
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Migrate
|
||||
|
||||
### 可迁移性
|
||||
|
||||
这个切分策略**直接可用**于任何需要为 Markdown 文档建 RAG 的项目。三个参数全部可配置：
|
||||
|
||||
```yaml
|
||||
# application.yml — 按文档类型调整
|
||||
document:
|
||||
chunk:
|
||||
max-size: 800 # 短文档(API文档)可设500，长文档(周报)可设1200
|
||||
overlap: 100 # 800的12.5%，保持比例即可
|
||||
```
|
||||
|
||||
### Steal-it 示例（17 行）
|
||||
|
||||
```java
|
||||
/**
|
||||
* 四层递进分块：标题 → 章节 → 段落 → 句子校准
|
||||
* 依赖：maxSize / overlap 两个参数
|
||||
*/
|
||||
public List<Chunk> chunk(String doc) {
|
||||
List<Chunk> result = new ArrayList<>();
|
||||
int globalIdx = 0;
|
||||
|
||||
// 第1层：按标题分章
|
||||
for (Section sec : splitByHeadings(doc)) {
|
||||
if (sec.content.length() <= maxSize) {
|
||||
// 第2层：短章节直接作为一个分块
|
||||
result.add(new Chunk(sec.content, sec.title, globalIdx++));
|
||||
} else {
|
||||
// 第3层：超长章节在段落边界切分
|
||||
String overlap = "";
|
||||
for (String para : sec.content.split("\n\n+")) {
|
||||
String candidate = overlap + para;
|
||||
if (candidate.length() > maxSize && !overlap.isEmpty()) {
|
||||
result.add(new Chunk(overlap, sec.title, globalIdx++));
|
||||
overlap = tailOverlap(overlap); // 第4层：句子校准
|
||||
}
|
||||
overlap = (overlap.isEmpty() ? "" : overlap + "\n\n") + para;
|
||||
}
|
||||
if (!overlap.isEmpty()) result.add(new Chunk(overlap, sec.title, globalIdx++));
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
```
|
||||
|
||||
### 落地陷阱
|
||||
|
||||
| 陷阱 | 原因 | 规避 |
|
||||
|------|------|------|
|
||||
| **非 Markdown 文档退化为单块** | `splitByHeadings()` 找不到标题时整个文档作为一个 Section | L93-96：返回一个 Section，后续段落切割仍生效 |
|
||||
| **代码块内的 `#` 被误识别为标题** | 正则不区分代码块和正文 | 未规避——可加反引号检测 `` ``` `` |
|
||||
| **overlap=0 时句子校准无效** | `getOverlapText` 第一行 `Math.min(0, length)=0` 返回空串 | L194：直接返回空字符串，跳过校准 |
|
||||
| **单段落超过 maxSize 不做切割** | `splitByParagraphs` 后每个段落作为一个单位 | L132 条件要求 `currentChunk.length() > 0`，首段落即使超长也会被单独保存为一块 |
|
||||
|
||||
---
|
||||
|
||||
### Self-review
|
||||
|
||||
- [x] 设计真实——每层切割均有代码行号证据
|
||||
- [x] 深度足够——追溯到正则、条件分支、句子校准的数学逻辑
|
||||
- [x] 迁移示例 17 行——提取了四层递进的核心骨架
|
||||
- [x] 陷阱具体到代码行——非 Markdown 退化为单块(L93)、单段落超长不切割(L132)
|
||||
|
||||
```
|
||||
Essence Report: SuperBizAgent-java — RAG 切片流程
|
||||
Lens: mechanical
|
||||
Design analyzed: 四层递进式文档分块算法
|
||||
Files examined: 3
|
||||
Pattern: Hierarchical Splitter with Sentence-Boundary-Aware Overlap
|
||||
Migration: 17-line steal-it skeleton
|
||||
HTML generated: no
|
||||
Status: complete
|
||||
```
|
||||
@@ -0,0 +1,314 @@
|
||||
# Essence Report: SuperBizAgent-java — RAG 实现
|
||||
|
||||
> **Lens:** mechanical（机械论——结构、接口、数据流）
|
||||
> **Design analyzed:** RAG 管道——从文档上传到 Agent 辅助检索的完整写入/读取双路径
|
||||
> **Files examined:** 12
|
||||
> **Pattern:** Pipeline-as-Services + Agent-Mediated Retrieval
|
||||
> **Status:** complete
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: 定位 — 设计目标确认
|
||||
|
||||
来自 `/explore` 报告的「设计二：完整的 RAG 管道（5 级流水线）」。用户指定深入 RAG 实现部分。
|
||||
|
||||
涉及 12 个核心文件，跨越 controller → service → client → constant 四层。
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Deep Dive — 逐文件追踪
|
||||
|
||||
### 核心文件清单
|
||||
|
||||
| # | 文件 | 角色 | 暴露接口 |
|
||||
|---|------|------|----------|
|
||||
| 1 | `constant/MilvusConstants.java` | Schema 契约常量 | `VECTOR_DIM=1024`, `COLLECTION_NAME="biz"` |
|
||||
| 2 | `client/MilvusClientFactory.java` | 数据库初始化 | `createClient()` → 自动建表+建索引 |
|
||||
| 3 | `config/DocumentChunkConfig.java` | 分块参数 | `maxSize=800`, `overlap=100` |
|
||||
| 4 | `dto/DocumentChunk.java` | 分块实体 | `content`, `startIndex/endIndex`, `chunkIndex`, `title` |
|
||||
| 5 | `service/DocumentChunkService.java` | 智能分块器 | `chunkDocument(content, filePath)` → `List<DocumentChunk>` |
|
||||
| 6 | `service/VectorEmbeddingService.java` | 向量化网关 | `generateEmbedding(text)` → `List<Float>` (1024-dim) |
|
||||
| 7 | `service/VectorIndexService.java` | 写入管道编排 | `indexSingleFile(path)` → 读→删旧→分块→向量化→写 |
|
||||
| 8 | `service/VectorSearchService.java` | 语义检索 | `searchSimilarDocuments(query, topK)` → `List<SearchResult>` |
|
||||
| 9 | `service/RagService.java` | 全栈 RAG 问答 | `queryStream(question, history, callback)` → SSE流式 |
|
||||
| 10 | `agent/tool/InternalDocsTools.java` | Agent 工具桥 | `queryInternalDocs(query)` → JSON（仅检索，不生文） |
|
||||
| 11 | `controller/FileUploadController.java` | 写入入口 | `POST /api/upload` → 文件存储 + 自动索引 |
|
||||
| 12 | `controller/ChatController.java` | 读取入口 | `POST /api/chat(_stream)` → ReactAgent + 工具调用 |
|
||||
|
||||
### 完整的调用链（双路径）
|
||||
|
||||
#### 写入路径（索引管道）
|
||||
|
||||
```
|
||||
POST /api/upload
|
||||
└─ FileUploadController.upload() [L34]
|
||||
├─ Files.copy() → 保存文件到 uploadPath
|
||||
└─ VectorIndexService.indexSingleFile() [L124]
|
||||
├─ Files.readString() [L135]
|
||||
├─ deleteExistingData() [L173]
|
||||
│ └─ milvusClient.delete() [L198]
|
||||
│ expr: metadata["_source"] == "/path/to/file"
|
||||
├─ chunkService.chunkDocument() [L142]
|
||||
│ ├─ splitByHeadings() [L61]
|
||||
│ │ └─ 正则: ^(#{1,6})\s+(.+)$
|
||||
│ ├─ chunkSection() × N [L104]
|
||||
│ │ ├─ splitByParagraphs() [L174]
|
||||
│ │ └─ getOverlapText() [L193]
|
||||
│ └─ → List<DocumentChunk>
|
||||
└─ for each chunk: [L146]
|
||||
├─ embeddingService.generateEmbedding() [L76]
|
||||
│ └─ DashScope TextEmbedding API → List<Float>[1024]
|
||||
└─ insertToMilvus() [L255]
|
||||
└─ UUID(source+chunkIndex) + vector + content + metadata(JSON)
|
||||
```
|
||||
|
||||
#### 读取路径（Agent 中介检索）
|
||||
|
||||
```
|
||||
POST /api/chat_stream
|
||||
└─ ChatController.chatStream() [L143]
|
||||
└─ chatService.createReactAgent() [L183]
|
||||
└─ tools: [DateTimeTools, InternalDocsTools, QueryMetricsTools, QueryLogsTools]
|
||||
└─ agent.stream(question) [L189]
|
||||
└─ Agent 自主决策 → 调用 queryInternalDocs
|
||||
└─ InternalDocsTools.queryInternalDocs() [L53]
|
||||
└─ VectorSearchService.searchSimilarDocuments() [L42]
|
||||
├─ embeddingService.generateQueryVector() [L47]
|
||||
├─ milvusClient.search() [L51]
|
||||
│ └─ L2距离, IVF_FLAT, nprobe=10
|
||||
└─ → List<SearchResult>{id, content, score, metadata}
|
||||
└─ return JSON to Agent
|
||||
└─ Agent 融合检索结果 + LLM推理 → 最终回答
|
||||
```
|
||||
|
||||
### 架构图
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph 写入路径
|
||||
UPLOAD[POST /api/upload]
|
||||
FC[FileUploadController]
|
||||
VIS[VectorIndexService]
|
||||
DCS[DocumentChunkService]
|
||||
VES[VectorEmbeddingService]
|
||||
MV_W[(Milvus)]
|
||||
end
|
||||
|
||||
subgraph 读取路径
|
||||
CHAT[POST /api/chat_stream]
|
||||
CC[ChatController]
|
||||
AGENT[ReactAgent]
|
||||
IDT[InternalDocsTools<br/>@Tool注解]
|
||||
VSS[VectorSearchService]
|
||||
MV_R[(Milvus)]
|
||||
LLM[DashScope LLM]
|
||||
end
|
||||
|
||||
UPLOAD --> FC
|
||||
FC --> VIS
|
||||
VIS --> DCS --> VIS
|
||||
VIS --> VES --> VIS
|
||||
VIS --> MV_W
|
||||
|
||||
CHAT --> CC
|
||||
CC --> AGENT
|
||||
AGENT -->|自主决策调用| IDT
|
||||
IDT --> VSS
|
||||
VSS --> VES --> VSS
|
||||
VSS --> MV_R
|
||||
IDT -->|JSON结果| AGENT
|
||||
AGENT --> LLM
|
||||
LLM -->|SSE流式| CC
|
||||
```
|
||||
|
||||
### 关键设计决策（代码证据）
|
||||
|
||||
#### 1. 幂等上传——元数据驱动的去重策略
|
||||
|
||||
```java
|
||||
// VectorIndexService.java:138-139
|
||||
// 删除该文件的旧数据（如果存在）
|
||||
deleteExistingData(path.toString());
|
||||
```
|
||||
|
||||
`deleteExistingData()` (L173-215) 使用 `metadata["_source"] == filePath` 作为删除表达式。每次上传同一文件时，先清空旧向量再写入新数据，保证数据一致性。
|
||||
|
||||
#### 2. 路径标准化——跨平台一致性
|
||||
|
||||
```java
|
||||
// VectorIndexService.java:176-178
|
||||
Path path = Paths.get(filePath).normalize();
|
||||
String normalizedPath = path.toString().replace(File.separator, "/");
|
||||
```
|
||||
|
||||
Windows `\` 和 Unix `/` 统一为正斜杠，避免 Milvus 表达式解析错误。在 `deleteExistingData()` 和 `buildMetadata()` 中均有应用。
|
||||
|
||||
#### 3. 重叠窗口 + 句子边界感知
|
||||
|
||||
```java
|
||||
// DocumentChunkService.java:132-148
|
||||
if (currentChunk.length() > 0 &&
|
||||
currentChunk.length() + paragraph.length() > chunkConfig.getMaxSize()) {
|
||||
// 保存当前分片
|
||||
String overlap = getOverlapText(chunkContent); // 提取重叠文本
|
||||
currentChunk = new StringBuilder(overlap); // 新分片以重叠文本开头
|
||||
```
|
||||
|
||||
`getOverlapText()` (L193-213) 更进一步：在重叠文本中寻找句子边界（`。？！`），避免在句子中间截断。当句子边界超过 `overlapSize/2` 时才使用，否则退回原始重叠策略。
|
||||
|
||||
#### 4. 检索与生成分离
|
||||
|
||||
`InternalDocsTools.queryInternalDocs()` 只做检索，不做生成。它将搜索结果序列化为 JSON 返回给 Agent，由 Agent 的 LLM 自行判断如何使用这些信息。
|
||||
|
||||
```java
|
||||
// InternalDocsTools.java:68
|
||||
String resultJson = objectMapper.writeValueAsString(searchResults);
|
||||
return resultJson;
|
||||
```
|
||||
|
||||
对比 `RagService.queryStream()` 则完整执行「检索→构建上下文→LLM 生成」三步，是一个独立的全栈 RAG 备用路径。
|
||||
|
||||
#### 5. Milvus Schema 设计
|
||||
|
||||
```java
|
||||
// MilvusClientFactory.java:109-142
|
||||
// 四个字段:
|
||||
// id VarChar(256) 主键 — UUID(source + chunkIndex)
|
||||
// vector FloatVector(1024) — text-embedding-v4 输出
|
||||
// content VarChar(8192) — 分块后的文本内容
|
||||
// metadata JSON — {_source, _extension, _file_name, chunkIndex, totalChunks, title}
|
||||
// 索引: IVF_FLAT, L2距离, nlist=128
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Extract Pattern
|
||||
|
||||
### 设计模式：Pipeline-as-Services + Agent-Mediated Retrieval
|
||||
|
||||
**问题：** 如何将知识库文档转化为 AI Agent 可检索、可利用的语义记忆？
|
||||
|
||||
**传统方案的问题：**
|
||||
- 关键词检索：无法理解语义相似的查询
|
||||
- 硬编码 FAQ：无法应对未见过的问题
|
||||
- 直接向量检索 + 固定提示词：所有问题都触发检索，浪费资源
|
||||
|
||||
**本项目的方案：两阶段架构**
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────┐
|
||||
│ STAGE 1: 写入管道 (离线/上传时触发) │
|
||||
│ │
|
||||
│ 文档 ──→ 智能分块 ──→ 向量化 ──→ Milvus存储 │
|
||||
│ (标题+段落 (text-embedding (IVF_FLAT │
|
||||
│ 边界感知) -v4, 1024-dim) L2索引) │
|
||||
│ │
|
||||
│ 接口契约: │
|
||||
│ IN: File → OUT: N × (vector + content + meta) │
|
||||
└──────────────────────────────────────────────────┘
|
||||
|
||||
┌──────────────────────────────────────────────────┐
|
||||
│ STAGE 2: 读取管道 (Agent 决策时触发) │
|
||||
│ │
|
||||
│ 用户问题 ──→ Agent 思考 ──→ 决定查知识库 │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ 向量检索 (L2距离) ──→ Top-K 文档片段 │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ Agent 融合检索结果 + LLM推理 → 回答 │
|
||||
│ │
|
||||
│ 接口契约: │
|
||||
│ IN: query(自然语言) → OUT: JSON(检索结果) │
|
||||
│ Agent 自主决定: 是否调用 / 如何使用结果 │
|
||||
└──────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 接口契约（隐式——通过 Spring DI 实现）
|
||||
|
||||
| 契约 | 生产者 | 消费者 | 数据形状 |
|
||||
|------|--------|--------|----------|
|
||||
| `List<DocumentChunk>` | DocumentChunkService | VectorIndexService | `{content, startIndex, endIndex, chunkIndex, title}` |
|
||||
| `List<Float>[1024]` | VectorEmbeddingService | VectorIndexService, VectorSearchService | DashScope text-embedding-v4 输出 |
|
||||
| `List<SearchResult>` | VectorSearchService | InternalDocsTools, RagService | `{id, content, score, metadata}` |
|
||||
| `StreamCallback` | RagService | (外部调用者) | `{onSearchResults, onContentChunk, onComplete, onError}` |
|
||||
|
||||
### 替代方案对比
|
||||
|
||||
| 方案 | 本项目 | LangChain4j | 纯 DashScope API |
|
||||
|------|--------|--------------|-------------------|
|
||||
| 分块策略 | 标题感知 + 段落边界 + 句子重叠 | 多种内置 Splitter | 无，需自建 |
|
||||
| 向量库 | Milvus (IVF_FLAT) | 多后端支持 | 无 |
|
||||
| Agent 集成 | Spring AI @Tool 注解，Agent 自主决策 | AiServices + @Tool | 无 Agent 框架 |
|
||||
| 去重 | metadata["_source"] 匹配删除 | 需自定义 | 不适用 |
|
||||
|
||||
### 为什么选择这种设计？
|
||||
|
||||
1. **「检索」和「生成」分离**：`InternalDocsTools` 只返回检索结果，生成由 Agent 的 LLM 完成。Agent 可以选择**不使用**检索结果（如果检索质量不高），或者**交叉验证**多次检索的结果
|
||||
2. **工具化 RAG**：将 RAG 暴露为 Agent 工具而非独立 API，让 Agent 在合适的时机触发检索——而非对所有问题都做 RAG
|
||||
3. **5 个独立 Service**：每个阶段可单独替换。想换分块策略？只改 `DocumentChunkService`。想换向量库？只改 `VectorSearchService` + `VectorIndexService`
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Migrate — 可迁移的设计
|
||||
|
||||
### 可迁移性评估
|
||||
|
||||
这个 RAG 设计**高度可迁移**到任何需要「知识库 + AI Agent」的 Java 项目。核心依赖是 Spring AI 生态 + 一个向量数据库。
|
||||
|
||||
### Steal-it 示例（12 行）
|
||||
|
||||
```java
|
||||
// 核心思想：Pipeline-as-Services + Agent Tool Bridge
|
||||
// 以下骨架可直接用于任何 Spring Boot 项目
|
||||
|
||||
// 1. 分块器：语义感知分割
|
||||
public List<Chunk> chunk(String doc) {
|
||||
return splitByHeadings(doc).stream()
|
||||
.flatMap(s -> splitToFit(s, MAX_SIZE, OVERLAP))
|
||||
.toList();
|
||||
}
|
||||
|
||||
// 2. Agent 工具桥：检索但不生文
|
||||
@Component
|
||||
class KnowledgeBaseTool {
|
||||
@Tool(description = "搜索内部知识库获取相关信息")
|
||||
public String search(@ToolParam(description="查询内容") String query) {
|
||||
List<Float> qv = embedder.embed(query); // 向量化
|
||||
var results = vectorDB.search(qv, TOP_K); // 语义检索
|
||||
return toJson(results); // 返回给Agent
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 落地陷阱
|
||||
|
||||
| 陷阱 | 说明 | 本项目如何规避 |
|
||||
|------|------|----------------|
|
||||
| **路径分隔符不一致** | Windows `\` vs Unix `/` 导致 Milvus 表达式解析失败 | `VectorIndexService.java:177` 强制 `replace(File.separator, "/")` |
|
||||
| **重复上传污染数据** | 同一文件多次上传产生重复向量 | `VectorIndexService.java:138-139` delete-before-insert |
|
||||
| **分块边界截断语义** | 固定长度切割可能切断句子 | `DocumentChunkService.java:203-206` 在重叠区找句子边界 |
|
||||
| **Agent 未触发工具** | Agent 不知道何时该查知识库 | `InternalDocsTools.java:49-52` @Tool description 用英文详细描述触发场景 |
|
||||
| **向量维度不匹配** | embedding 模型输出维度与 Milvus schema 不一致 | `MilvusConstants.java:18` 集中管理 `VECTOR_DIM=1024` |
|
||||
| **API Key 未初始化** | 静态 Constants 被其他线程覆盖 | `VectorEmbeddingService.java:86-89` 每次调用前检查并修复 |
|
||||
|
||||
### Self-review
|
||||
|
||||
- [x] 设计真实存在 — 每个声明均有文件+行号证据
|
||||
- [x] 分析深度足够 — 完整追踪了写入/读取两条全路径
|
||||
- [x] 迁移示例≤20行 — 仅提取 Pipeline + Tool Bridge 骨架
|
||||
- [x] 陷阱具体 — 每个都有代码规避证据
|
||||
- [x] 可解释为什么优于替代方案 — Agent 自主决策 vs 强制 RAG
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
Essence Report: SuperBizAgent-java
|
||||
Lens: mechanical
|
||||
Design analyzed: RAG 管道 — Pipeline-as-Services + Agent-Mediated Retrieval
|
||||
Files examined: 12
|
||||
Pattern: Pipeline-as-Services + Agent Tool Bridge
|
||||
Migration: 12-line steal-it skeleton
|
||||
HTML generated: no
|
||||
Status: complete
|
||||
```
|
||||
@@ -0,0 +1,352 @@
|
||||
# Explore Report: SuperBizAgent-java
|
||||
|
||||
> 生成时间: 2026-04-30
|
||||
> Project type: **code repository**
|
||||
> Phases completed: 4/4
|
||||
> Diagram included: yes
|
||||
> Core designs: 3
|
||||
> Status: complete
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Positioning & Structure
|
||||
|
||||
### 这是什么项目
|
||||
|
||||
SuperBizAgent-java 是一个基于 **Spring AI + Alibaba DashScope (Qwen)** 的智能运维 AI Agent 平台。它将大语言模型、向量检索增强生成（RAG）和多智能体协作（Planner-Executor-Replanner）整合为一体，面向企业 IT 运维场景提供：
|
||||
|
||||
- **智能文档问答**：上传运维知识库文档（Markdown/TXT），通过 RAG 管道实现向量化检索 + LLM 流式生成回答
|
||||
- **告警分析自动化**：多 Agent 协作分析 Prometheus 告警，结合日志查询（腾讯云 CLS）和内部知识库，生成结构化的告警分析报告
|
||||
- **MCP 协议集成**：通过 Spring AI MCP Client 连接外部工具服务，扩展 Agent 能力边界
|
||||
|
||||
### 为什么值得研究
|
||||
|
||||
| 维度 | 价值 |
|
||||
|------|------|
|
||||
| **AI 框架落地** | Spring AI Alibaba 生态的完整实践——ReactAgent、SupervisorAgent、Tool 注册、流式对话 |
|
||||
| **多 Agent 协作** | 非玩具级的 Planner-Executor-Replanner 监督循环，实际解决告警分析这种开放性问题 |
|
||||
| **RAG 工程化** | 完整的文档分块→向量化→Milvus 存储→语义检索→流式生成的端到端管道 |
|
||||
| **MCP 协议** | 业界较早将 MCP (Model Context Protocol) 用于生产场景的 Java 案例 |
|
||||
|
||||
### 适合谁
|
||||
|
||||
- Spring Boot / Java 开发者学习 AI Agent 框架的落地模式
|
||||
- AIOps / SRE 工程师了解智能运维 Agent 的架构设计
|
||||
- 对 Spring AI Alibaba 生态感兴趣的技术决策者
|
||||
|
||||
### 项目规模
|
||||
|
||||
| 指标 | 数值 |
|
||||
|------|------|
|
||||
| Java 源文件 | ~25 个 |
|
||||
| 代码行数 | ~2500 行 |
|
||||
| API 端点 | 7 个 |
|
||||
| Agent 工具 | 4 个 |
|
||||
| 知识库文档 | 5 篇 |
|
||||
|
||||
### 技术栈
|
||||
|
||||
```
|
||||
应用层 Spring Boot 3.2 / Java 17
|
||||
AI 层 Spring AI Alibaba 1.1.0 / Qwen3-Max / text-embedding-v4
|
||||
存储层 Milvus 2.5 (向量库) / MinIO (对象存储)
|
||||
集成层 MCP Client (WebFlux SSE) / Prometheus
|
||||
部署 Docker Compose (Milvus + etcd + MinIO + Attu)
|
||||
```
|
||||
|
||||
### 与替代方案的对比
|
||||
|
||||
| 方案 | 优势 | 劣势 |
|
||||
|------|------|------|
|
||||
| 本项目 (Spring AI Alibaba) | 完整生态、国产模型、Java 原生 | 社区相对年轻 |
|
||||
| LangChain4j | 社区活跃、模型支持广 | 多 Agent 模式需自行构建 |
|
||||
| Python LangChain | 生态最丰富 | 非 Java 技术栈 |
|
||||
| 纯 DashScope API | 简单直接 | 缺乏 Agent 编排、工具调用框架 |
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Flow
|
||||
|
||||
### 架构总览
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph 前端
|
||||
WEB[Web UI<br/>index.html + app.js]
|
||||
end
|
||||
|
||||
subgraph 控制层
|
||||
CC[ChatController<br/>/api/chat /api/chat_stream]
|
||||
AO[AIOpsController<br/>/api/ai_ops]
|
||||
UP[FileUploadController<br/>/api/upload]
|
||||
HC[MilvusCheckController<br/>/milvus/health]
|
||||
end
|
||||
|
||||
subgraph 服务层
|
||||
CS[ChatService<br/>ReactAgent 编排]
|
||||
AIS[AiOpsService<br/>多Agent 协作]
|
||||
RS[RagService<br/>RAG 流式问答]
|
||||
VIS[VectorIndexService<br/>文件索引管道]
|
||||
VSS[VectorSearchService<br/>向量相似搜索]
|
||||
VES[VectorEmbeddingService<br/>文本向量化]
|
||||
DCS[DocumentChunkService<br/>智能文档分块]
|
||||
end
|
||||
|
||||
subgraph Agent工具
|
||||
DT[DateTimeTools]
|
||||
IDT[InternalDocsTools]
|
||||
QMT[QueryMetricsTools]
|
||||
QLT[QueryLogsTools]
|
||||
end
|
||||
|
||||
subgraph 外部服务
|
||||
DS[DashScope API<br/>Qwen3-Max / Embedding]
|
||||
MV[Milvus<br/>向量数据库]
|
||||
PM[Prometheus<br/>监控告警]
|
||||
CLS[腾讯云CLS<br/>MCP SSE]
|
||||
end
|
||||
|
||||
WEB --> CC
|
||||
WEB --> AO
|
||||
WEB --> UP
|
||||
WEB --> HC
|
||||
|
||||
CC --> CS
|
||||
CC --> RS
|
||||
AO --> AIS
|
||||
UP --> VIS
|
||||
|
||||
CS --> DT & IDT & QMT & QLT
|
||||
AIS --> DT & IDT & QMT & QLT
|
||||
|
||||
CS --> DS
|
||||
RS --> DS
|
||||
RS --> VSS
|
||||
VIS --> DCS --> VES --> MV
|
||||
VSS --> MV
|
||||
QMT --> PM
|
||||
QLT --> CLS
|
||||
|
||||
VES --> DS
|
||||
```
|
||||
|
||||
### 主要运行时流程
|
||||
|
||||
#### 流程 A：RAG 智能问答（文档→检索→生成）
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor User
|
||||
participant Ctrl as FileUploadController
|
||||
participant VIS as VectorIndexService
|
||||
participant DCS as DocumentChunkService
|
||||
participant VES as VectorEmbeddingService
|
||||
participant MV as Milvus
|
||||
participant RS as RagService
|
||||
participant DS as DashScope
|
||||
|
||||
Note over User,DS: === 索引阶段 ===
|
||||
User->>Ctrl: POST /api/upload (file.md)
|
||||
Ctrl->>VIS: indexSingleFile(file)
|
||||
VIS->>VIS: 删除旧向量(按source路径匹配)
|
||||
VIS->>DCS: chunkDocument(content)
|
||||
DCS-->>VIS: List<DocumentChunk>
|
||||
loop 每个分块
|
||||
VIS->>VES: generateEmbedding(chunk)
|
||||
VES->>DS: text-embedding-v4 API
|
||||
DS-->>VES: float[1024]
|
||||
VES-->>VIS: 向量
|
||||
end
|
||||
VIS->>MV: insert(向量 + 原文 + metadata)
|
||||
MV-->>VIS: OK
|
||||
|
||||
Note over User,DS: === 问答阶段 ===
|
||||
User->>Ctrl: POST /api/chat (question)
|
||||
Ctrl->>RS: generateAnswerStream(question)
|
||||
RS->>VES: 向量化问题
|
||||
VES->>DS: text-embedding-v4
|
||||
DS-->>RS: query_vector[1024]
|
||||
RS->>MV: search(query_vector, topK=3)
|
||||
MV-->>RS: 3条最相似文档片段
|
||||
RS->>DS: Generation API (提示词 + 上下文 + 问题)
|
||||
DS-->>User: SSE 流式生成回答
|
||||
```
|
||||
|
||||
#### 流程 B：AIOps 多 Agent 告警分析
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor User
|
||||
participant Ctrl as ChatController
|
||||
participant AIS as AiOpsService
|
||||
participant Sup as SupervisorAgent
|
||||
participant P as PlannerAgent
|
||||
participant E as ExecutorAgent
|
||||
participant Tools as Agent Tools
|
||||
participant DS as DashScope
|
||||
|
||||
User->>Ctrl: POST /api/ai_ops (告警信息)
|
||||
Ctrl->>AIS: executeAiOpsAnalysis(request)
|
||||
|
||||
Note over AIS, DS: 启动监督循环
|
||||
AIS->>Sup: 启动，传入 Planner + Executor
|
||||
|
||||
loop Planner-Executor-Replanner
|
||||
Sup->>P: 分析当前状态，决定下一步
|
||||
alt 需要制定/修订计划
|
||||
P-->>User: SSE: 📋 分析计划...
|
||||
else 需要执行步骤
|
||||
P-->>Sup: EXECUTE
|
||||
Sup->>E: 执行计划第一步
|
||||
E->>Tools: 调用工具收集证据
|
||||
Tools-->>E: 日志/告警/文档信息
|
||||
E-->>User: SSE: 🔍 执行结果...
|
||||
E-->>Sup: 反馈 + 证据
|
||||
Note over Sup: 将执行结果反馈给Planner
|
||||
else 分析完成
|
||||
P-->>Sup: FINISH
|
||||
end
|
||||
end
|
||||
|
||||
Sup-->>User: SSE: ✅ Markdown 告警分析报告
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Start Path
|
||||
|
||||
### 最小启动步骤
|
||||
|
||||
```bash
|
||||
# 1. 启动基础设施（Milvus + etcd + MinIO）
|
||||
cd D:\zhu\project\SuperBizAgent-java
|
||||
docker compose -f vector-database.yml up -d
|
||||
|
||||
# 2. 设置 API Key 环境变量
|
||||
export DASHSCOPE_API_KEY="your-dashscope-api-key"
|
||||
|
||||
# 3. 启动应用
|
||||
mvn spring-boot:run
|
||||
# 应用启动在 http://localhost:9900
|
||||
|
||||
# 4. 打开 Web 测试页面
|
||||
# http://localhost:9900/index.html
|
||||
```
|
||||
|
||||
### 学习起点
|
||||
|
||||
1. **第一入口**：`src/main/java/org/example/Main.java` — Spring Boot 启动类，了解组件扫描范围
|
||||
2. **核心对话**：`src/main/java/org/example/controller/ChatController.java` — 所有 API 端点定义，理解请求路由
|
||||
3. **Agent 编排**：`src/main/java/org/example/service/ChatService.java` — ReactAgent 如何注册工具、处理对话
|
||||
4. **多 Agent 协作**：`src/main/java/org/example/service/AiOpsService.java` — Planner-Executor-Replanner 模式完整实现
|
||||
5. **RAG 管道**：按 `VectorIndexService → DocumentChunkService → VectorEmbeddingService → RagService` 顺序阅读
|
||||
|
||||
### 建议的第一个修改
|
||||
|
||||
在 `QueryMetricsTools.java` 的 `queryPrometheusAlerts()` 方法中添加一个 Mock 数据，观察 Agent 如何将新的工具输出整合到对话中。修改后重新提问相关问题即可看到效果。
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Core Designs
|
||||
|
||||
### 设计一：Planner-Executor-Replanner 监督循环
|
||||
|
||||
**位置**：`src/main/java/org/example/service/AiOpsService.java`
|
||||
|
||||
**是什么**：一个三层多 Agent 协作模式，用监督者控制循环来解决开放性的告警分析问题。
|
||||
|
||||
```
|
||||
SupervisorAgent (监督者)
|
||||
├── PlannerAgent (规划者)
|
||||
│ └── 决策三个状态: PLAN → 制定/修订计划
|
||||
│ EXECUTE → 交给执行者
|
||||
│ FINISH → 输出最终报告
|
||||
└── ExecutorAgent (执行者)
|
||||
└── 执行计划中的第一步
|
||||
└── 调用工具获取真实数据
|
||||
└── 返回反馈给 Planner 重新规划
|
||||
```
|
||||
|
||||
**为什么重要**：
|
||||
- 不是简单的单次 Agent 调用，而是通过**循环反馈**逐步逼近准确分析
|
||||
- Planner 根据 Executor 返回的证据**动态调整计划**（即 Replan 机制）
|
||||
- 通过 SSE 将每一步的中间结果实时推送给前端，用户体验好
|
||||
- 工具调用是**实际的**：Prometheus 查询、日志搜索、知识库检索，不是 mock 玩具
|
||||
|
||||
**关键实现细节**：
|
||||
```java
|
||||
// SupervisorAgent 创建并传入子 Agent
|
||||
SupervisorAgent supervisor = SupervisorAgent.builder()
|
||||
.supervisorAgent(supervisor)
|
||||
.subAgents(plannerAgent, executorAgent)
|
||||
.build();
|
||||
```
|
||||
|
||||
### 设计二：完整的 RAG 管道（5 级流水线）
|
||||
|
||||
**位置**：`VectorIndexService` → `DocumentChunkService` → `VectorEmbeddingService` → `VectorSearchService` → `RagService`
|
||||
|
||||
**是什么**：从原始文档到流式问答输出的完整 RAG 管道，涉及 5 个松耦合的服务组件。
|
||||
|
||||
| 阶段 | 组件 | 关键技术点 |
|
||||
|------|------|------------|
|
||||
| 1. 智能分块 | DocumentChunkService | 按 Markdown 标题层级 + 段落边界分割，800 字符/块，100 字符重叠 |
|
||||
| 2. 向量化 | VectorEmbeddingService | DashScope text-embedding-v4，1024 维，支持批量 |
|
||||
| 3. 向量存储 | MilvusClientFactory | IVF_FLAT 索引，L2 距离，自动去重（按 source 路径） |
|
||||
| 4. 语义检索 | VectorSearchService | Top-K 配置化（default 3），返回原文 + 相似度分数 |
|
||||
| 5. 流式生成 | RagService | DashScope Generation API，SSE 流式输出，支持 system prompt |
|
||||
|
||||
**为什么重要**：
|
||||
- 每个阶段**独立可替换**——可以换分块策略、换向量库、换 LLM
|
||||
- **幂等上传**：同一文件重新上传时，先删除旧向量再写入，保证数据一致性
|
||||
- 分块策略考虑了 Markdown 的文档结构（标题层级），而不是简单的固定长度切割
|
||||
|
||||
### 设计三：工具即插即用的 Agent 工具系统
|
||||
|
||||
**位置**：`src/main/java/org/example/agent/tool/*.java`
|
||||
|
||||
**是什么**：基于 Spring AI `@Tool` 注解的工具系统，Agent 自动发现并可调用。
|
||||
|
||||
```java
|
||||
// 工具定义示例
|
||||
@Component
|
||||
public class DateTimeTools {
|
||||
@Tool(description = "获取当前日期和时间")
|
||||
public String getCurrentDateTime() { ... }
|
||||
}
|
||||
```
|
||||
|
||||
**核心设计决策**：
|
||||
|
||||
| 决策 | 做法 | 原因 |
|
||||
|------|------|------|
|
||||
| Mock 开关 | `QueryMetricsTools` 和 `QueryLogsTools` 都有 `mockEnabled` 配置 | 开发/演示时不需要真实 Prometheus/CLS 环境 |
|
||||
| MCP 优先 | 当 MCP Client 可用时，自动排除 `QueryLogsTools` | 避免工具重复，MCP 提供更丰富的日志能力 |
|
||||
| JSON Schema 生成 | 使用 `jsonschema-generator` 为工具参数生成 schema | 让 LLM 理解工具的参数类型和约束 |
|
||||
| 工具注册 | `ChatService` 和 `AiOpsService` 各自注册工具集 | Agent 只获得需要的能力，避免干扰 |
|
||||
|
||||
**ChatService 工具注册**：
|
||||
```java
|
||||
// 构建时注册所有可用工具
|
||||
ReactAgent agent = ReactAgent.builder()
|
||||
.tools(dateTimeTools, internalDocsTools,
|
||||
queryMetricsTools, queryLogsTools)
|
||||
.build();
|
||||
```
|
||||
|
||||
**为什么重要**：
|
||||
- Agent 工具系统是 AI Agent 的**能力边界**——定义了 Agent 能做什么
|
||||
- Mock/Real 模式切换体现了**开发友好性**
|
||||
- MCP 协议的集成展示了**可扩展性**——Agent 可以从外部获取新能力
|
||||
|
||||
---
|
||||
|
||||
## 总结
|
||||
|
||||
SuperBizAgent-java 是一个小而完整的 AI Agent 实践项目。它的三个核心竞争力是：
|
||||
|
||||
1. **多 Agent 协作**（Planner-Executor-Replanner）——不是玩具，是真正解决问题的模式
|
||||
2. **工程化的 RAG 管道**——5 级流水线、幂等上传、智能分块
|
||||
3. **Spring AI 生态的完整实践**——从 @Tool 注解到 MCP 协议，展示了 Java 生态做 AI Agent 的成熟路径
|
||||
|
||||
对于想将 AI Agent 引入企业运维场景的 Java 团队，这是一个很好的学习起点和脚手架。
|
||||
@@ -0,0 +1,282 @@
|
||||
# 当前分片策略问题分析与根因
|
||||
|
||||
> 基于 `DocumentChunkServiceTest` 可视化测试的运行结果
|
||||
> 配置：`maxSize=800, overlap=100`（默认） / 可视化测试使用 `maxSize=300/200, overlap=50/30`
|
||||
|
||||
---
|
||||
|
||||
## 问题总览
|
||||
|
||||
| # | 问题 | 严重程度 | 根因归类 |
|
||||
|---|------|----------|----------|
|
||||
| 1 | 标题独立成空壳块 | 中 | 标题分割逻辑 |
|
||||
| 2 | 有序列表被拆散 | 高 | 段落级切割 + 缺少结构感知 |
|
||||
| 3 | 英文块 token 密度远低于中文块 | 高 | 字符计数代替 token 计数 |
|
||||
| 4 | 硬截断点在语义转折处无特殊处理 | 中 | 仅依赖 maxSize 触发 |
|
||||
| 5 | overlap 窗口对中文句号后截取命中率低 | 低 | 句子校准逻辑覆盖不全 |
|
||||
|
||||
---
|
||||
|
||||
## 问题 1：标题独立成空壳块
|
||||
|
||||
### 现象
|
||||
|
||||
运维文档 `maxSize=300` 下，H1 标题产生了一个只有 14 字符的分块：
|
||||
|
||||
```
|
||||
Chunk #0
|
||||
│ Title: CPU高负载问题排查指南
|
||||
│ Range: [0→14] (14字符)
|
||||
│ Content:
|
||||
│ │ # CPU高负载问题排查指南
|
||||
```
|
||||
|
||||
紧随其后的 `## 问题现象` 被分到下一个块。14 字符的块没有任何可检索的实质内容。
|
||||
|
||||
### 根因
|
||||
|
||||
```java
|
||||
// DocumentChunkService.java:71-83
|
||||
while (matcher.find()) {
|
||||
// 保存上一个章节
|
||||
if (lastEnd < matcher.start()) {
|
||||
String sectionContent = content.substring(lastEnd, matcher.start()).trim();
|
||||
if (!sectionContent.isEmpty()) { // ← 条件：content 非空
|
||||
sections.add(new Section(currentTitle, sectionContent, lastEnd));
|
||||
}
|
||||
}
|
||||
currentTitle = matcher.group(2).trim();
|
||||
lastEnd = matcher.start();
|
||||
}
|
||||
```
|
||||
|
||||
`splitByHeadings()` 遍历标题时，`lastEnd` 指向当前标题起始位置，`matcher.start()` 是下一个标题的起始位置。当 H1 后紧跟 H2（中间只有 `#` 行本身的内容），`content.substring(lastEnd, matcher.start())` 取出的是 **H1 标题行本身 + H1 标题行和 H2 之间的空白**。
|
||||
|
||||
关键问题：
|
||||
- H1 标题行被当作上一个 section 的 "content" 保存（因为中间文本不为空——标题行本身是文本）
|
||||
- 但实质上标题不应该独立成为一个可检索的分块
|
||||
|
||||
### 影响
|
||||
|
||||
- 向量库中出现大量无效向量（仅含标题、无实质内容）
|
||||
- 检索时可能召回标题块，Agent 得不到有用信息
|
||||
- 浪费 Milvus 存储空间
|
||||
|
||||
---
|
||||
|
||||
## 问题 2：有序列表被拆散
|
||||
|
||||
### 现象
|
||||
|
||||
排查步骤 1-4 在 Chunk #2，第 5 步被单独踢到 Chunk #3：
|
||||
|
||||
```
|
||||
Chunk #2 → 1. 登录服务器... 2. 使用 ps... 3. 查看应用日志... 4. 检查数据库...
|
||||
Chunk #3 → 5. 检查JVM内存...
|
||||
```
|
||||
|
||||
Agent 调用工具拿到 Chunk #2 时，排查步骤不完整，可能漏掉关键操作。
|
||||
|
||||
### 根因
|
||||
|
||||
```java
|
||||
// DocumentChunkService.java:174-188
|
||||
private List<String> splitByParagraphs(String content) {
|
||||
List<String> paragraphs = new ArrayList<>();
|
||||
String[] parts = content.split("\n\n+"); // ← 双换行分割
|
||||
for (String part : parts) {
|
||||
String trimmed = part.trim();
|
||||
if (!trimmed.isEmpty()) {
|
||||
paragraphs.add(trimmed);
|
||||
}
|
||||
}
|
||||
return paragraphs;
|
||||
}
|
||||
```
|
||||
|
||||
```java
|
||||
// DocumentChunkService.java:132-148
|
||||
if (currentChunk.length() > 0 &&
|
||||
currentChunk.length() + paragraph.length() > chunkConfig.getMaxSize()) {
|
||||
// 触发切分——不关心这个段落属于什么语义结构
|
||||
String overlap = getOverlapText(chunkContent);
|
||||
currentChunk = new StringBuilder(overlap);
|
||||
}
|
||||
currentChunk.append(paragraph).append("\n\n");
|
||||
```
|
||||
|
||||
两层根因：
|
||||
1. `splitByParagraphs()` 只认 `\n\n+` 作为段落分割符，不识别 **有序列表**（`1. \n2. \n3.` 之间通常是单换行）
|
||||
2. `chunkSection()` 走到字符上限就切，完全不感知"这是一个列表的第几项"——列表项之间的语义强关联被忽略
|
||||
|
||||
### 影响
|
||||
|
||||
- 排查步骤、操作指南类文档的完整性被破坏
|
||||
- RAG 检索召回不完整的步骤列表，Agent 据此操作可能导致遗漏
|
||||
- 这是运维场景的致命问题——运维文档大量使用列表
|
||||
|
||||
---
|
||||
|
||||
## 问题 3：英文块 token 密度远低于中文块
|
||||
|
||||
### 现象
|
||||
|
||||
可视化测试数据：
|
||||
|
||||
```
|
||||
中文: 218字符 → 2个分块（约218 tokens，密度 ~1.0 token/字符）
|
||||
英文: 602字符 → 3个分块（约150 tokens，密度 ~0.25 token/字符）
|
||||
```
|
||||
|
||||
同样 `maxSize=200`，英文 602 字符装了 150 token 还产生 3 个分块；中文 218 字符装了 218 token 只产生 2 个分块。中文块的实际 token 负担是英文的 **~4x**。
|
||||
|
||||
### 根因
|
||||
|
||||
```java
|
||||
// DocumentChunkConfig.java:18
|
||||
private int maxSize = 800; // 字符数上限
|
||||
|
||||
// DocumentChunkService.java:132-133
|
||||
if (currentChunk.length() + paragraph.length() > chunkConfig.getMaxSize()) {
|
||||
// 这里比的是 Java String.length() — 字符数，不是 token 数
|
||||
```
|
||||
|
||||
Java 的 `String.length()` 对每个 Unicode 字符（包括中文）都返回 1。但 LLM tokenizer 对中文和英文的 token 化效率完全不同：
|
||||
|
||||
```
|
||||
"这是中文" → 4 字符 → ~4 tokens (1:1)
|
||||
"This is English" → 15 字符 → ~4 tokens (3.75:1)
|
||||
```
|
||||
|
||||
用字符数作为切割上限，相当于：
|
||||
- 中文块：可以装 800 token（甚至更多）
|
||||
- 英文块：只能装 ~200 token
|
||||
|
||||
LLM 上下文窗口是按 token 计费的，这种偏差意味着**中文知识库的 RAG 开销是英文的 4 倍**。
|
||||
|
||||
### 影响
|
||||
|
||||
- LLM 调用成本不可预测（中英混排时波动大）
|
||||
- 中文知识库的上下文窗口利用率极易超标
|
||||
- 无法对 prompt 的 token 预算做精确控制
|
||||
|
||||
---
|
||||
|
||||
## 问题 4：硬截断在语义转折处无特殊处理
|
||||
|
||||
### 现象
|
||||
|
||||
同问题 2 的根因延伸。当前逻辑：
|
||||
|
||||
```
|
||||
段落1 + 段落2 + 段落3 + ... + 段落N → 总字符数 < maxSize → 继续追加
|
||||
→ 总字符数 > maxSize → 立刻切
|
||||
```
|
||||
|
||||
不考虑「段落 N 和段落 N+1 是否属于同一语义单元」。两个语义上需要绑定的段落恰好越过 maxSize 边界就会被拆散。
|
||||
|
||||
### 根因
|
||||
|
||||
```java
|
||||
// DocumentChunkService.java:132
|
||||
if (currentChunk.length() + paragraph.length() > chunkConfig.getMaxSize()) {
|
||||
```
|
||||
|
||||
触发条件只有一个——字符数。不缺以下信号：
|
||||
- 相邻段落的语义相似度（可用 embedding 计算）
|
||||
- 当前缓冲区是否处于列表/表格/代码块内部
|
||||
- 当前位置是否是 Markdown 层级的自然边界（如 `##` 标题前）
|
||||
|
||||
### 影响
|
||||
|
||||
- 切出来的分块边界在语义上不可预测
|
||||
- 同一主题的内容可能跨越两个分块，召回时只能拿到一半上下文
|
||||
|
||||
---
|
||||
|
||||
## 问题 5：overlap 句子校准对中文覆盖不全
|
||||
|
||||
### 现象
|
||||
|
||||
测试用的中文句子边界校准场景中，文档字数不足 `maxSize=100`，未触发切分。但即便触发，当前校准逻辑存在盲区：
|
||||
|
||||
```java
|
||||
// DocumentChunkService.java:203-206
|
||||
int lastSentenceEnd = Math.max(
|
||||
overlap.lastIndexOf('。'), // 只有三个终止符
|
||||
Math.max(overlap.lastIndexOf('？'), overlap.lastIndexOf('！'))
|
||||
);
|
||||
```
|
||||
|
||||
### 根因
|
||||
|
||||
中文句子终止符不止 `。？！` 三种：
|
||||
|
||||
| 终止符 | 是否覆盖 | 遗漏场景 |
|
||||
|--------|----------|----------|
|
||||
| `。` | ✅ | — |
|
||||
| `？` | ✅ | — |
|
||||
| `！` | ✅ | — |
|
||||
| `；`（分号） | ❌ | 长复句的语义断点 |
|
||||
| `：`（冒号） | ❌ | 列表/说明的引入点 |
|
||||
| `……` | ❌ | 省略号表示语义未尽 |
|
||||
| `\n`（换行） | ❌ | 中文短句常用换行代替标点 |
|
||||
|
||||
阈值逻辑也有盲区：
|
||||
|
||||
```java
|
||||
if (lastSentenceEnd > overlapSize / 2) {
|
||||
// only apply if sentence boundary is in the LATER half of overlap
|
||||
}
|
||||
```
|
||||
|
||||
如果句子边界在重叠区的前半段（即离截断点不到 overlap/2），直接退回原始截取——但实际上即使在前半段，也比随机截取更好。
|
||||
|
||||
### 影响
|
||||
|
||||
- 中文内容的重叠窗口可能从句子中间截取
|
||||
- 新分块的"种子"文本不完整，影响该块的语义完整性
|
||||
|
||||
---
|
||||
|
||||
## 根因总结
|
||||
|
||||
所有 5 个问题的根源收敛到两点：
|
||||
|
||||
### 根因 A：切割触发器只有一个维度——字符数
|
||||
|
||||
```
|
||||
currentChunk.length() + paragraph.length() > maxSize → 切！
|
||||
```
|
||||
|
||||
这个条件不知道：
|
||||
- 这个"paragraph"是列表项还是普通段落？（问题 2）
|
||||
- 中文还是英文？（问题 3）
|
||||
- 和上一条内容语义紧密还是已经转移话题？（问题 4）
|
||||
- 这个位置是在 Markdown 结构树上的什么层级？（问题 1）
|
||||
|
||||
### 根因 B：文档结构感知仅限于正则标题
|
||||
|
||||
```java
|
||||
Pattern headingPattern = Pattern.compile("^(#{1,6})\\s+(.+)$", Pattern.MULTILINE);
|
||||
```
|
||||
|
||||
这是唯一的结构感知入口。正则比 AST 脆弱，无法区分：
|
||||
- 代码块内的 `#` 注释 vs 真正的 Markdown 标题
|
||||
- 列表项 vs 段落
|
||||
- 代码块 vs 正文
|
||||
- 表格 vs 正文
|
||||
|
||||
---
|
||||
|
||||
## 修复优先级建议
|
||||
|
||||
| 优先级 | 问题 | 对策 | 改动量 |
|
||||
|--------|------|------|--------|
|
||||
| P0 | 问题 3（中英 token 密度） | 字符计数 → token 计数 | ~10 行 |
|
||||
| P0 | 问题 2（列表拆散） | 增加列表结构感知 | ~30 行 |
|
||||
| P1 | 问题 1（标题空壳） | 标题与下一个 H2 之间内容为空时合并 | ~15 行 |
|
||||
| P1 | 问题 4（硬截断） | 语义相似度辅助决策切点 | ~30 行 |
|
||||
| P2 | 问题 5（句子校准覆盖） | 增加终止符 + 降低阈值条件 | ~5 行 |
|
||||
|
||||
最终方案：替换为 Spring AI `TokenTextSplitter`，同时保留本项目特有的 `title` 元数据传播能力（因为 `TokenTextSplitter` 也不感知 Markdown 标题）。
|
||||
@@ -0,0 +1,200 @@
|
||||
# Plan: 分片策略第 4 步重构
|
||||
|
||||
> 分支: `refactor/rag-chunking-strategy`
|
||||
> 状态: 规划中
|
||||
> 范围: 仅改 `DocumentChunkService.chunkSection()` 一个方法
|
||||
|
||||
---
|
||||
|
||||
## 背景
|
||||
|
||||
经 debug 确认，当前分片流程的 1/2/3 步逻辑正确：
|
||||
|
||||
```
|
||||
第1步 chunkDocument() → splitByHeadings(content) ✅ 保持不变
|
||||
第2步 for each Section → 循环章节 ✅ 保持不变
|
||||
第3步 chunkSection() 入口 → 容量短路判断 + splitByParagraphs ✅ 保持不变
|
||||
第4步 chunkSection() 累积循环 → 段落累积 + 字符触发切分 ❌ 需重构
|
||||
```
|
||||
|
||||
**第 4 步的两个核心问题：**
|
||||
|
||||
| 问题 | 现象 |
|
||||
|------|------|
|
||||
| A. 丢失顺序 | `trim()` + 手工拼接 `\n\n` 导致 `currentStartIndex` 漂移 |
|
||||
| B. 结构无感知 | 有序列表项被拆散到不同分块（排查步骤 1-4 在一块，第 5 步在另一块） |
|
||||
|
||||
---
|
||||
|
||||
## 目标
|
||||
|
||||
改造 `chunkSection()` 的段落累积循环，使其：
|
||||
|
||||
1. **不丢顺序** — 用原始文本索引替代手工拼装的 `currentStartIndex`
|
||||
2. **感知列表结构** — 有序/无序列表项之间不在中间切断
|
||||
3. **Token 感知** — 用启发式 token 估算替代纯字符计数（为后续 Spring AI TokenTextSplitter 做准备）
|
||||
4. **软边界** — 在接近上限时查找语义安全切点，而非硬截断
|
||||
|
||||
---
|
||||
|
||||
## 不改的部分
|
||||
|
||||
| 组件 | 理由 |
|
||||
|------|------|
|
||||
| `splitByHeadings()` | 标题分割正确，正则够用 |
|
||||
| `getOverlapText()` | 句子校准逻辑保留，作为安全网 |
|
||||
| `DocumentChunk` 数据结构 | 字段完备，无需新增 |
|
||||
| `DocumentChunkConfig` | 增加 `maxTokens` 字段，保留原字段兼容 |
|
||||
| `VectorIndexService` | 消费者改动延后到下一阶段 |
|
||||
|
||||
---
|
||||
|
||||
## 改动方案
|
||||
|
||||
### 改动 1: `DocumentChunkConfig` — 增加 token 配置
|
||||
|
||||
```java
|
||||
// 新增字段
|
||||
private int maxTokens = 500; // token 上限（中文约500字，英文约2000字符）
|
||||
private int maxTokensHard = 600; // 硬上限（maxTokens × 1.2）
|
||||
|
||||
// 保留原字段作为向后兼容
|
||||
private int maxSize = 800; // 保留但标记 @Deprecated
|
||||
```
|
||||
|
||||
### 改动 2: `chunkSection()` — 改造累积循环
|
||||
|
||||
**当前逻辑（伪代码）：**
|
||||
|
||||
```
|
||||
for each paragraph:
|
||||
if length + paragraph > maxSize → 切分 → 从 overlap 开始新块
|
||||
append paragraph + "\n\n"
|
||||
```
|
||||
|
||||
**新逻辑（伪代码）：**
|
||||
|
||||
```
|
||||
for each paragraph:
|
||||
currentTokens = estimateTokens(buffer)
|
||||
paraTokens = estimateTokens(paragraph)
|
||||
|
||||
if currentTokens + paraTokens > maxTokens:
|
||||
if isInUnbreakableContext(buffer, paragraph):
|
||||
if currentTokens + paraTokens > maxTokensHard:
|
||||
→ 必须切（硬上限保护）
|
||||
else:
|
||||
→ 不切，继续累积（容忍超出，保护列表完整性）
|
||||
else:
|
||||
→ 切分（段落边界 = 安全切点）
|
||||
→ 从 overlap 开始新块
|
||||
else:
|
||||
→ 不切，继续累积
|
||||
|
||||
append paragraph + "\n\n"
|
||||
```
|
||||
|
||||
### 改动 3: 新增 `estimateTokens()` — 启发式 token 估算
|
||||
|
||||
```java
|
||||
/**
|
||||
* 启发式 token 估算（无需外部依赖）
|
||||
* 中文: ~1 字符/token
|
||||
* 英文/数字: ~4 字符/token
|
||||
* 标点/空白: 忽略
|
||||
*/
|
||||
private int estimateTokens(String text) {
|
||||
int tokens = 0;
|
||||
for (char c : text.toCharArray()) {
|
||||
if (Character.UnicodeBlock.of(c) == Character.UnicodeBlock.CJK_UNIFIED_IDEOGRAPHS
|
||||
|| Character.UnicodeBlock.of(c) == Character.UnicodeBlock.CJK_UNIFIED_IDEOGRAPHS_EXTENSION_A) {
|
||||
tokens += 1; // 中文字符 1:1
|
||||
} else if (Character.isWhitespace(c)) {
|
||||
// 空白字符不计
|
||||
} else {
|
||||
tokens += 1; // 非中文凑 4 个算 1 token（简化）
|
||||
}
|
||||
}
|
||||
// 非中文部分 / 4
|
||||
return tokens;
|
||||
}
|
||||
```
|
||||
|
||||
### 改动 4: 新增 `isInUnbreakableContext()` — 结构感知
|
||||
|
||||
```java
|
||||
/**
|
||||
* 判断当前段落是否属于不可中断的结构
|
||||
* 返回 true = 不能在当前位置切分
|
||||
*/
|
||||
private boolean isInUnbreakableContext(String buffer, String nextParagraph) {
|
||||
// 有序列表: "1. " "2. " "3. " 格式
|
||||
if (nextParagraph.matches("^\\d{1,2}\\.\\s.*")) {
|
||||
// 前一个段落也是列表项 → 不切
|
||||
String lastLine = getLastNonEmptyLine(buffer);
|
||||
if (lastLine.matches("^\\d{1,2}\\.\\s.*|.*\\n\\d{1,2}\\.\\s.*")) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
// 无序列表: "- " 或 "* " 格式
|
||||
if (nextParagraph.matches("^[-*]\\s.*")) {
|
||||
String lastLine = getLastNonEmptyLine(buffer);
|
||||
if (lastLine.matches("^[-*]\\s.*|.*\\n[-*]\\s.*")) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
// 代码块: ``` 内部不切
|
||||
if (buffer.contains("```") && countOccurrences(buffer, "```") % 2 == 1) {
|
||||
return true; // 在未闭合的代码块内 → 不切
|
||||
}
|
||||
return false;
|
||||
}
|
||||
```
|
||||
|
||||
### 改动 5: 修复 index 漂移
|
||||
|
||||
```java
|
||||
// 当前问题：用手工拼装的 chunkContent.length() 推算 offset
|
||||
// String chunkContent = currentChunk.toString().trim(); ← trim 丢字符
|
||||
// currentStartIndex = currentStartIndex + chunkContent.length() - overlap.length(); ← 漂移
|
||||
|
||||
// 改为：用段落在原始文档中的实际位置
|
||||
// 对每个 paragraph 记录其在 section.content 中的 offset，切分时直接使用
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 改动文件清单
|
||||
|
||||
| 文件 | 改动 | 行数变化 |
|
||||
|------|------|----------|
|
||||
| `config/DocumentChunkConfig.java` | +2 字段 | +8 |
|
||||
| `service/DocumentChunkService.java` | 改造 `chunkSection()` + 3 个新方法 | ~+50 / -20 |
|
||||
| `test/.../DocumentChunkServiceTest.java` | 新增列表结构感知 + token 估算用例 | +40 |
|
||||
|
||||
总计改动约 80 行，仅影响一个核心方法。
|
||||
|
||||
---
|
||||
|
||||
## 验收标准
|
||||
|
||||
| # | 用例 | 预期 |
|
||||
|---|------|------|
|
||||
| 1 | 有序列表（5 项，每项 50 字符，maxTokens=180） | 5 项不拆散，容忍略超上限 |
|
||||
| 2 | 有序列表（20 项，超 maxTokensHard） | 在硬上限处切，但不在列表项中间切 |
|
||||
| 3 | 纯段落（10 段，每段 100 字符，maxTokens=300） | 在段落边界切 |
|
||||
| 4 | 中文 800 字 vs 英文 3200 字符 | 分块数接近 |
|
||||
| 5 | H1→空的→H2（标题空壳） | 仍有（不在本次修复范围） |
|
||||
| 6 | 原有测试：空文档、短文档、标题分割、重叠、chunkIndex | 全部通过 |
|
||||
|
||||
---
|
||||
|
||||
## 后续阶段
|
||||
|
||||
| 阶段 | 内容 | 依赖 |
|
||||
|------|------|------|
|
||||
| **Phase 1（本次）** | 改 `chunkSection()` — token + 列表感知 | 无 |
|
||||
| Phase 2 | 标题空壳问题修复（`splitByHeadings` 合并相邻空 section） | Phase 1 |
|
||||
| Phase 3 | 可选：切换到 Spring AI `TokenTextSplitter` | Phase 1/2 |
|
||||
| Phase 4 | 语义相似度辅助切点决策 | Phase 1 |
|
||||
| Phase 5 | Markdown AST 解析替代正则 | 低优先级 |
|
||||
@@ -0,0 +1,40 @@
|
||||
# ChatModel + Embedding 解耦 Design
|
||||
|
||||
## 架构摘要
|
||||
|
||||
当前代码直接使用 DashScope 具体实现类 → 改为面向 Spring AI 抽象接口编程，通过 Spring Boot 自动注入切换实现。
|
||||
|
||||
## 关键决策
|
||||
|
||||
- ChatModel：Spring Boot Starter 自动注册 Bean，通过 `@Autowired ChatModel` 注入，不再手动工厂创建
|
||||
- EmbeddingModel：Spring Boot Starter 自动注册 Bean，通过 `@Autowired EmbeddingModel` 注入，替代 DashScope TextEmbedding SDK
|
||||
- RagService 流式对话：用 `ChatModel.stream(Prompt)` 返回 `Flux<ChatResponse>` 替代 DashScope Generation
|
||||
- VECTOR_DIM：从 `application.yml` 配置读取，替代 `MilvusConstants.VECTOR_DIM` 常量
|
||||
|
||||
## 模块地图
|
||||
|
||||
| 模块 | 职责 | 改动 |
|
||||
| --- | --- | --- |
|
||||
| ChatService | 封装 ChatModel + ReactAgent | 删除工厂方法，注入 ChatModel |
|
||||
| ChatController | HTTP API 入口 | 删除 DashScope import，使用注入 ChatModel |
|
||||
| AiOpsService | 多 Agent 协作 | DashScopeChatModel → ChatModel |
|
||||
| VectorEmbeddingService | 向量化 | DashScope SDK → EmbeddingModel 接口 |
|
||||
| RagService | RAG 流式对话 | DashScope Generation → ChatModel.stream() |
|
||||
| MilvusConstants | Milvus 常量 | VECTOR_DIM 改为配置化 |
|
||||
| MilvusProperties | Milvus 配置 | 新增 vectorDim 字段 |
|
||||
| application.yml | 配置 | 新增 vector-dim 配置项 |
|
||||
|
||||
## 接口影响
|
||||
|
||||
- 级别：L2 内部接口（所有消费者在同一实现范围内）
|
||||
- 判级原因：方法签名从具体类改为接口，调用方需同步修改，但都在本项目内
|
||||
- 不改变外部 API（/api/chat, /api/chat_stream, /api/ai_ops 的 HTTP 响应不变）
|
||||
|
||||
## 架构风险
|
||||
|
||||
- RagService 流式适配最复杂：DashScope Generation 返回 Flowable<GenerationResult>，Spring AI ChatModel.stream() 返回 Flux<ChatResponse>，需适配 StreamCallback 接口
|
||||
- 缓解：Spring AI 的 Flux 与项目已有的 SSE 推送逻辑天然兼容
|
||||
- ChatModel Bean 冲突：多 starter 并存时需 @Primary 或条件注解区分默认实现
|
||||
- 缓解：当前只保留 DashScope starter，不引入多 starter；未来切换时删除旧 starter 即可
|
||||
- DashScopeConfig 通用性：`spring.ai.dashscope.chat.options.timeout` 是厂商绑定配置键
|
||||
- 缓解：本次保留该配置（只做解耦不换实现）；换模型时改配置键
|
||||
@@ -0,0 +1,49 @@
|
||||
# ChatModel + Embedding 解耦 Proposal
|
||||
|
||||
## 问题
|
||||
|
||||
项目 5 个 Java 文件硬编码 DashScope 具体实现类，而非 Spring AI 抽象接口：
|
||||
|
||||
- ChatService/ChatController/AiOpsService：方法签名用 `DashScopeChatModel` 而非 `ChatModel`
|
||||
- VectorEmbeddingService：完全绕过 Spring AI，直接用 DashScope SDK 的 `TextEmbedding`
|
||||
- RagService：完全绕过 Spring AI，直接用 DashScope SDK 的 `Generation`（流式对话）
|
||||
|
||||
导致替换 LLM 或 Embedding 模型需要改代码而非改配置。
|
||||
|
||||
## 建议方案
|
||||
|
||||
**面向 Spring AI 报表接口编程**：
|
||||
- Chat 部分：`DashScopeChatModel` → `ChatModel` 接口，通过 Spring Boot 自动注入
|
||||
- Embedding 部分：DashScope SDK `TextEmbedding` → Spring AI `EmbeddingModel` 接口
|
||||
- RagService 流式对话：DashScope SDK `Generation` → Spring AI `ChatModel` 流式接口 (`stream()`)
|
||||
|
||||
通过 Spring Boot Starter + `application.yml` 配置切换模型实现，无需改代码。
|
||||
|
||||
## 范围
|
||||
|
||||
- 本次要做：
|
||||
- ChatService：删除 `createDashScopeApi()` / `createChatModel()` 工厂方法，改为注入 `ChatModel`
|
||||
- ChatController：删除 DashScope import 和手动构建，改为使用注入的 `ChatModel`
|
||||
- AiOpsService：方法签名 `DashScopeChatModel` → `ChatModel`
|
||||
- VectorEmbeddingService：DashScope SDK → Spring AI `EmbeddingModel`
|
||||
- RagService：DashScope SDK `Generation` → Spring AI `ChatModel` stream
|
||||
- DashScopeConfig：通用化配置（保留 DashScope starter 配置，但代码层不再硬编码 DashScope 类）
|
||||
- application.yml：保持现有 DashScope 配置，增加模型切换说明
|
||||
|
||||
- 本次不做：
|
||||
- 不替换 DashScope 为其他提供商（只做解耦，不换实现）
|
||||
- 不修改 Agent Framework 本身
|
||||
- 不改 Milvus 相关代码
|
||||
- 不改 MCP 客户端配置
|
||||
|
||||
## 关键约束
|
||||
|
||||
- ReactAgent.builder().model() 已接受 ChatModel 接口（已验证）
|
||||
- Spring AI 的 EmbeddingModel 接口可替代 DashScope TextEmbedding
|
||||
- Spring AI 的 ChatModel.stream() 可替代 DashScope Generation 流式接口
|
||||
- DashScope starter 仍需保留作为默认实现（通过 pom 依赖 + yml 配置）
|
||||
|
||||
## 风险
|
||||
|
||||
- RagService 流式对话的迁移可能最复杂：DashScope SDK 返回 RxJava Flowable，Spring AI ChatModel.stream() 返回 Flux，需要适配 SSE 推送逻辑
|
||||
- VectorEmbeddingService 维度可能变化：DashScope text-embedding-v4 输出 1024 维，替换模型后维度不同，需要同步修改 Milvus VECTOR_DIM 常量
|
||||
@@ -0,0 +1,24 @@
|
||||
# ChatModel + Embedding 解耦 Specs
|
||||
|
||||
## 可观察行为规格
|
||||
|
||||
### S1: Chat 接口不变
|
||||
- `/api/chat`, `/api/chat_stream`, `/api/ai_ops` 的 HTTP 入参/出参/响应结构完全不变
|
||||
- 功能行为不变：工具调用、Agent 协作、SSE 流式推送照旧工作
|
||||
|
||||
### S2: 模型切换只需改配置
|
||||
- 替换 DashScope starter 为 OpenAI starter + 改 yml 配置 → ChatModel 自动注入不同实现
|
||||
- 替换 embedding 模型只需改 yml 的 `dashscope.embedding.model` 和 `milvus.vector-dim`
|
||||
- 不需要改任何 Java 代码
|
||||
|
||||
### S3: VECTOR_DIM 从配置读取
|
||||
- `MilvusClientFactory.createBizCollection()` 使用 MilvusProperties.getVectorDim() 而非 MilvusConstants.VECTOR_DIM
|
||||
- 切换 embedding 模型后改 yml 的 `milvus.vector-dim` 即可适配新维度
|
||||
|
||||
### S4: VectorEmbeddingService 行为不变
|
||||
- generateEmbedding/generateEmbeddings/generateQueryVector 的签名和返回类型不变
|
||||
- 内部实现从 DashScope SDK 切换到 Spring AI EmbeddingModel
|
||||
|
||||
### S5: RagService 流式对话行为不变
|
||||
- queryStream 方法签名和 StreamCallback 接口不变
|
||||
- 内部实现从 DashScope Generation 切换到 Spring AI ChatModel.stream()
|
||||
@@ -0,0 +1,22 @@
|
||||
# ChatModel + Embedding 解耦 Tasks
|
||||
|
||||
## 需求追踪
|
||||
|
||||
| 需求 | 状态 | 备注 |
|
||||
| --- | --- | --- |
|
||||
| ChatService 解耦 DashScopeChatModel | 待处理 | 改为注入 ChatModel |
|
||||
| ChatController 解耦 DashScope | 待处理 | 删除手动构建逻辑 |
|
||||
| AiOpsService 解耦 DashScopeChatModel | 待处理 | 方法签名改为 ChatModel |
|
||||
| VectorEmbeddingService 解耦 DashScope SDK | 待处理 | 改为注入 EmbeddingModel |
|
||||
| RagService 解耦 DashScope Generation | 待处理 | 改为 ChatModel.stream() |
|
||||
| VECTOR_DIM 配置化 | 待处理 | 从 yml 读取 |
|
||||
|
||||
## 实现任务
|
||||
|
||||
- [ ] T1: MilvusProperties 新增 vectorDim 字段 + getter/setter，application.yml 新增 `milvus.vector-dim: 1024`
|
||||
- [ ] T2: MilvusConstants.VECTOR_DIM 改为从 MilvusProperties 动态读取（MilvusClientFactory 传入）
|
||||
- [ ] T3: ChatService — 删除 createDashScopeApi/createChatModel/createStandardChatModel，新增 @Autowired ChatModel；createReactAgent 参数改为 ChatModel
|
||||
- [ ] T4: ChatController — 删除 DashScope import 和手动构建（行83-84, 171-172, 292-301），改为使用注入 ChatModel 或 ChatService 传入
|
||||
- [ ] T5: AiOpsService — executeAiOpsAnalysis/buildPlannerAgent/buildExecutorAgent 参数类型 DashScopeChatModel → ChatModel
|
||||
- [ ] T6: VectorEmbeddingService — 删除 DashScope SDK import + TextEmbedding 字段 + @PostConstruct init()，改为 @Autowired EmbeddingModel；generateEmbedding 改为调用 EmbeddingModel.embed()
|
||||
- [ ] T7: RagService — 删除 DashScope SDK import + Generation 字段 + Constants.apiKey，改为 @Autowired ChatModel；generateAnswerStream 改为 ChatModel.stream(Prompt) + Flux 适配 StreamCallback
|
||||
@@ -138,6 +138,13 @@
|
||||
<version>4.36.0</version>
|
||||
</dependency>
|
||||
|
||||
<!-- 测试依赖 -->
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<!-- Spring AI MCP Client - 使用 WebFlux 版本 -->
|
||||
<!-- 注意：spring-ai-starter-mcp-client-webflux 已经包含了 mcp-annotations，无需单独引入 -->
|
||||
<dependency>
|
||||
|
||||
@@ -78,10 +78,16 @@ public class MilvusClientFactory {
|
||||
ConnectParam.Builder builder = ConnectParam.newBuilder()
|
||||
.withHost(milvusProperties.getHost())
|
||||
.withPort(milvusProperties.getPort())
|
||||
.withDatabaseName(milvusProperties.getDatabase())
|
||||
.withConnectTimeout(milvusProperties.getTimeout(), TimeUnit.MILLISECONDS);
|
||||
|
||||
// 如果配置了用户名和密码
|
||||
if (milvusProperties.getUsername() != null && !milvusProperties.getUsername().isEmpty()) {
|
||||
// Zilliz Cloud: token + SSL
|
||||
if (milvusProperties.getToken() != null && !milvusProperties.getToken().isEmpty()) {
|
||||
builder.withToken(milvusProperties.getToken());
|
||||
builder.withSecure(true);
|
||||
}
|
||||
// 本地 Milvus: username + password
|
||||
else if (milvusProperties.getUsername() != null && !milvusProperties.getUsername().isEmpty()) {
|
||||
builder.withAuthorization(milvusProperties.getUsername(), milvusProperties.getPassword());
|
||||
}
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ import org.springframework.context.annotation.Configuration;
|
||||
public class DocumentChunkConfig {
|
||||
|
||||
/**
|
||||
* 每个分片的最大字符数
|
||||
* 每个分片的最大字符数（保留向后兼容）
|
||||
*/
|
||||
private int maxSize = 800;
|
||||
|
||||
@@ -22,6 +22,18 @@ public class DocumentChunkConfig {
|
||||
*/
|
||||
private int overlap = 100;
|
||||
|
||||
/**
|
||||
* 每个分片的最大 token 数（中文~1:1，英文~0.25:1）
|
||||
* 替代 maxSize 作为切割触发器
|
||||
*/
|
||||
private int maxTokens = 500;
|
||||
|
||||
/**
|
||||
* 硬上限 token 数 = maxTokens × 1.2
|
||||
* 仅在不可中断上下文（列表、代码块）内触发
|
||||
*/
|
||||
private int maxTokensHard = 600;
|
||||
|
||||
public void setMaxSize(int maxSize) {
|
||||
this.maxSize = maxSize;
|
||||
}
|
||||
@@ -29,4 +41,12 @@ public class DocumentChunkConfig {
|
||||
public void setOverlap(int overlap) {
|
||||
this.overlap = overlap;
|
||||
}
|
||||
|
||||
public void setMaxTokens(int maxTokens) {
|
||||
this.maxTokens = maxTokens;
|
||||
}
|
||||
|
||||
public void setMaxTokensHard(int maxTokensHard) {
|
||||
this.maxTokensHard = maxTokensHard;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -13,6 +13,8 @@ public class MilvusProperties {
|
||||
private String password = "";
|
||||
private String database = "default";
|
||||
private Long timeout = 10000L;
|
||||
private String token = "";
|
||||
private boolean secure = false;
|
||||
|
||||
public String getHost() {
|
||||
return host;
|
||||
@@ -62,6 +64,22 @@ public class MilvusProperties {
|
||||
this.timeout = timeout;
|
||||
}
|
||||
|
||||
public String getToken() {
|
||||
return token;
|
||||
}
|
||||
|
||||
public void setToken(String token) {
|
||||
this.token = token;
|
||||
}
|
||||
|
||||
public boolean isSecure() {
|
||||
return secure;
|
||||
}
|
||||
|
||||
public void setSecure(boolean secure) {
|
||||
this.secure = secure;
|
||||
}
|
||||
|
||||
public String getAddress() {
|
||||
return host + ":" + port;
|
||||
}
|
||||
|
||||
@@ -100,14 +100,21 @@ public class DocumentChunkService {
|
||||
|
||||
/**
|
||||
* 对单个章节进行分片
|
||||
* <p>
|
||||
* 核心改造（Phase 1）：
|
||||
* - Token 估算替代字符计数
|
||||
* - 感知有序/无序列表结构，不在列表中间切断
|
||||
* - 软边界（maxTokens）+ 硬上限（maxTokensHard）双重控制
|
||||
* - 修复 currentStartIndex 漂移：用段落原始位置而非手工推算
|
||||
*/
|
||||
private List<DocumentChunk> chunkSection(Section section, int startChunkIndex) {
|
||||
List<DocumentChunk> chunks = new ArrayList<>();
|
||||
String content = section.content;
|
||||
String title = section.title;
|
||||
|
||||
// 如果章节内容小于最大尺寸，直接作为一个分片
|
||||
if (content.length() <= chunkConfig.getMaxSize()) {
|
||||
// 短章节直接作为一个分片（用 token 估算替代字符数做短路判断）
|
||||
if (content.length() <= chunkConfig.getMaxSize()
|
||||
&& estimateTokens(content) <= chunkConfig.getMaxTokens()) {
|
||||
DocumentChunk chunk = new DocumentChunk(
|
||||
content,
|
||||
section.startIndex,
|
||||
@@ -120,45 +127,71 @@ public class DocumentChunkService {
|
||||
}
|
||||
|
||||
// 章节内容较长，需要进一步分片
|
||||
// 优先在段落边界分割
|
||||
List<String> paragraphs = splitByParagraphs(content);
|
||||
if (paragraphs.isEmpty()) {
|
||||
return chunks;
|
||||
}
|
||||
|
||||
StringBuilder currentChunk = new StringBuilder();
|
||||
int currentStartIndex = section.startIndex;
|
||||
// 定位每个段落在 section.content 中的位置（修复 index 漂移）
|
||||
List<ParagraphPos> paraPositions = locateParagraphPositions(paragraphs, content);
|
||||
|
||||
// 当前分片的段落范围
|
||||
int chunkParaStart = 0; // 当前分片第一个段落的索引（在 paragraphs 中）
|
||||
StringBuilder buffer = new StringBuilder();
|
||||
int tokenCount = 0;
|
||||
int chunkIndex = startChunkIndex;
|
||||
|
||||
for (String paragraph : paragraphs) {
|
||||
// 如果当前分片加上新段落超过最大尺寸
|
||||
if (currentChunk.length() > 0 &&
|
||||
currentChunk.length() + paragraph.length() > chunkConfig.getMaxSize()) {
|
||||
for (int i = 0; i < paragraphs.size(); i++) {
|
||||
String paragraph = paragraphs.get(i);
|
||||
int paraTokens = estimateTokens(paragraph);
|
||||
|
||||
// 保存当前分片
|
||||
String chunkContent = currentChunk.toString().trim();
|
||||
DocumentChunk chunk = new DocumentChunk(
|
||||
chunkContent,
|
||||
currentStartIndex,
|
||||
currentStartIndex + chunkContent.length(),
|
||||
chunkIndex++
|
||||
);
|
||||
chunk.setTitle(title);
|
||||
chunks.add(chunk);
|
||||
// 判断是否需要切分
|
||||
if (buffer.length() > 0 && tokenCount + paraTokens > chunkConfig.getMaxTokens()) {
|
||||
|
||||
// 开始新分片，包含重叠部分
|
||||
String overlap = getOverlapText(chunkContent);
|
||||
currentChunk = new StringBuilder(overlap);
|
||||
currentStartIndex = currentStartIndex + chunkContent.length() - overlap.length();
|
||||
// 检查是否处于不可中断的上下文中
|
||||
if (isInUnbreakableContext(buffer.toString(), paragraph)) {
|
||||
// 硬上限保护：即使不可中断也不能无限膨胀
|
||||
if (tokenCount + paraTokens > chunkConfig.getMaxTokensHard()) {
|
||||
logger.debug(" 触及硬上限 ({} tokens)，强制切分", tokenCount + paraTokens);
|
||||
chunkParaStart = saveChunkAndGetNextStart(
|
||||
chunks, section, paraPositions,
|
||||
chunkParaStart, i, title, chunkIndex);
|
||||
chunkIndex++;
|
||||
|
||||
String prevChunkContent = chunks.get(chunks.size() - 1).getContent();
|
||||
String overlap = getOverlapText(prevChunkContent);
|
||||
buffer = new StringBuilder(overlap);
|
||||
tokenCount = estimateTokens(overlap);
|
||||
}
|
||||
// 否则：容忍超出（软边界）
|
||||
} else {
|
||||
// 安全切点：段落边界
|
||||
chunkParaStart = saveChunkAndGetNextStart(
|
||||
chunks, section, paraPositions,
|
||||
chunkParaStart, i, title, chunkIndex);
|
||||
chunkIndex++;
|
||||
|
||||
// 新分片以重叠文本开头
|
||||
String prevChunkContent = chunks.get(chunks.size() - 1).getContent();
|
||||
String overlap = getOverlapText(prevChunkContent);
|
||||
buffer = new StringBuilder(overlap);
|
||||
tokenCount = estimateTokens(overlap);
|
||||
}
|
||||
}
|
||||
|
||||
currentChunk.append(paragraph).append("\n\n");
|
||||
buffer.append(paragraph).append("\n\n");
|
||||
tokenCount += paraTokens;
|
||||
}
|
||||
|
||||
// 保存最后一个分片
|
||||
if (currentChunk.length() > 0) {
|
||||
String chunkContent = currentChunk.toString().trim();
|
||||
if (buffer.length() > 0 && chunkParaStart < paragraphs.size()) {
|
||||
String chunkContent = buffer.toString().trim();
|
||||
int actualStart = paraPositions.get(chunkParaStart).start;
|
||||
int actualEnd = paraPositions.get(paragraphs.size() - 1).end;
|
||||
DocumentChunk chunk = new DocumentChunk(
|
||||
chunkContent,
|
||||
currentStartIndex,
|
||||
currentStartIndex + chunkContent.length(),
|
||||
section.startIndex + actualStart,
|
||||
section.startIndex + actualEnd,
|
||||
chunkIndex
|
||||
);
|
||||
chunk.setTitle(title);
|
||||
@@ -168,6 +201,36 @@ public class DocumentChunkService {
|
||||
return chunks;
|
||||
}
|
||||
|
||||
/**
|
||||
* 保存当前分块，返回下一个分块的起始段落索引
|
||||
* <p>
|
||||
* 从 section.content 中提取原始文本（而非手工拼装），修复 index 漂移问题
|
||||
*/
|
||||
private int saveChunkAndGetNextStart(
|
||||
List<DocumentChunk> chunks,
|
||||
Section section,
|
||||
List<ParagraphPos> paraPositions,
|
||||
int fromPara,
|
||||
int toPara,
|
||||
String title,
|
||||
int chunkIndex) {
|
||||
|
||||
int actualStart = paraPositions.get(fromPara).start;
|
||||
int actualEnd = paraPositions.get(toPara - 1).end;
|
||||
String originalText = section.content.substring(actualStart, actualEnd);
|
||||
|
||||
DocumentChunk chunk = new DocumentChunk(
|
||||
originalText,
|
||||
section.startIndex + actualStart,
|
||||
section.startIndex + actualEnd,
|
||||
chunkIndex
|
||||
);
|
||||
chunk.setTitle(title);
|
||||
chunks.add(chunk);
|
||||
|
||||
return toPara; // 下一个分块的起始段落索引
|
||||
}
|
||||
|
||||
/**
|
||||
* 按段落分割文本
|
||||
*/
|
||||
@@ -186,6 +249,106 @@ public class DocumentChunkService {
|
||||
return paragraphs;
|
||||
}
|
||||
|
||||
/**
|
||||
* 定位每个段落在原始文本中的字符偏移
|
||||
*/
|
||||
private List<ParagraphPos> locateParagraphPositions(List<String> paragraphs, String sectionContent) {
|
||||
List<ParagraphPos> positions = new ArrayList<>();
|
||||
int searchFrom = 0;
|
||||
for (String p : paragraphs) {
|
||||
int idx = sectionContent.indexOf(p, searchFrom);
|
||||
if (idx >= 0) {
|
||||
positions.add(new ParagraphPos(idx, idx + p.length()));
|
||||
searchFrom = idx + p.length();
|
||||
} else {
|
||||
// fallback: 段落在原文中找不到（不应该发生）
|
||||
positions.add(new ParagraphPos(searchFrom, searchFrom + p.length()));
|
||||
searchFrom += p.length();
|
||||
}
|
||||
}
|
||||
return positions;
|
||||
}
|
||||
|
||||
/**
|
||||
* 启发式 token 估算（无需外部依赖）
|
||||
* <p>
|
||||
* 中文(BMP): ~1 字符/token
|
||||
* 英文/数字/标点: ~4 字符/token
|
||||
* 空白字符忽略
|
||||
*/
|
||||
private int estimateTokens(String text) {
|
||||
int nonCjkCount = 0;
|
||||
int cjkCount = 0;
|
||||
for (char c : text.toCharArray()) {
|
||||
if (Character.isWhitespace(c)) {
|
||||
continue;
|
||||
}
|
||||
Character.UnicodeBlock block = Character.UnicodeBlock.of(c);
|
||||
if (block == Character.UnicodeBlock.CJK_UNIFIED_IDEOGRAPHS
|
||||
|| block == Character.UnicodeBlock.CJK_UNIFIED_IDEOGRAPHS_EXTENSION_A
|
||||
|| block == Character.UnicodeBlock.CJK_UNIFIED_IDEOGRAPHS_EXTENSION_B
|
||||
|| block == Character.UnicodeBlock.CJK_COMPATIBILITY_IDEOGRAPHS) {
|
||||
cjkCount++;
|
||||
} else {
|
||||
nonCjkCount++;
|
||||
}
|
||||
}
|
||||
return cjkCount + (nonCjkCount + 3) / 4; // 非中文每 4 字符算 1 token，向上取整
|
||||
}
|
||||
|
||||
/**
|
||||
* 判断当前段落是否属于不可中断的结构
|
||||
* <p>
|
||||
* 不可中断结构包括：
|
||||
* - 有序列表项（"1. ", "2. " 格式）
|
||||
* - 无序列表项（"- " 或 "* " 格式）
|
||||
* - 未闭合的代码块（``` 内）
|
||||
*/
|
||||
private boolean isInUnbreakableContext(String buffer, String nextParagraph) {
|
||||
// 有序列表：判断 buffer 末尾和下一段是否都是列表项
|
||||
if (nextParagraph.matches("^\\d{1,2}\\.\\s.*")) {
|
||||
String lastLine = getLastNonEmptyLine(buffer);
|
||||
if (lastLine != null && lastLine.matches("^\\d{1,2}\\.\\s.*")) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
// 无序列表："- " 或 "* " 格式
|
||||
if (nextParagraph.matches("^[-*]\\s.*")) {
|
||||
String lastLine = getLastNonEmptyLine(buffer);
|
||||
if (lastLine != null && lastLine.matches("^[-*]\\s.*")) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
// 代码块：``` 未闭合
|
||||
if (buffer.contains("```")) {
|
||||
int count = 0;
|
||||
for (int i = 0; i <= buffer.length() - 3; i++) {
|
||||
if (buffer.substring(i).startsWith("```")) {
|
||||
count++;
|
||||
i += 2;
|
||||
}
|
||||
}
|
||||
if (count % 2 == 1) {
|
||||
return true; // 奇数个 ``` → 在代码块内部
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* 获取 buffer 中最后一行非空白文本
|
||||
*/
|
||||
private String getLastNonEmptyLine(String buffer) {
|
||||
String[] lines = buffer.split("\n");
|
||||
for (int i = lines.length - 1; i >= 0; i--) {
|
||||
String line = lines[i].trim();
|
||||
if (!line.isEmpty()) {
|
||||
return line;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* 获取重叠文本
|
||||
* 从文本末尾提取指定长度的内容作为下一个分片的开头
|
||||
@@ -212,6 +375,19 @@ public class DocumentChunkService {
|
||||
return overlap.trim();
|
||||
}
|
||||
|
||||
/**
|
||||
* 段落在原文中的位置
|
||||
*/
|
||||
private static class ParagraphPos {
|
||||
final int start;
|
||||
final int end;
|
||||
|
||||
ParagraphPos(int start, int end) {
|
||||
this.start = start;
|
||||
this.end = end;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 章节数据类
|
||||
*/
|
||||
|
||||
@@ -12,12 +12,14 @@ file:
|
||||
allowed-extensions: txt,md
|
||||
|
||||
milvus:
|
||||
host: localhost
|
||||
port: 19530
|
||||
host: in03-4a578da0f27ce9d.serverless.aws-eu-central-1.cloud.zilliz.com
|
||||
port: 443
|
||||
username: ""
|
||||
password: ""
|
||||
database: default
|
||||
database: db_4a578da0f27ce9d
|
||||
timeout: 10000
|
||||
token: ${MILVUS_TOKEN:}
|
||||
secure: true
|
||||
|
||||
# Spring AI Alibaba DashScope 配置
|
||||
spring:
|
||||
|
||||
@@ -0,0 +1,539 @@
|
||||
package org.example.service;
|
||||
|
||||
import org.example.config.DocumentChunkConfig;
|
||||
import org.example.dto.DocumentChunk;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Nested;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/**
|
||||
* 当前分片策略的单元测试 — 覆盖旧能力回归 + Phase 1 新增能力
|
||||
*/
|
||||
@DisplayName("DocumentChunkService 分片策略")
|
||||
class DocumentChunkServiceTest {
|
||||
|
||||
private DocumentChunkService service;
|
||||
private DocumentChunkConfig config;
|
||||
|
||||
@BeforeEach
|
||||
void setUp() {
|
||||
config = new DocumentChunkConfig();
|
||||
config.setMaxSize(800);
|
||||
config.setMaxTokens(500);
|
||||
config.setMaxTokensHard(600);
|
||||
config.setOverlap(100);
|
||||
service = new DocumentChunkService();
|
||||
try {
|
||||
var field = DocumentChunkService.class.getDeclaredField("chunkConfig");
|
||||
field.setAccessible(true);
|
||||
field.set(service, config);
|
||||
} catch (Exception e) {
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
}
|
||||
|
||||
// ==================== 回归：边界条件 ====================
|
||||
|
||||
@Nested
|
||||
@DisplayName("边界条件")
|
||||
class BoundaryTests {
|
||||
|
||||
@Test
|
||||
@DisplayName("null 内容 → 空列表")
|
||||
void nullContent_returnsEmpty() {
|
||||
List<DocumentChunk> chunks = service.chunkDocument(null, "/test/null.md");
|
||||
assertTrue(chunks.isEmpty());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("空字符串 → 空列表")
|
||||
void emptyContent_returnsEmpty() {
|
||||
List<DocumentChunk> chunks = service.chunkDocument(" \n ", "/test/empty.md");
|
||||
assertTrue(chunks.isEmpty());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("短文档（≤maxSize）→ 1个分块")
|
||||
void shortDocument_singleChunk() {
|
||||
String content = "这是一篇短文档，内容不超过800个字符。";
|
||||
List<DocumentChunk> chunks = service.chunkDocument(content, "/test/short.md");
|
||||
|
||||
assertEquals(1, chunks.size());
|
||||
assertEquals(content, chunks.get(0).getContent());
|
||||
assertEquals(0, chunks.get(0).getChunkIndex());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("恰好 maxSize 边界 → 1个分块")
|
||||
void exactlyMaxSize_singleChunk() {
|
||||
String content = "A".repeat(800);
|
||||
List<DocumentChunk> chunks = service.chunkDocument(content, "/test/boundary.md");
|
||||
assertEquals(1, chunks.size());
|
||||
}
|
||||
}
|
||||
|
||||
// ==================== 回归：标题分割 ====================
|
||||
|
||||
@Nested
|
||||
@DisplayName("Markdown 标题分割")
|
||||
class HeadingSplitTests {
|
||||
|
||||
@Test
|
||||
@DisplayName("单个 H1 标题 → section 继承标题")
|
||||
void singleHeading_titlePropagates() {
|
||||
String content = "# CPU高负载问题\n\n这是CPU高负载的描述内容。";
|
||||
List<DocumentChunk> chunks = service.chunkDocument(content, "/test/cpu.md");
|
||||
|
||||
assertEquals(1, chunks.size());
|
||||
assertEquals("CPU高负载问题", chunks.get(0).getTitle());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("多个标题 → 按标题边界分割")
|
||||
void multipleHeadings_splitAtHeadings() {
|
||||
String content =
|
||||
"# CPU高负载\n\nCPU问题的详细描述。\n\n" +
|
||||
"# 内存高负载\n\n内存问题的详细描述。";
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(content, "/test/multi.md");
|
||||
|
||||
assertEquals(2, chunks.size());
|
||||
assertEquals("CPU高负载", chunks.get(0).getTitle());
|
||||
assertEquals("内存高负载", chunks.get(1).getTitle());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("多级标题（H1/H2/H3）→ 标题独立不冲突")
|
||||
void multiLevelHeadings() {
|
||||
String content =
|
||||
"# 一级标题\n\n一级内容。\n\n" +
|
||||
"## 二级标题\n\n二级内容。\n\n" +
|
||||
"### 三级标题\n\n三级内容。";
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(content, "/test/levels.md");
|
||||
assertEquals(3, chunks.size());
|
||||
assertEquals("一级标题", chunks.get(0).getTitle());
|
||||
assertEquals("二级标题", chunks.get(1).getTitle());
|
||||
assertEquals("三级标题", chunks.get(2).getTitle());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("H1-H6 全部支持")
|
||||
void allHeadingLevels() {
|
||||
StringBuilder sb = new StringBuilder();
|
||||
for (int i = 1; i <= 6; i++) {
|
||||
sb.append("#".repeat(i)).append(" 标题").append(i).append("\n\n内容").append(i).append("。\n\n");
|
||||
}
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(sb.toString(), "/test/h1h6.md");
|
||||
assertEquals(6, chunks.size());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("无标题文档 → 整个文档作为1个 section")
|
||||
void noHeadings_entireAsOneSection() {
|
||||
String content = "纯文本没有标题。\n\n第二段内容。\n\n第三段内容。";
|
||||
List<DocumentChunk> chunks = service.chunkDocument(content, "/test/nohead.md");
|
||||
assertFalse(chunks.isEmpty());
|
||||
assertNull(chunks.get(0).getTitle());
|
||||
}
|
||||
}
|
||||
|
||||
// ==================== 回归：段落边界切分 ====================
|
||||
|
||||
@Nested
|
||||
@DisplayName("超长章节 — 段落边界切分")
|
||||
class ParagraphSplitTests {
|
||||
|
||||
@Test
|
||||
@DisplayName("短章节（≤maxSize）→ 不进入段落切割")
|
||||
void shortSection_noParagraphSplit() {
|
||||
StringBuilder sb = new StringBuilder();
|
||||
sb.append("# 测试\n\n");
|
||||
for (int i = 0; i < 5; i++) {
|
||||
sb.append("段落").append(i).append("：这是一段短内容。\n\n");
|
||||
}
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(sb.toString(), "/test/short_sec.md");
|
||||
assertEquals(1, chunks.size());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("超长章节 → 在段落边界切分")
|
||||
void longSection_splitsAtParagraphBoundaries() {
|
||||
config.setMaxSize(50);
|
||||
config.setMaxTokens(30);
|
||||
|
||||
StringBuilder sb = new StringBuilder();
|
||||
sb.append("# 长章节\n\n");
|
||||
for (int i = 0; i < 10; i++) {
|
||||
sb.append("段落").append(i).append("：ABCDEFGHIJKLMNOPQRSTUVWXYZ。\n\n");
|
||||
}
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(sb.toString(), "/test/long_sec.md");
|
||||
assertTrue(chunks.size() >= 2, "超长章节应切分为多个分块，实际: " + chunks.size());
|
||||
|
||||
// 所有分块携带相同的 title
|
||||
for (DocumentChunk c : chunks) {
|
||||
assertEquals("长章节", c.getTitle());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ==================== 回归：chunkIndex 元数据 ====================
|
||||
|
||||
@Nested
|
||||
@DisplayName("分块元数据")
|
||||
class ChunkMetadataTests {
|
||||
|
||||
@Test
|
||||
@DisplayName("chunkIndex 自增且唯一")
|
||||
void chunkIndexSequential() {
|
||||
config.setMaxSize(50);
|
||||
config.setMaxTokens(30);
|
||||
|
||||
StringBuilder sb = new StringBuilder("# Meta\n\n");
|
||||
for (int i = 0; i < 10; i++) {
|
||||
sb.append("段落").append(i).append("：填充内容以触发切分机制。ABCDE。\n\n");
|
||||
}
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(sb.toString(), "/test/meta.md");
|
||||
assertTrue(chunks.size() >= 2);
|
||||
|
||||
for (int i = 0; i < chunks.size(); i++) {
|
||||
assertEquals(i, chunks.get(i).getChunkIndex(),
|
||||
"chunkIndex 应从0开始连续递增");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("startIndex/endIndex 范围合法 — 无漂移")
|
||||
void indexRangeValid_noDrift() {
|
||||
String content = "# 标题\n\n测试内容。";
|
||||
List<DocumentChunk> chunks = service.chunkDocument(content, "/test/index.md");
|
||||
|
||||
for (DocumentChunk c : chunks) {
|
||||
assertTrue(c.getStartIndex() >= 0);
|
||||
assertTrue(c.getEndIndex() > c.getStartIndex(),
|
||||
"endIndex(" + c.getEndIndex() + ") 应 > startIndex(" + c.getStartIndex() + ")");
|
||||
assertTrue(c.getEndIndex() <= content.length());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ==================== 新增：Token 估算 ====================
|
||||
|
||||
@Nested
|
||||
@DisplayName("Token 估算")
|
||||
class TokenEstimationTests {
|
||||
|
||||
@Test
|
||||
@DisplayName("纯中文 800 字符 ≈ 800 tokens → 短章节不切")
|
||||
void pureChinese_fewerTokensThanMax() {
|
||||
config.setMaxTokens(400);
|
||||
|
||||
StringBuilder sb = new StringBuilder();
|
||||
sb.append("# 中文测试\n\n");
|
||||
// 纯中文 ~300 字符 ≈ 300 tokens
|
||||
for (int i = 0; i < 3; i++) {
|
||||
sb.append("这是纯中文测试内容的第十").append(i).append("段落。");
|
||||
sb.append("每个中文字符大约占用一个令牌的位置。");
|
||||
sb.append("因此这段文本的令牌数大致等于字符数。\n\n");
|
||||
}
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(sb.toString(), "/test/cn_tokens.md");
|
||||
// 300 字符 ≈ 300 tokens < 400 maxTokens → 1 个分块
|
||||
assertEquals(1, chunks.size());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("纯英文 2000 字符 ≈ 500 tokens → 刚好不超过上限")
|
||||
void pureEnglish_moreCharactersSameTokens() {
|
||||
config.setMaxTokens(200);
|
||||
config.setMaxTokensHard(250);
|
||||
|
||||
StringBuilder sb = new StringBuilder();
|
||||
sb.append("# English Test\n\n");
|
||||
for (int i = 0; i < 8; i++) {
|
||||
sb.append("This is paragraph number ").append(i)
|
||||
.append(" containing English text. ")
|
||||
.append("English characters are much cheaper in tokens. ")
|
||||
.append("More filler text here to reach the limit properly. ")
|
||||
.append("Yet another sentence for good measure. ")
|
||||
.append("Still more words needed to reach token limit here.\n\n");
|
||||
}
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(sb.toString(), "/test/en_tokens.md");
|
||||
// 大量英文才占少量 token → 分块数应少于用字符计数的版本
|
||||
assertTrue(chunks.size() >= 2, "1200+ 字符英文应切分");
|
||||
}
|
||||
}
|
||||
|
||||
// ==================== 新增：列表结构感知 ====================
|
||||
|
||||
@Nested
|
||||
@DisplayName("列表结构感知")
|
||||
class ListStructureTests {
|
||||
|
||||
@Test
|
||||
@DisplayName("有序列表项之间不切分 — 即使超过 maxTokens")
|
||||
void orderedList_notSplitBetweenItems() {
|
||||
config.setMaxTokens(80);
|
||||
config.setMaxTokensHard(200);
|
||||
config.setOverlap(30);
|
||||
|
||||
StringBuilder sb = new StringBuilder();
|
||||
sb.append("# 排查步骤\n\n");
|
||||
// 5个有序列表项，每项 ~40 字符 ≈ 40 tokens，总共 ~200 tokens
|
||||
for (int i = 1; i <= 5; i++) {
|
||||
sb.append(i).append(". 这是排查步骤第").append(i)
|
||||
.append("项，包含具体的操作指引和注意事项说明。\n\n");
|
||||
}
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(sb.toString(), "/test/ordered_list.md");
|
||||
|
||||
// 5项应保持在一起（未触及 hard 上限）
|
||||
assertEquals(1, chunks.size(),
|
||||
"有序列表项不应被拆散，实际分块数: " + chunks.size());
|
||||
|
||||
String content = chunks.get(0).getContent();
|
||||
assertTrue(content.contains("1. "), "应包含第1项");
|
||||
assertTrue(content.contains("5. "), "应包含第5项");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("有序列表触及硬上限 → 在列表项边界强制切分")
|
||||
void orderedList_hardLimitSplits() {
|
||||
config.setMaxTokens(50);
|
||||
config.setMaxTokensHard(100);
|
||||
config.setOverlap(20);
|
||||
|
||||
StringBuilder sb = new StringBuilder();
|
||||
sb.append("# 长列表\n\n");
|
||||
// 每项 ~60 tokens，硬上限 100 → 最多装 1 项多
|
||||
for (int i = 1; i <= 6; i++) {
|
||||
sb.append(i).append(". 这是很长的排查步骤内容，包含详细的说明信息。")
|
||||
.append("每个步骤都要执行多个检查操作。继续填充文本以增加令牌计数。\n\n");
|
||||
}
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(sb.toString(), "/test/long_list.md");
|
||||
|
||||
System.out.println(" 长列表硬上限测试 — 实际分块数: " + chunks.size());
|
||||
for (DocumentChunk c : chunks) {
|
||||
System.out.println(" Chunk #" + c.getChunkIndex() + ": " + c.getContent().length() + "字符 "
|
||||
+ "| start=" + c.getStartIndex() + " end=" + c.getEndIndex()
|
||||
+ " | preview=" + c.getContent().substring(0, Math.min(60, c.getContent().length())).replace("\n", "\\n"));
|
||||
}
|
||||
|
||||
// 硬上限会强制切分，但每个分块内的列表项应保持连续
|
||||
assertTrue(chunks.size() >= 2, "长列表应至少触发1次切分，实际: " + chunks.size());
|
||||
|
||||
// 验证：除了第一个分块（可能是标题），其余应包含列表项
|
||||
for (int i = 1; i < chunks.size(); i++) {
|
||||
DocumentChunk c = chunks.get(i);
|
||||
assertFalse(c.getContent().isEmpty());
|
||||
assertTrue(c.getContent().matches("(?s).*\\d+\\.\\s.*"),
|
||||
"非标题分块应包含列表项，Chunk #" + c.getChunkIndex()
|
||||
+ " preview: " + c.getContent().substring(0, Math.min(60, c.getContent().length())));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("无序列表项之间不切分")
|
||||
void unorderedList_notSplitBetweenItems() {
|
||||
config.setMaxTokens(80);
|
||||
config.setMaxTokensHard(200);
|
||||
|
||||
StringBuilder sb = new StringBuilder();
|
||||
sb.append("# 检查清单\n\n");
|
||||
for (int i = 1; i <= 5; i++) {
|
||||
sb.append("- 检查项").append(i).append("：确认服务运行状态正常并记录相关指标。\n\n");
|
||||
}
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(sb.toString(), "/test/unordered_list.md");
|
||||
assertEquals(1, chunks.size(), "无序列表项不应被拆散");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("列表结束后普通段落应从下一段落开始新分块")
|
||||
void listEnds_normalParagraphStartsNewChunk() {
|
||||
config.setMaxTokens(150);
|
||||
config.setMaxTokensHard(250);
|
||||
|
||||
StringBuilder sb = new StringBuilder();
|
||||
sb.append("# 文档\n\n");
|
||||
// 先一个普通段落
|
||||
sb.append("这是介绍段落，描述系统的整体架构和设计思路。\n\n");
|
||||
// 有序列表
|
||||
for (int i = 1; i <= 3; i++) {
|
||||
sb.append(i).append(". 列表项第").append(i).append("条，包含操作说明。\n\n");
|
||||
}
|
||||
// 普通段落
|
||||
sb.append("这是总结段落，包含上述操作完成后需要关注的监控指标。\n\n");
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(sb.toString(), "/test/list_mixed.md");
|
||||
assertTrue(chunks.size() >= 1);
|
||||
// 列表项应保持在一起
|
||||
for (DocumentChunk c : chunks) {
|
||||
String content = c.getContent();
|
||||
// 分块中不应有孤立的单个列表项（除非只有一个）
|
||||
if (content.contains("1. ") && content.contains("3. ")) {
|
||||
// 这个分块包含了全部3个列表项 → 正确
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ==================== 新增：代码块结构感知 ====================
|
||||
|
||||
@Nested
|
||||
@DisplayName("代码块结构感知")
|
||||
class CodeBlockTests {
|
||||
|
||||
@Test
|
||||
@DisplayName("代码块内部不切分")
|
||||
void codeBlock_notSplitInside() {
|
||||
config.setMaxTokens(60);
|
||||
config.setMaxTokensHard(200);
|
||||
config.setOverlap(20);
|
||||
|
||||
String content =
|
||||
"# 代码示例\n\n" +
|
||||
"以下是配置代码：\n\n" +
|
||||
"```yaml\n" +
|
||||
"server:\n" +
|
||||
" port: 8080\n" +
|
||||
" host: localhost\n" +
|
||||
" timeout: 30s\n" +
|
||||
"```\n\n" +
|
||||
"配置说明结束。";
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(content, "/test/code.md");
|
||||
|
||||
// 代码块应保持完整（未触及硬上限）
|
||||
// 验证：至少有一个分块包含完整的 ```...```
|
||||
boolean foundCompleteBlock = false;
|
||||
for (DocumentChunk c : chunks) {
|
||||
String text = c.getContent();
|
||||
if (text.contains("```yaml") && text.contains("```") &&
|
||||
text.indexOf("```yaml") < text.lastIndexOf("```")) {
|
||||
foundCompleteBlock = true;
|
||||
}
|
||||
}
|
||||
// 可能整体在一个分块中
|
||||
assertTrue(chunks.size() >= 1);
|
||||
}
|
||||
}
|
||||
|
||||
// ==================== 可视化 ====================
|
||||
|
||||
@Nested
|
||||
@DisplayName("可视化 — 打印切分结果")
|
||||
class VisualInspectionTests {
|
||||
|
||||
@Test
|
||||
@DisplayName("模拟运维文档 — 展示新策略效果")
|
||||
void realWorldAIOpsDoc() {
|
||||
config.setMaxTokens(150);
|
||||
config.setMaxTokensHard(200);
|
||||
config.setOverlap(40);
|
||||
|
||||
String doc = """
|
||||
# CPU高负载问题排查指南
|
||||
|
||||
## 问题现象
|
||||
|
||||
服务器CPU使用率持续超过90%，系统响应变慢，用户反馈页面加载超时。
|
||||
监控告警系统连续发出多条CPU使用率告警。
|
||||
|
||||
## 排查步骤
|
||||
|
||||
1. 登录服务器，执行 top 命令查看当前CPU使用率最高的进程。记录进程ID和CPU占用百分比。
|
||||
|
||||
2. 使用 ps aux | grep {进程名} 确认相关服务的运行状态。检查是否有异常进程占用资源。
|
||||
|
||||
3. 查看应用日志，重点关注最近15分钟的ERROR级别日志。使用 tail -n 500 命令。
|
||||
|
||||
4. 检查数据库连接池状态，确认是否有慢查询或连接泄漏。查看慢查询日志。
|
||||
|
||||
5. 检查JVM内存使用情况和GC日志。使用 jstat -gcutil {pid} 1000 命令观察GC频率。
|
||||
|
||||
## 常见原因
|
||||
|
||||
1. 死循环或递归调用导致CPU满载。检查是否有未设置退出条件的循环逻辑。
|
||||
2. 大量正则表达式匹配操作。检查是否有未编译的正则在循环中使用。
|
||||
|
||||
## 解决方案
|
||||
|
||||
根据排查结果采取对应措施：代码问题则回滚或热修复；资源不足则扩容。
|
||||
处理完成后持续观察监控指标30分钟，确认CPU使用率恢复正常。
|
||||
""";
|
||||
|
||||
List<DocumentChunk> chunks = service.chunkDocument(doc, "/kb/cpu_high_usage.md");
|
||||
|
||||
System.out.println("========================================");
|
||||
System.out.println(" Phase 1 新策略效果 — 模拟运维文档");
|
||||
System.out.println(" 配置: maxTokens=150, hard=200, overlap=40");
|
||||
System.out.println(" 总字符数: " + doc.length());
|
||||
System.out.println(" 总分块数: " + chunks.size());
|
||||
System.out.println("========================================\n");
|
||||
|
||||
for (DocumentChunk c : chunks) {
|
||||
System.out.println("┌─ Chunk #" + c.getChunkIndex());
|
||||
System.out.println("│ Title: " + (c.getTitle() != null ? c.getTitle() : "(无)"));
|
||||
System.out.println("│ Range: [" + c.getStartIndex() + "→" + c.getEndIndex() + "] (" + c.getContent().length() + "字符)");
|
||||
// 显示前150字符
|
||||
String preview = c.getContent().length() > 120
|
||||
? c.getContent().substring(0, 120).replace("\n", "\\n") + "..."
|
||||
: c.getContent().replace("\n", "\\n");
|
||||
System.out.println("│ Preview: " + preview);
|
||||
System.out.println("└──────────────────────\n");
|
||||
}
|
||||
|
||||
assertTrue(chunks.size() >= 3, "应产生多个分块");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("中英混排对比 — token vs 字符计数差异")
|
||||
void mixedContentComparison() {
|
||||
config.setMaxTokens(100);
|
||||
config.setMaxTokensHard(150);
|
||||
config.setOverlap(30);
|
||||
|
||||
String chinese = "这是中文内容示范。中文每个字符在LLM中约占用1个token。" +
|
||||
"因此这段文本在上下文窗口中占用的token数较多。" +
|
||||
"继续填充文字以触发切分逻辑，验证中文token估算是否合理。" +
|
||||
"更多中文文本来增加令牌计数。";
|
||||
|
||||
String english = "This is English content. Each word may take one or two tokens. " +
|
||||
"A sentence like this one actually consumes relatively few tokens compared to " +
|
||||
"Chinese characters. More English text to reach the same token count as above. " +
|
||||
"Still need more words because English is very efficient in tokenization. " +
|
||||
"Adding even more content to make this paragraph long enough to test properly.";
|
||||
|
||||
List<DocumentChunk> cnChunks = service.chunkDocument("# CN\n\n" + chinese + "\n\n" + chinese, "/test/cn.md");
|
||||
List<DocumentChunk> enChunks = service.chunkDocument("# EN\n\n" + english + "\n\n" + english, "/test/en.md");
|
||||
|
||||
System.out.println("========================================");
|
||||
System.out.println(" Token 计数对比");
|
||||
System.out.println(" 配置: maxTokens=100, overlap=30");
|
||||
System.out.println("========================================");
|
||||
System.out.println(" 中文文档: " + (chinese.length() * 2) + "字符 → " + cnChunks.size() + "个分块");
|
||||
System.out.println(" 英文文档: " + (english.length() * 2) + "字符 → " + enChunks.size() + "个分块");
|
||||
|
||||
for (DocumentChunk c : cnChunks) {
|
||||
System.out.println(" 中文Chunk#" + c.getChunkIndex() + ": " + c.getContent().length() + "字符");
|
||||
}
|
||||
for (DocumentChunk c : enChunks) {
|
||||
System.out.println(" 英文Chunk#" + c.getChunkIndex() + ": " + c.getContent().length() + "字符");
|
||||
}
|
||||
System.out.println(" ★ 现在中文和英文的分块数更接近（基于 token 而非字符）");
|
||||
System.out.println("========================================");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,237 @@
|
||||
package org.example.service;
|
||||
|
||||
import io.milvus.client.MilvusServiceClient;
|
||||
import io.milvus.grpc.DataType;
|
||||
import io.milvus.grpc.FlushResponse;
|
||||
import io.milvus.grpc.MutationResult;
|
||||
import io.milvus.grpc.SearchResults;
|
||||
import io.milvus.grpc.ShowCollectionsResponse;
|
||||
import io.milvus.common.clientenum.ConsistencyLevelEnum;
|
||||
import io.milvus.param.ConnectParam;
|
||||
import io.milvus.param.IndexType;
|
||||
import io.milvus.param.MetricType;
|
||||
import io.milvus.param.R;
|
||||
import io.milvus.param.RpcStatus;
|
||||
import io.milvus.param.collection.*;
|
||||
import io.milvus.param.dml.InsertParam;
|
||||
import io.milvus.param.dml.SearchParam;
|
||||
import io.milvus.param.index.CreateIndexParam;
|
||||
import io.milvus.response.SearchResultsWrapper;
|
||||
import org.junit.jupiter.api.*;
|
||||
|
||||
import java.util.Arrays;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
@DisplayName("Milvus 连接验证")
|
||||
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
|
||||
class MilvusConnectionTest {
|
||||
|
||||
private static final String COLLECTION = "conn_test";
|
||||
private static final int DIM = 128;
|
||||
|
||||
private static MilvusServiceClient client;
|
||||
|
||||
@BeforeAll
|
||||
static void connect() {
|
||||
String host = envOrDefault("MILVUS_HOST",
|
||||
"in03-4a578da0f27ce9d.serverless.aws-eu-central-1.cloud.zilliz.com");
|
||||
int port = Integer.parseInt(envOrDefault("MILVUS_PORT", "443"));
|
||||
String token = System.getenv("MILVUS_TOKEN");
|
||||
|
||||
assertNotNull(token, "环境变量 MILVUS_TOKEN 未设置");
|
||||
|
||||
ConnectParam connectParam = ConnectParam.newBuilder()
|
||||
.withHost(host)
|
||||
.withPort(port)
|
||||
.withToken(token)
|
||||
.withSecure(true)
|
||||
.withDatabaseName("db_4a578da0f27ce9d")
|
||||
.withConnectTimeout(30, TimeUnit.SECONDS)
|
||||
.build();
|
||||
|
||||
client = new MilvusServiceClient(connectParam);
|
||||
System.out.println("连接目标: " + host + ":" + port);
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void disconnect() {
|
||||
if (client != null) {
|
||||
try {
|
||||
client.dropCollection(DropCollectionParam.newBuilder()
|
||||
.withCollectionName(COLLECTION).build());
|
||||
} catch (Exception ignored) {}
|
||||
client.close();
|
||||
}
|
||||
}
|
||||
|
||||
private static String safeMsg(R<?> resp) {
|
||||
try {
|
||||
return resp.getMessage();
|
||||
} catch (Exception e) {
|
||||
return "(no message)";
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@Order(1)
|
||||
@DisplayName("1. 连接成功 - 能列出 collection")
|
||||
void listCollections() {
|
||||
R<ShowCollectionsResponse> resp = client.showCollections(
|
||||
ShowCollectionsParam.newBuilder().build());
|
||||
|
||||
System.out.println("listCollections status: " + resp.getStatus() + ", msg: " + safeMsg(resp));
|
||||
assertEquals(0, resp.getStatus(), "连接失败，status=" + resp.getStatus());
|
||||
|
||||
List<String> names = resp.getData().getCollectionNamesList();
|
||||
System.out.println("现有 collections: " + names);
|
||||
}
|
||||
|
||||
@Test
|
||||
@Order(2)
|
||||
@DisplayName("2. 创建测试 collection")
|
||||
void createCollection() {
|
||||
client.dropCollection(DropCollectionParam.newBuilder()
|
||||
.withCollectionName(COLLECTION).build());
|
||||
|
||||
FieldType idField = FieldType.newBuilder()
|
||||
.withName("id")
|
||||
.withDataType(DataType.Int64)
|
||||
.withPrimaryKey(true)
|
||||
.withAutoID(true)
|
||||
.build();
|
||||
|
||||
FieldType vectorField = FieldType.newBuilder()
|
||||
.withName("vector")
|
||||
.withDataType(DataType.FloatVector)
|
||||
.withDimension(DIM)
|
||||
.build();
|
||||
|
||||
CollectionSchemaParam schema = CollectionSchemaParam.newBuilder()
|
||||
.addFieldType(idField)
|
||||
.addFieldType(vectorField)
|
||||
.build();
|
||||
|
||||
R<RpcStatus> resp = client.createCollection(
|
||||
CreateCollectionParam.newBuilder()
|
||||
.withCollectionName(COLLECTION)
|
||||
.withSchema(schema)
|
||||
.build());
|
||||
|
||||
System.out.println("createCollection status: " + resp.getStatus() + ", msg: " + safeMsg(resp));
|
||||
assertEquals(0, resp.getStatus(), "创建 collection 失败");
|
||||
}
|
||||
|
||||
@Test
|
||||
@Order(3)
|
||||
@DisplayName("3. 插入数据 + flush")
|
||||
void insertAndFlush() {
|
||||
List<Float> vec1 = makeVector(1.0f);
|
||||
List<Float> vec2 = makeVector(2.0f);
|
||||
List<Float> vec3 = makeVector(3.0f);
|
||||
|
||||
List<InsertParam.Field> fields = Collections.singletonList(
|
||||
new InsertParam.Field("vector", Arrays.asList(vec1, vec2, vec3))
|
||||
);
|
||||
|
||||
R<MutationResult> insertResp = client.insert(
|
||||
InsertParam.newBuilder()
|
||||
.withCollectionName(COLLECTION)
|
||||
.withFields(fields)
|
||||
.build());
|
||||
|
||||
System.out.println("insert status: " + insertResp.getStatus() + ", msg: " + safeMsg(insertResp));
|
||||
assertEquals(0, insertResp.getStatus(), "插入失败");
|
||||
|
||||
// 官方示例要求：insert 后必须 flush，数据才对搜索可见
|
||||
R<FlushResponse> flushResp = client.flush(FlushParam.newBuilder()
|
||||
.withCollectionNames(Collections.singletonList(COLLECTION))
|
||||
.withSyncFlush(true)
|
||||
.withSyncFlushWaitingTimeout(30L)
|
||||
.build());
|
||||
|
||||
System.out.println("flush status: " + flushResp.getStatus() + ", msg: " + safeMsg(flushResp));
|
||||
assertEquals(0, flushResp.getStatus(), "flush 失败");
|
||||
System.out.println("插入 3 条数据并 flush 完成");
|
||||
}
|
||||
|
||||
@Test
|
||||
@Order(4)
|
||||
@DisplayName("4. 创建索引 + 加载")
|
||||
void createIndexAndLoad() {
|
||||
R<RpcStatus> indexResp = client.createIndex(
|
||||
CreateIndexParam.newBuilder()
|
||||
.withCollectionName(COLLECTION)
|
||||
.withFieldName("vector")
|
||||
.withIndexType(IndexType.AUTOINDEX)
|
||||
.withMetricType(MetricType.L2)
|
||||
.build());
|
||||
|
||||
System.out.println("createIndex status: " + indexResp.getStatus() + ", msg: " + safeMsg(indexResp));
|
||||
assertEquals(0, indexResp.getStatus(), "创建索引失败");
|
||||
|
||||
R<RpcStatus> loadResp = client.loadCollection(
|
||||
LoadCollectionParam.newBuilder()
|
||||
.withCollectionName(COLLECTION)
|
||||
.withSyncLoad(true)
|
||||
.withSyncLoadWaitingTimeout(30L)
|
||||
.build());
|
||||
|
||||
System.out.println("load status: " + loadResp.getStatus() + ", msg: " + safeMsg(loadResp));
|
||||
assertEquals(0, loadResp.getStatus(), "加载失败");
|
||||
System.out.println("索引创建 + 加载完成");
|
||||
}
|
||||
|
||||
@Test
|
||||
@Order(5)
|
||||
@DisplayName("5. 向量搜索")
|
||||
void search() throws InterruptedException {
|
||||
Thread.sleep(3000);
|
||||
|
||||
List<Float> queryVec = makeVector(1.1f);
|
||||
|
||||
R<SearchResults> resp = null;
|
||||
for (int retry = 0; retry < 10; retry++) {
|
||||
resp = client.search(
|
||||
SearchParam.newBuilder()
|
||||
.withCollectionName(COLLECTION)
|
||||
.withMetricType(MetricType.L2)
|
||||
.withTopK(2)
|
||||
.withVectors(Collections.singletonList(queryVec))
|
||||
.withVectorFieldName("vector")
|
||||
.withParams("{}")
|
||||
.withConsistencyLevel(ConsistencyLevelEnum.STRONG)
|
||||
.build());
|
||||
|
||||
if (resp.getStatus() == 0) break;
|
||||
System.out.println("search retry " + (retry + 1) + ": status=" + resp.getStatus() + ", msg=" + safeMsg(resp));
|
||||
Thread.sleep(5000);
|
||||
}
|
||||
|
||||
System.out.println("search status: " + resp.getStatus() + ", msg: " + safeMsg(resp));
|
||||
assertEquals(0, resp.getStatus(), "搜索失败");
|
||||
|
||||
SearchResultsWrapper wrapper = new SearchResultsWrapper(resp.getData().getResults());
|
||||
List<SearchResultsWrapper.IDScore> scores = wrapper.getIDScore(0);
|
||||
|
||||
assertFalse(scores.isEmpty(), "搜索结果不应为空");
|
||||
System.out.println("搜索结果 (top " + scores.size() + "):");
|
||||
for (SearchResultsWrapper.IDScore idScore : scores) {
|
||||
System.out.println(" score=" + idScore.getScore() + ", id=" + idScore.getLongID());
|
||||
}
|
||||
}
|
||||
|
||||
private static List<Float> makeVector(float val) {
|
||||
Float[] arr = new Float[DIM];
|
||||
Arrays.fill(arr, val);
|
||||
return Arrays.asList(arr);
|
||||
}
|
||||
|
||||
private static String envOrDefault(String key, String defaultVal) {
|
||||
String val = System.getenv(key);
|
||||
return (val != null && !val.isEmpty()) ? val : defaultVal;
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user