Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 zhuyongxin
|
246c99b954 | ||
|
zhuyongxin
|
f01866c1a2 | ||
|
zhuyongxin
|
6919092b83 | ||
|
zhuyongxin
|
5b827fe90e | ||
|
zhuyongxin
|
b0f288ae36 | ||
|
zhuyongxin
|
1ff7f09d25 | ||
|
zhuyongxin
|
fd89d84fc0 | ||
|
zhuyongxin
|
9050487307 | ||
|
zhuyongxin
|
4f5316d473 |
@@ -0,0 +1,156 @@
|
||||
---
|
||||
name: openspec-apply-change
|
||||
description: Implement tasks from an OpenSpec change. Use when the user wants to start implementing, continue implementation, or work through tasks.
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.3.1"
|
||||
---
|
||||
|
||||
Implement tasks from an OpenSpec change.
|
||||
|
||||
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Select the change**
|
||||
|
||||
If a name is provided, use it. Otherwise:
|
||||
- Infer from conversation context if the user mentioned a change
|
||||
- Auto-select if only one active change exists
|
||||
- If ambiguous, run `openspec list --json` to get available changes and use the **AskUserQuestion tool** to let the user select
|
||||
|
||||
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:apply <other>`).
|
||||
|
||||
2. **Check status to understand the schema**
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
Parse the JSON to understand:
|
||||
- `schemaName`: The workflow being used (e.g., "spec-driven")
|
||||
- Which artifact contains the tasks (typically "tasks" for spec-driven, check status for others)
|
||||
|
||||
3. **Get apply instructions**
|
||||
|
||||
```bash
|
||||
openspec instructions apply --change "<name>" --json
|
||||
```
|
||||
|
||||
This returns:
|
||||
- `contextFiles`: artifact ID -> array of concrete file paths (varies by schema - could be proposal/specs/design/tasks or spec/tests/implementation/docs)
|
||||
- Progress (total, complete, remaining)
|
||||
- Task list with status
|
||||
- Dynamic instruction based on current state
|
||||
|
||||
**Handle states:**
|
||||
- If `state: "blocked"` (missing artifacts): show message, suggest using openspec-continue-change
|
||||
- If `state: "all_done"`: congratulate, suggest archive
|
||||
- Otherwise: proceed to implementation
|
||||
|
||||
4. **Read context files**
|
||||
|
||||
Read every file path listed under `contextFiles` from the apply instructions output.
|
||||
The files depend on the schema being used:
|
||||
- **spec-driven**: proposal, specs, design, tasks
|
||||
- Other schemas: follow the contextFiles from CLI output
|
||||
|
||||
5. **Show current progress**
|
||||
|
||||
Display:
|
||||
- Schema being used
|
||||
- Progress: "N/M tasks complete"
|
||||
- Remaining tasks overview
|
||||
- Dynamic instruction from CLI
|
||||
|
||||
6. **Implement tasks (loop until done or blocked)**
|
||||
|
||||
For each pending task:
|
||||
- Show which task is being worked on
|
||||
- Make the code changes required
|
||||
- Keep changes minimal and focused
|
||||
- Mark task complete in the tasks file: `- [ ]` → `- [x]`
|
||||
- Continue to next task
|
||||
|
||||
**Pause if:**
|
||||
- Task is unclear → ask for clarification
|
||||
- Implementation reveals a design issue → suggest updating artifacts
|
||||
- Error or blocker encountered → report and wait for guidance
|
||||
- User interrupts
|
||||
|
||||
7. **On completion or pause, show status**
|
||||
|
||||
Display:
|
||||
- Tasks completed this session
|
||||
- Overall progress: "N/M tasks complete"
|
||||
- If all done: suggest archive
|
||||
- If paused: explain why and wait for guidance
|
||||
|
||||
**Output During Implementation**
|
||||
|
||||
```
|
||||
## Implementing: <change-name> (schema: <schema-name>)
|
||||
|
||||
Working on task 3/7: <task description>
|
||||
[...implementation happening...]
|
||||
✓ Task complete
|
||||
|
||||
Working on task 4/7: <task description>
|
||||
[...implementation happening...]
|
||||
✓ Task complete
|
||||
```
|
||||
|
||||
**Output On Completion**
|
||||
|
||||
```
|
||||
## Implementation Complete
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Progress:** 7/7 tasks complete ✓
|
||||
|
||||
### Completed This Session
|
||||
- [x] Task 1
|
||||
- [x] Task 2
|
||||
...
|
||||
|
||||
All tasks complete! Ready to archive this change.
|
||||
```
|
||||
|
||||
**Output On Pause (Issue Encountered)**
|
||||
|
||||
```
|
||||
## Implementation Paused
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Progress:** 4/7 tasks complete
|
||||
|
||||
### Issue Encountered
|
||||
<description of the issue>
|
||||
|
||||
**Options:**
|
||||
1. <option 1>
|
||||
2. <option 2>
|
||||
3. Other approach
|
||||
|
||||
What would you like to do?
|
||||
```
|
||||
|
||||
**Guardrails**
|
||||
- Keep going through tasks until done or blocked
|
||||
- Always read context files before starting (from the apply instructions output)
|
||||
- If task is ambiguous, pause and ask before implementing
|
||||
- If implementation reveals issues, pause and suggest artifact updates
|
||||
- Keep code changes minimal and scoped to each task
|
||||
- Update task checkbox immediately after completing each task
|
||||
- Pause on errors, blockers, or unclear requirements - don't guess
|
||||
- Use contextFiles from CLI output, don't assume specific file names
|
||||
|
||||
**Fluid Workflow Integration**
|
||||
|
||||
This skill supports the "actions on a change" model:
|
||||
|
||||
- **Can be invoked anytime**: Before all artifacts are done (if tasks exist), after partial implementation, interleaved with other actions
|
||||
- **Allows artifact updates**: If implementation reveals design issues, suggest updating artifacts - not phase-locked, work fluidly
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
name: openspec-archive-change
|
||||
description: Archive a completed change in the experimental workflow. Use when the user wants to finalize and archive a change after implementation is complete.
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.3.1"
|
||||
---
|
||||
|
||||
Archive a completed change in the experimental workflow.
|
||||
|
||||
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **If no change name provided, prompt for selection**
|
||||
|
||||
Run `openspec list --json` to get available changes. Use the **AskUserQuestion tool** to let the user select.
|
||||
|
||||
Show only active changes (not already archived).
|
||||
Include the schema used for each change if available.
|
||||
|
||||
**IMPORTANT**: Do NOT guess or auto-select a change. Always let the user choose.
|
||||
|
||||
2. **Check artifact completion status**
|
||||
|
||||
Run `openspec status --change "<name>" --json` to check artifact completion.
|
||||
|
||||
Parse the JSON to understand:
|
||||
- `schemaName`: The workflow being used
|
||||
- `artifacts`: List of artifacts with their status (`done` or other)
|
||||
|
||||
**If any artifacts are not `done`:**
|
||||
- Display warning listing incomplete artifacts
|
||||
- Use **AskUserQuestion tool** to confirm user wants to proceed
|
||||
- Proceed if user confirms
|
||||
|
||||
3. **Check task completion status**
|
||||
|
||||
Read the tasks file (typically `tasks.md`) to check for incomplete tasks.
|
||||
|
||||
Count tasks marked with `- [ ]` (incomplete) vs `- [x]` (complete).
|
||||
|
||||
**If incomplete tasks found:**
|
||||
- Display warning showing count of incomplete tasks
|
||||
- Use **AskUserQuestion tool** to confirm user wants to proceed
|
||||
- Proceed if user confirms
|
||||
|
||||
**If no tasks file exists:** Proceed without task-related warning.
|
||||
|
||||
4. **Assess delta spec sync state**
|
||||
|
||||
Check for delta specs at `openspec/changes/<name>/specs/`. If none exist, proceed without sync prompt.
|
||||
|
||||
**If delta specs exist:**
|
||||
- Compare each delta spec with its corresponding main spec at `openspec/specs/<capability>/spec.md`
|
||||
- Determine what changes would be applied (adds, modifications, removals, renames)
|
||||
- Show a combined summary before prompting
|
||||
|
||||
**Prompt options:**
|
||||
- If changes needed: "Sync now (recommended)", "Archive without syncing"
|
||||
- If already synced: "Archive now", "Sync anyway", "Cancel"
|
||||
|
||||
If user chooses sync, use Task tool (subagent_type: "general-purpose", prompt: "Use Skill tool to invoke openspec-sync-specs for change '<name>'. Delta spec analysis: <include the analyzed delta spec summary>"). Proceed to archive regardless of choice.
|
||||
|
||||
5. **Perform the archive**
|
||||
|
||||
Create the archive directory if it doesn't exist:
|
||||
```bash
|
||||
mkdir -p openspec/changes/archive
|
||||
```
|
||||
|
||||
Generate target name using current date: `YYYY-MM-DD-<change-name>`
|
||||
|
||||
**Check if target already exists:**
|
||||
- If yes: Fail with error, suggest renaming existing archive or using different date
|
||||
- If no: Move the change directory to archive
|
||||
|
||||
```bash
|
||||
mv openspec/changes/<name> openspec/changes/archive/YYYY-MM-DD-<name>
|
||||
```
|
||||
|
||||
6. **Display summary**
|
||||
|
||||
Show archive completion summary including:
|
||||
- Change name
|
||||
- Schema that was used
|
||||
- Archive location
|
||||
- Whether specs were synced (if applicable)
|
||||
- Note about any warnings (incomplete artifacts/tasks)
|
||||
|
||||
**Output On Success**
|
||||
|
||||
```
|
||||
## Archive Complete
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Archived to:** openspec/changes/archive/YYYY-MM-DD-<name>/
|
||||
**Specs:** ✓ Synced to main specs (or "No delta specs" or "Sync skipped")
|
||||
|
||||
All artifacts complete. All tasks complete.
|
||||
```
|
||||
|
||||
**Guardrails**
|
||||
- Always prompt for change selection if not provided
|
||||
- Use artifact graph (openspec status --json) for completion checking
|
||||
- Don't block archive on warnings - just inform and confirm
|
||||
- Preserve .openspec.yaml when moving to archive (it moves with the directory)
|
||||
- Show clear summary of what happened
|
||||
- If sync is requested, use openspec-sync-specs approach (agent-driven)
|
||||
- If delta specs exist, always run the sync assessment and show the combined summary before prompting
|
||||
@@ -0,0 +1,288 @@
|
||||
---
|
||||
name: openspec-explore
|
||||
description: Enter explore mode - a thinking partner for exploring ideas, investigating problems, and clarifying requirements. Use when the user wants to think through something before or during a change.
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.3.1"
|
||||
---
|
||||
|
||||
Enter explore mode. Think deeply. Visualize freely. Follow the conversation wherever it goes.
|
||||
|
||||
**IMPORTANT: Explore mode is for thinking, not implementing.** You may read files, search code, and investigate the codebase, but you must NEVER write code or implement features. If the user asks you to implement something, remind them to exit explore mode first and create a change proposal. You MAY create OpenSpec artifacts (proposals, designs, specs) if the user asks—that's capturing thinking, not implementing.
|
||||
|
||||
**This is a stance, not a workflow.** There are no fixed steps, no required sequence, no mandatory outputs. You're a thinking partner helping the user explore.
|
||||
|
||||
---
|
||||
|
||||
## The Stance
|
||||
|
||||
- **Curious, not prescriptive** - Ask questions that emerge naturally, don't follow a script
|
||||
- **Open threads, not interrogations** - Surface multiple interesting directions and let the user follow what resonates. Don't funnel them through a single path of questions.
|
||||
- **Visual** - Use ASCII diagrams liberally when they'd help clarify thinking
|
||||
- **Adaptive** - Follow interesting threads, pivot when new information emerges
|
||||
- **Patient** - Don't rush to conclusions, let the shape of the problem emerge
|
||||
- **Grounded** - Explore the actual codebase when relevant, don't just theorize
|
||||
|
||||
---
|
||||
|
||||
## What You Might Do
|
||||
|
||||
Depending on what the user brings, you might:
|
||||
|
||||
**Explore the problem space**
|
||||
- Ask clarifying questions that emerge from what they said
|
||||
- Challenge assumptions
|
||||
- Reframe the problem
|
||||
- Find analogies
|
||||
|
||||
**Investigate the codebase**
|
||||
- Map existing architecture relevant to the discussion
|
||||
- Find integration points
|
||||
- Identify patterns already in use
|
||||
- Surface hidden complexity
|
||||
|
||||
**Compare options**
|
||||
- Brainstorm multiple approaches
|
||||
- Build comparison tables
|
||||
- Sketch tradeoffs
|
||||
- Recommend a path (if asked)
|
||||
|
||||
**Visualize**
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Use ASCII diagrams liberally │
|
||||
├─────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌────────┐ ┌────────┐ │
|
||||
│ │ State │────────▶│ State │ │
|
||||
│ │ A │ │ B │ │
|
||||
│ └────────┘ └────────┘ │
|
||||
│ │
|
||||
│ System diagrams, state machines, │
|
||||
│ data flows, architecture sketches, │
|
||||
│ dependency graphs, comparison tables │
|
||||
│ │
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Surface risks and unknowns**
|
||||
- Identify what could go wrong
|
||||
- Find gaps in understanding
|
||||
- Suggest spikes or investigations
|
||||
|
||||
---
|
||||
|
||||
## OpenSpec Awareness
|
||||
|
||||
You have full context of the OpenSpec system. Use it naturally, don't force it.
|
||||
|
||||
### Check for context
|
||||
|
||||
At the start, quickly check what exists:
|
||||
```bash
|
||||
openspec list --json
|
||||
```
|
||||
|
||||
This tells you:
|
||||
- If there are active changes
|
||||
- Their names, schemas, and status
|
||||
- What the user might be working on
|
||||
|
||||
### When no change exists
|
||||
|
||||
Think freely. When insights crystallize, you might offer:
|
||||
|
||||
- "This feels solid enough to start a change. Want me to create a proposal?"
|
||||
- Or keep exploring - no pressure to formalize
|
||||
|
||||
### When a change exists
|
||||
|
||||
If the user mentions a change or you detect one is relevant:
|
||||
|
||||
1. **Read existing artifacts for context**
|
||||
- `openspec/changes/<name>/proposal.md`
|
||||
- `openspec/changes/<name>/design.md`
|
||||
- `openspec/changes/<name>/tasks.md`
|
||||
- etc.
|
||||
|
||||
2. **Reference them naturally in conversation**
|
||||
- "Your design mentions using Redis, but we just realized SQLite fits better..."
|
||||
- "The proposal scopes this to premium users, but we're now thinking everyone..."
|
||||
|
||||
3. **Offer to capture when decisions are made**
|
||||
|
||||
| Insight Type | Where to Capture |
|
||||
|----------------------------|--------------------------------|
|
||||
| New requirement discovered | `specs/<capability>/spec.md` |
|
||||
| Requirement changed | `specs/<capability>/spec.md` |
|
||||
| Design decision made | `design.md` |
|
||||
| Scope changed | `proposal.md` |
|
||||
| New work identified | `tasks.md` |
|
||||
| Assumption invalidated | Relevant artifact |
|
||||
|
||||
Example offers:
|
||||
- "That's a design decision. Capture it in design.md?"
|
||||
- "This is a new requirement. Add it to specs?"
|
||||
- "This changes scope. Update the proposal?"
|
||||
|
||||
4. **The user decides** - Offer and move on. Don't pressure. Don't auto-capture.
|
||||
|
||||
---
|
||||
|
||||
## What You Don't Have To Do
|
||||
|
||||
- Follow a script
|
||||
- Ask the same questions every time
|
||||
- Produce a specific artifact
|
||||
- Reach a conclusion
|
||||
- Stay on topic if a tangent is valuable
|
||||
- Be brief (this is thinking time)
|
||||
|
||||
---
|
||||
|
||||
## Handling Different Entry Points
|
||||
|
||||
**User brings a vague idea:**
|
||||
```
|
||||
User: I'm thinking about adding real-time collaboration
|
||||
|
||||
You: Real-time collab is a big space. Let me think about this...
|
||||
|
||||
COLLABORATION SPECTRUM
|
||||
════════════════════════════════════════════
|
||||
|
||||
Awareness Coordination Sync
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌────────┐ ┌────────┐ ┌────────┐
|
||||
│Presence│ │Cursors │ │ CRDT │
|
||||
│ "3 │ │ Multi │ │Conflict│
|
||||
│online" │ │ select │ │ free │
|
||||
└────────┘ └────────┘ └────────┘
|
||||
│ │ │
|
||||
trivial moderate complex
|
||||
|
||||
Where's your head at?
|
||||
```
|
||||
|
||||
**User brings a specific problem:**
|
||||
```
|
||||
User: The auth system is a mess
|
||||
|
||||
You: [reads codebase]
|
||||
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ CURRENT AUTH FLOW │
|
||||
└─────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────┼─────────────┐
|
||||
▼ ▼ ▼
|
||||
┌─────────┐ ┌─────────┐ ┌─────────┐
|
||||
│ Google │ │ GitHub │ │ Email │
|
||||
│ OAuth │ │ OAuth │ │ Magic │
|
||||
└────┬────┘ └────┬────┘ └────┬────┘
|
||||
│ │ │
|
||||
└─────────────┼─────────────┘
|
||||
▼
|
||||
┌───────────┐
|
||||
│ Session │
|
||||
└─────┬─────┘
|
||||
│
|
||||
▼
|
||||
┌───────────┐
|
||||
│ Perms │
|
||||
└───────────┘
|
||||
|
||||
I see three tangles. Which one's burning?
|
||||
```
|
||||
|
||||
**User is stuck mid-implementation:**
|
||||
```
|
||||
User: /opsx:explore add-auth-system
|
||||
The OAuth integration is more complex than expected
|
||||
|
||||
You: [reads change artifacts]
|
||||
|
||||
You're on task 4: "Implement OAuth flow"
|
||||
|
||||
Let me trace what's involved...
|
||||
|
||||
[draws diagram, explores options, suggests paths]
|
||||
|
||||
Want to update the design to reflect this?
|
||||
Or add a spike task to investigate?
|
||||
```
|
||||
|
||||
**User wants to compare options:**
|
||||
```
|
||||
User: Should we use Postgres or SQLite?
|
||||
|
||||
You: Generic answer is boring. What's the context?
|
||||
|
||||
User: A CLI tool that tracks local dev environments
|
||||
|
||||
You: That changes everything.
|
||||
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ CLI TOOL DATA STORAGE │
|
||||
└─────────────────────────────────────────────────┘
|
||||
|
||||
Key constraints:
|
||||
• No daemon running
|
||||
• Must work offline
|
||||
• Single user
|
||||
|
||||
SQLite Postgres
|
||||
Deployment embedded ✓ needs server ✗
|
||||
Offline yes ✓ no ✗
|
||||
Single file yes ✓ no ✗
|
||||
|
||||
SQLite. Not even close.
|
||||
|
||||
Unless... is there a sync component?
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Ending Discovery
|
||||
|
||||
There's no required ending. Discovery might:
|
||||
|
||||
- **Flow into a proposal**: "Ready to start? I can create a change proposal."
|
||||
- **Result in artifact updates**: "Updated design.md with these decisions"
|
||||
- **Just provide clarity**: User has what they need, moves on
|
||||
- **Continue later**: "We can pick this up anytime"
|
||||
|
||||
When it feels like things are crystallizing, you might summarize:
|
||||
|
||||
```
|
||||
## What We Figured Out
|
||||
|
||||
**The problem**: [crystallized understanding]
|
||||
|
||||
**The approach**: [if one emerged]
|
||||
|
||||
**Open questions**: [if any remain]
|
||||
|
||||
**Next steps** (if ready):
|
||||
- Create a change proposal
|
||||
- Keep exploring: just keep talking
|
||||
```
|
||||
|
||||
But this summary is optional. Sometimes the thinking IS the value.
|
||||
|
||||
---
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Don't implement** - Never write code or implement features. Creating OpenSpec artifacts is fine, writing application code is not.
|
||||
- **Don't fake understanding** - If something is unclear, dig deeper
|
||||
- **Don't rush** - Discovery is thinking time, not task time
|
||||
- **Don't force structure** - Let patterns emerge naturally
|
||||
- **Don't auto-capture** - Offer to save insights, don't just do it
|
||||
- **Do visualize** - A good diagram is worth many paragraphs
|
||||
- **Do explore the codebase** - Ground discussions in reality
|
||||
- **Do question assumptions** - Including the user's and your own
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
name: openspec-propose
|
||||
description: Propose a new change with all artifacts generated in one step. Use when the user wants to quickly describe what they want to build and get a complete proposal with design, specs, and tasks ready for implementation.
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.3.1"
|
||||
---
|
||||
|
||||
Propose a new change - create the change and generate all artifacts in one step.
|
||||
|
||||
I'll create a change with artifacts:
|
||||
- proposal.md (what & why)
|
||||
- design.md (how)
|
||||
- tasks.md (implementation steps)
|
||||
|
||||
When ready to implement, run /opsx:apply
|
||||
|
||||
---
|
||||
|
||||
**Input**: The user's request should include a change name (kebab-case) OR a description of what they want to build.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **If no clear input provided, ask what they want to build**
|
||||
|
||||
Use the **AskUserQuestion tool** (open-ended, no preset options) to ask:
|
||||
> "What change do you want to work on? Describe what you want to build or fix."
|
||||
|
||||
From their description, derive a kebab-case name (e.g., "add user authentication" → `add-user-auth`).
|
||||
|
||||
**IMPORTANT**: Do NOT proceed without understanding what the user wants to build.
|
||||
|
||||
2. **Create the change directory**
|
||||
```bash
|
||||
openspec new change "<name>"
|
||||
```
|
||||
This creates a scaffolded change at `openspec/changes/<name>/` with `.openspec.yaml`.
|
||||
|
||||
3. **Get the artifact build order**
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
Parse the JSON to get:
|
||||
- `applyRequires`: array of artifact IDs needed before implementation (e.g., `["tasks"]`)
|
||||
- `artifacts`: list of all artifacts with their status and dependencies
|
||||
|
||||
4. **Create artifacts in sequence until apply-ready**
|
||||
|
||||
Use the **TodoWrite tool** to track progress through the artifacts.
|
||||
|
||||
Loop through artifacts in dependency order (artifacts with no pending dependencies first):
|
||||
|
||||
a. **For each artifact that is `ready` (dependencies satisfied)**:
|
||||
- Get instructions:
|
||||
```bash
|
||||
openspec instructions <artifact-id> --change "<name>" --json
|
||||
```
|
||||
- The instructions JSON includes:
|
||||
- `context`: Project background (constraints for you - do NOT include in output)
|
||||
- `rules`: Artifact-specific rules (constraints for you - do NOT include in output)
|
||||
- `template`: The structure to use for your output file
|
||||
- `instruction`: Schema-specific guidance for this artifact type
|
||||
- `outputPath`: Where to write the artifact
|
||||
- `dependencies`: Completed artifacts to read for context
|
||||
- Read any completed dependency files for context
|
||||
- Create the artifact file using `template` as the structure
|
||||
- Apply `context` and `rules` as constraints - but do NOT copy them into the file
|
||||
- Show brief progress: "Created <artifact-id>"
|
||||
|
||||
b. **Continue until all `applyRequires` artifacts are complete**
|
||||
- After creating each artifact, re-run `openspec status --change "<name>" --json`
|
||||
- Check if every artifact ID in `applyRequires` has `status: "done"` in the artifacts array
|
||||
- Stop when all `applyRequires` artifacts are done
|
||||
|
||||
c. **If an artifact requires user input** (unclear context):
|
||||
- Use **AskUserQuestion tool** to clarify
|
||||
- Then continue with creation
|
||||
|
||||
5. **Show final status**
|
||||
```bash
|
||||
openspec status --change "<name>"
|
||||
```
|
||||
|
||||
**Output**
|
||||
|
||||
After completing all artifacts, summarize:
|
||||
- Change name and location
|
||||
- List of artifacts created with brief descriptions
|
||||
- What's ready: "All artifacts created! Ready for implementation."
|
||||
- Prompt: "Run `/opsx:apply` or ask me to implement to start working on the tasks."
|
||||
|
||||
**Artifact Creation Guidelines**
|
||||
|
||||
- Follow the `instruction` field from `openspec instructions` for each artifact type
|
||||
- The schema defines what each artifact should contain - follow it
|
||||
- Read dependency artifacts for context before creating new ones
|
||||
- Use `template` as the structure for your output file - fill in its sections
|
||||
- **IMPORTANT**: `context` and `rules` are constraints for YOU, not content for the file
|
||||
- Do NOT copy `<context>`, `<rules>`, `<project_context>` blocks into the artifact
|
||||
- These guide what you write, but should never appear in the output
|
||||
|
||||
**Guardrails**
|
||||
- Create ALL artifacts needed for implementation (as defined by schema's `apply.requires`)
|
||||
- Always read dependency artifacts before creating a new one
|
||||
- If context is critically unclear, ask the user - but prefer making reasonable decisions to keep momentum
|
||||
- If a change with that name already exists, ask if user wants to continue it or create a new one
|
||||
- Verify each artifact file exists after writing before proceeding to next
|
||||
@@ -56,3 +56,7 @@ uploads/
|
||||
/server.pid
|
||||
.claude/settings.local.json
|
||||
.opencode/plugins/emdash-notifications.js
|
||||
|
||||
### Windows / Runtime Artifacts
|
||||
*.stackdump
|
||||
NUL
|
||||
|
||||
@@ -1,29 +0,0 @@
|
||||
Stack trace:
|
||||
Frame Function Args
|
||||
0007FFFFB920 00021005FE8E (000210285F68, 00021026AB6E, 000000000000, 0007FFFFA820) msys-2.0.dll+0x1FE8E
|
||||
0007FFFFB920 0002100467F9 (000000000000, 000000000000, 000000000000, 0007FFFFBBF8) msys-2.0.dll+0x67F9
|
||||
0007FFFFB920 000210046832 (000210286019, 0007FFFFB7D8, 000000000000, 000000000000) msys-2.0.dll+0x6832
|
||||
0007FFFFB920 000210068CF6 (000000000000, 000000000000, 000000000000, 000000000000) msys-2.0.dll+0x28CF6
|
||||
0007FFFFB920 000210068E24 (0007FFFFB930, 000000000000, 000000000000, 000000000000) msys-2.0.dll+0x28E24
|
||||
0007FFFFBC00 00021006A225 (0007FFFFB930, 000000000000, 000000000000, 000000000000) msys-2.0.dll+0x2A225
|
||||
End of stack trace
|
||||
Loaded modules:
|
||||
000100400000 bash.exe
|
||||
7FF9B93D0000 ntdll.dll
|
||||
7FF9B79A0000 KERNEL32.DLL
|
||||
7FF9B6860000 KERNELBASE.dll
|
||||
7FF9B8740000 USER32.dll
|
||||
7FF9B6830000 win32u.dll
|
||||
7FF9B84F0000 GDI32.dll
|
||||
7FF9B6CD0000 gdi32full.dll
|
||||
7FF9B6790000 msvcp_win.dll
|
||||
000210040000 msys-2.0.dll
|
||||
7FF9B7000000 ucrtbase.dll
|
||||
7FF9B7370000 advapi32.dll
|
||||
7FF9B8E40000 msvcrt.dll
|
||||
7FF9B85B0000 sechost.dll
|
||||
7FF9B6FD0000 bcrypt.dll
|
||||
7FF9B90F0000 RPCRT4.dll
|
||||
7FF9B5F20000 CRYPTBASE.DLL
|
||||
7FF9B6710000 bcryptPrimitives.dll
|
||||
7FF9B86E0000 IMM32.DLL
|
||||
@@ -4,6 +4,7 @@
|
||||
|
||||
| 日期 | slug | 领域 | 关键词 | 状态 |
|
||||
|---|---|---|---|---|
|
||||
| 2026-07-03 | mvp-demo-trace-acceptance | MVP Demo/trace/acceptance | mvp-demo, trace API, diagnosis_session, agent_step, tool_invocation, feedback | openspec/changes/archive/2026-07-03-mvp-demo-trace-acceptance | archived |
|
||||
| 2026-05-29 | chatmodel-abstraction | 解耦/多模型路由 | ChatModel, EmbeddingModel, DeepSeek, BGE-M3, SiliconFlow, Spring AI | archived |
|
||||
| 2026-06-23 | phase1-infrastructure | 基础设施/文档管理 | MySQL, Redis, Milvus, Flyway, JPA, 向量检索, 类别过滤 | archived |
|
||||
| 2026-06-24 | lookup-knowledge-integration | 知识库检索 | L0精确匹配, L1语义检索, frontmatter, 混合检索 | archived |
|
||||
@@ -12,3 +13,4 @@
|
||||
| 2026-06-29 | confidence-feedback | 质量评估/反馈机制 | evidence_score, selfEvaluation, feedback, useful, not_useful, case_library, BAD_CASE, tool_invocation规则引擎, 反馈按钮, sessionId回传 | openspec/changes/confidence-feedback | archived |
|
||||
| 2026-06-30 | session-dedup-knowledge-map | 去重/知识图谱 | RetrievedDocTracker, KnowledgeDomainService, knowledge_domain, covers, whenToRetrieve, Planner注入, ISS-001 | openspec/changes/archive/2026-06-30-session-dedup-knowledge-map | archived |
|
||||
| 2026-07-01 | executor-action-memory-relevance | 检索质量/行动记忆 | relevanceLevel, completenessHint, Min-Max归一化, RetrievedDocTracker域级记录, Executor检索约束, ISS-002 | openspec/changes/archive/2026-07-01-executor-action-memory-relevance | archived |
|
||||
| 2026-07-02 | chat-verifier-agent | Chat质量门禁/可追溯验证 | Verifier, groundedness_score, facts_checked, evidence_refs, tool_trace_summary, self_evaluation | openspec/changes/archive/2026-07-03-chat-verifier-agent | archived |
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
# Acceptance: chat-verifier-agent
|
||||
|
||||
## Classification
|
||||
|
||||
standard
|
||||
|
||||
## Task Status
|
||||
|
||||
| Task | Status | Notes |
|
||||
| --- | --- | --- |
|
||||
| Verifier prompt | Done | Strict JSON schema, verdict matrix, fact classifications, and `evidence_refs` are defined. |
|
||||
| VerifierInputHook | Done | Explicit verifier payload replaces raw conversation history. |
|
||||
| ChatService integration | Done | Planner, executor, and verifier are called explicitly with max two rounds. |
|
||||
| Verdict routing | Done | PASS, LOW_CONFID, and REJECT paths are handled in code. |
|
||||
| Trace summary | Done | Evidence summaries include `trace_ref` and `source_invocation_ids`. |
|
||||
| self_evaluation merge | Done | `rule_evaluation` and `verifier_evaluation` are preserved independently. |
|
||||
| Verifier observability | Done | `verifier_evaluation` persists facts, evidence refs, trace summary, rationale, score, and round. |
|
||||
|
||||
## Static Verification
|
||||
|
||||
- [x] OpenSpec artifacts exist: `proposal.md`, `design.md`, `specs/chat-verifier-agent/spec.md`, `tasks.md`, `.committed`.
|
||||
- [x] `change.json` exists and has `metadata.status = committed`.
|
||||
- [x] `.archive-ready` exists.
|
||||
- [x] devflow archive-prep files exist: `brief.md`, `evidence.md`, `decisions.md`, `acceptance.md`.
|
||||
- [x] `devflow/index.md` contains `chat-verifier-agent` with status `archived`.
|
||||
|
||||
## Script Verification
|
||||
|
||||
- [x] `mvn -q -DskipTests compile` passed.
|
||||
|
||||
## Runtime Verification
|
||||
|
||||
- [x] POST `/api/chat` with a complex question returned successfully.
|
||||
- [x] Runtime session `9138f064` showed planner, executor, and verifier execution in logs.
|
||||
- [x] Runtime session `9138f064` wrote `verifier_evaluation.verdict = LOW_CONFID`.
|
||||
- [x] Runtime session `9138f064` wrote `facts_checked[*].evidence_refs`.
|
||||
- [x] Runtime session `9138f064` wrote `tool_trace_summary[*].source_invocation_ids`.
|
||||
- [x] LOW_CONFID final answer included disclaimer and verifier-derived evidence gaps.
|
||||
|
||||
## Unverified
|
||||
|
||||
| Scenario | Reason | Risk | Follow-up |
|
||||
| --- | --- | --- | --- |
|
||||
| PASS runtime path | The exercised complex runtime case produced LOW_CONFID. | Low; PASS routing is simple pass-through after parsed verifier decision. | Add a fixture or deterministic verifier test if this becomes product-critical. |
|
||||
| REJECT runtime path | No forced contradiction case was run after traceability changes. | Medium; REJECT is the safety-critical degraded path. | Add a targeted test with a fabricated claim and evidence contradiction. |
|
||||
| Document-path-level evidence mapping | Current implementation records invocation ids and source document labels, not guaranteed canonical document paths for every retrieval mode. | Low for current audit need; medium for future UI drill-down. | Extend retrieval details with canonical document paths in a later change. |
|
||||
|
||||
## Remaining Risks
|
||||
|
||||
1. Verifier output still depends on model compliance with JSON schema; code falls back to LOW_CONFID on missing or invalid output.
|
||||
2. `AgentLoggingHook` is shared by several agent paths; current changes preserve compile and runtime behavior but should be watched in AiOps flows.
|
||||
3. `SupervisorAgent` construction remains as legacy residue in `ChatService`; runtime orchestration is explicit, but a later cleanup should remove unused supervisor construction.
|
||||
|
||||
## Archive State
|
||||
|
||||
- [x] OpenSpec change is archive-ready.
|
||||
- [x] OpenSpec change has been moved to `openspec/changes/archive/2026-07-03-chat-verifier-agent/`.
|
||||
- [x] Main spec exists at `openspec/specs/chat-verifier-agent/spec.md`.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Brief: chat-verifier-agent
|
||||
|
||||
## Background
|
||||
|
||||
The complex Chat path previously returned Executor answers without a synchronous quality gate. Existing rule scoring was asynchronous and post-hoc, so it could not prevent unsupported answers from reaching users.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Add a Verifier Agent after Executor in the complex chat path.
|
||||
2. Require structured verifier output with `PASS`, `LOW_CONFID`, or `REJECT`.
|
||||
3. Route final user output in code based on verifier verdict.
|
||||
4. Persist verifier results under `diagnosis_session.self_evaluation.verifier_evaluation`.
|
||||
5. Preserve rule scoring under `rule_evaluation`.
|
||||
6. Make verifier decisions traceable to real tool invocations through `evidence_refs` and `source_invocation_ids`.
|
||||
|
||||
## Scope
|
||||
|
||||
- `ChatService`: explicit `planner -> executor -> verifier` orchestration, max two rounds, verdict routing, retry context, verifier persistence.
|
||||
- `VerifierInputHook`: explicit verifier input payload.
|
||||
- `ToolTraceSummaryService`: evidence summary from persisted tool calls.
|
||||
- `VerifierContextHolder`: round-local verifier context.
|
||||
- `SelfEvaluationMergeService`: safe JSON merge for evaluation channels.
|
||||
- `AgentLoggingHook`: concise verifier thought and fuller structured output retention.
|
||||
- `chat-verifier-prompt.md`: verifier contract, verdict matrix, and traceability schema.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Verifier does not call tools.
|
||||
- Verifier does not rewrite Executor output.
|
||||
- Single-agent chat path remains outside this change.
|
||||
- No database schema migration is included.
|
||||
- Document-path-level evidence attribution is deferred; current traceability is invocation-level with source document labels.
|
||||
|
||||
## Related OpenSpec
|
||||
|
||||
`openspec/changes/archive/2026-07-03-chat-verifier-agent/`
|
||||
@@ -0,0 +1,123 @@
|
||||
# Decisions: chat-verifier-agent
|
||||
|
||||
## 过程日志
|
||||
|
||||
### Clarify 阶段
|
||||
|
||||
**入口摘要**: 在 Chat 多 Agent 链路中新增 Verifier Agent，作为 Executor 输出后的质量门禁，做事实核查。
|
||||
|
||||
**slug**: `chat-verifier-agent`
|
||||
|
||||
**规模分档**: standard
|
||||
|
||||
### Context 阶段
|
||||
|
||||
**devflow/index.md 使用状态**: 已命中。前序 change `executor-action-memory-relevance`（archived）提供了 Chat 多 Agent 当前链路（Supervisor → Planner → Executor）。
|
||||
|
||||
**不能违反的历史决策**:
|
||||
1. Executor 已有完整的行动记忆和归一化质量等级，Verifier 不需要重复验证检索质量
|
||||
2. Chat Supervisor 的职责是调度，Verifier 作为子 Agent 加入后不改变 Supervisor 的定位
|
||||
3. 已有 evidence_score 做事后评分，Verifier 是事前门禁，两者不冲突
|
||||
|
||||
**需进入 OpenSpec 的上下文点**:
|
||||
1. Verifier 不需要工具调用，只是一个质量核查 Agent
|
||||
2. Verifier 需要访问 Executor 的输出 + 工具调用记录
|
||||
3. Supervisor prompt 需要重写以包含 Verifier 调度规则
|
||||
4. groundedness_score 的阈值需要在代码中定义
|
||||
|
||||
### Grill 阶段 — Question Pool
|
||||
|
||||
| # | 维度 | 问题 | 模式 | 状态 |
|
||||
|---|------|------|------|------|
|
||||
| Q1 | 术语 | evidence_score（事后评分）与 Verifier（事前门禁）职责是否冲突？ | evidence-driven | 已解决 |
|
||||
| Q2 | 边界 | Verifier 需要的"工具调用记录"在 SupervisorAgent 中是否自动传递？ | evidence-driven | 已解决 |
|
||||
| Q3 | 边界 | LOW_CONFID < 0.5 回调 Planner 后的新输出是否再次走 Verifier？循环上限多少？ | user-interview | 已解决 |
|
||||
| Q4 | 验收 | Verifier 判决结果如何可观测？是否写入 agent_step 或 tool_invocation？ | user-interview | 已解决 |
|
||||
| Q5 | 验收 | 当前 Supervisor 硬编码 prompt 是否支持多 Agent 路由变更？ | evidence-driven | 已解决 |
|
||||
| Q6 | 技术 | Verifier 如何隔离 Executor 的中间推理过程，只看到干净的 query + tool 记录 + 最终答案？ | user-interview | 已解决 |
|
||||
| Q7 | 验收 | groundedness_score 阈值（0.5）是否需要配置化？ | user-interview | 已解决 |
|
||||
|
||||
### Evidence-driven 结论
|
||||
|
||||
| 结论 | 证据来源 | 是否已汇报用户 |
|
||||
|------|---------|-------------|
|
||||
| evidence_score（异步事后）与 Verifier（同步事前门禁）不冲突 | EvaluationService.java: @Async 注解 | 已汇报 |
|
||||
| SupervisorAgent 自动传递完整对话状态，Verifier 无需额外传递工具记录 | Spring AI Alibaba SupervisorAgent 实现 | 已汇报 |
|
||||
| Supervisor prompt 为字符串字面量，直接修改即可 | ChatService.java:353 .systemPrompt("...") | 已汇报 |
|
||||
|
||||
### User-interview 记录
|
||||
|
||||
| 问题 | 用户原话 | 确认状态 | OpenSpec 回写 |
|
||||
|------|---------|---------|-------------|
|
||||
| Q3: LOW_CONFID < 0.5 回调 Planner 循环上限？ | "可以，回调一次" | 已确认 | 已回写 proposal |
|
||||
| Q4: Verifier 判决写入哪里做可观测？ | "可以"（写入 diagnosis_session.self_evaluation JSON） | 已确认 | 已回写 proposal |
|
||||
| Q6: Verifier 如何隔离 Executor 中间推理？ | "用 MessagesModelHook 过滤 messages" | 已确认 | 已回写 design |
|
||||
| Q7: groundedness_score 阈值是否需要配置化？ | "需要配置化" | 已确认 | 已回写 design |
|
||||
|
||||
### Specify 阶段 — Cross-Artifact 对齐检查
|
||||
|
||||
| 上游 → 下游 | 检查内容 | 状态 |
|
||||
|---|---|---|
|
||||
| proposal → design | 范围、约束、关键承诺是否进入 design | 已对齐 |
|
||||
| design → specs | 关键决策、模块地图是否进入 specs | 已对齐 |
|
||||
| specs → tasks | 可观察行为是否被 tasks 覆盖为可执行切片 | 已对齐 |
|
||||
|
||||
**接口影响分级**：
|
||||
- buildChatVerifierAgent() 新增方法 → L1（内部方法，无外部消费者）
|
||||
- VerifierInputHook 类 → L1（内部 Hook，无外部消费者）
|
||||
- Supervisor prompt 重写 → L1（仅影响 Chat 多 Agent 内部调度）
|
||||
- subAgents 列表变更 → L1（Supervisor 内部配置）
|
||||
- verifier.low-confidence-threshold 配置 → L1（新增配置项，不改已有配置）
|
||||
|
||||
### Audit 阶段
|
||||
|
||||
**模块链路**：
|
||||
|
||||
```
|
||||
用户 → Supervisor → Planner(步骤) → Executor(答案+工具记录)
|
||||
│
|
||||
Supervisor 调用 Verifier
|
||||
│
|
||||
[VerifierInputHook BEFORE_MODEL]
|
||||
├─ 保留：system prompt + user query
|
||||
├─ 保留：tool call 记录（输入+返回）
|
||||
├─ 保留：Executor 最终答案
|
||||
└─ 去除：Executor 中间推理、Planner 规划过程
|
||||
│
|
||||
Verifier 判决
|
||||
│
|
||||
┌─── PASS ───→ 直接输出
|
||||
├─── LOW_CONFID≥0.5 → 带声明输出
|
||||
├─── LOW_CONFID<0.5 → 回调 Planner(一次)
|
||||
└─── REJECT → 降级输出
|
||||
│
|
||||
写入 self_evaluation JSON
|
||||
```
|
||||
|
||||
**架构风险评估**（5 句以内）：
|
||||
1. Verifier 是轻量 Agent（无工具、无外部依赖），架构风险低。
|
||||
2. MessagesModelHook 纯过滤逻辑，不引入新数据源。
|
||||
3. LOW_CONFID 分级处理 + 回调仅一次的设计，避免无限循环风险。
|
||||
4. REJECT 降级确保编造内容不到达用户。
|
||||
5. 审计结论不影响现有 design/tasks，无需回写。
|
||||
|
||||
### 关键取舍
|
||||
|
||||
- 决策：LOW_CONFID < 0.5 回调 Planner 一次
|
||||
- 原因：给系统一次修正机会，但避免无限循环
|
||||
- 影响：Supervisor prompt 需维护"已回调"状态
|
||||
- 风险接受：用户已确认
|
||||
|
||||
- 决策：Verifier 判决写入 diagnosis_session.self_evaluation JSON
|
||||
- 原因：不改表结构，与 evidence_score 统一可观测体系
|
||||
- 影响：ChatService 后处理需追加 JSON
|
||||
- 风险接受：用户已确认
|
||||
|
||||
### Archive-Ready Update
|
||||
|
||||
- 实现调整：最终运行链路由 `ChatService` 显式调用 `planner -> executor -> verifier`，不再依赖 Supervisor prompt 保证 verifier 被调用。
|
||||
- 可追溯性补充：`tool_trace_summary` 增加 `trace_ref`、`source_invocation_ids`、查询样本、检索层级、相关性等级和来源文档标签。
|
||||
- 可追溯性补充：`facts_checked[*].evidence_refs` 被 prompt 要求、代码解析并持久化。
|
||||
- 验证记录：`mvn -q -DskipTests compile` 通过。
|
||||
- 验证记录：运行会话 `9138f064` 走通 planner、executor、verifier，并持久化 `verifier_evaluation.facts_checked[*].evidence_refs` 与 `tool_trace_summary[*].source_invocation_ids`。
|
||||
- 当前状态：OpenSpec change 已归档到 `openspec/changes/archive/2026-07-03-chat-verifier-agent/`，主规格已同步到 `openspec/specs/chat-verifier-agent/spec.md`。
|
||||
@@ -0,0 +1,52 @@
|
||||
# Evidence: chat-verifier-agent
|
||||
|
||||
## Code Evidence
|
||||
|
||||
### Complex chat path now invokes verifier deterministically
|
||||
|
||||
- File: `src/main/java/com/superbiz/agent/service/ChatService.java`
|
||||
- Evidence: `executeChatComplex` calls planner, executor, then verifier directly through `callAgent(...)`.
|
||||
- Conclusion: runtime no longer depends on prompt-only Supervisor behavior to call verifier.
|
||||
|
||||
### Verifier receives explicit inputs
|
||||
|
||||
- File: `src/main/java/com/superbiz/agent/hook/VerifierInputHook.java`
|
||||
- Evidence: the hook builds a JSON payload with `original_query`, `executor_final_answer`, `tool_trace_summary`, and `retry_context`.
|
||||
- Conclusion: verifier input is stable and does not depend on guessing the last assistant message from raw history.
|
||||
|
||||
### Tool evidence is traceable to persisted invocations
|
||||
|
||||
- File: `src/main/java/com/superbiz/agent/service/ToolTraceSummaryService.java`
|
||||
- Evidence: summaries include `trace_ref`, `source_invocation_ids`, `query_samples`, `retrieval_layers`, `relevance_levels`, and `source_documents`.
|
||||
- Conclusion: verifier facts can be correlated with actual `tool_invocation` rows.
|
||||
|
||||
### Verifier facts preserve evidence references
|
||||
|
||||
- File: `src/main/java/com/superbiz/agent/service/ChatService.java`
|
||||
- Evidence: verifier parsing preserves `facts_checked[*].evidence_refs` and persists `tool_trace_summary` under `verifier_evaluation`.
|
||||
- Conclusion: `self_evaluation` now contains both verifier judgments and the evidence index used to form them.
|
||||
|
||||
### Evaluation channels no longer overwrite each other
|
||||
|
||||
- File: `src/main/java/com/superbiz/agent/service/SelfEvaluationMergeService.java`
|
||||
- Evidence: rule and verifier evaluations are merged into separate keys.
|
||||
- Conclusion: asynchronous rule scoring preserves verifier output.
|
||||
|
||||
### Verifier logging is less noisy
|
||||
|
||||
- File: `src/main/java/com/superbiz/agent/hook/AgentLoggingHook.java`
|
||||
- Evidence: verifier `thought` stores a concise verdict summary, while fuller model output remains available in structured storage.
|
||||
- Conclusion: `agent_step.thought` is no longer a misleading place for full verifier JSON.
|
||||
|
||||
## Runtime Evidence
|
||||
|
||||
- Compile verification passed: `mvn -q -DskipTests compile`.
|
||||
- Runtime session `9138f064` executed `planner -> executor -> verifier`.
|
||||
- Runtime session `9138f064` persisted `verifier_evaluation.facts_checked[*].evidence_refs`.
|
||||
- Runtime session `9138f064` persisted `verifier_evaluation.tool_trace_summary[*].source_invocation_ids`.
|
||||
|
||||
## Design Evidence
|
||||
|
||||
- `LOW_CONFID` returns a fixed disclaimer and verifier-derived gaps.
|
||||
- `REJECT` returns degraded output and does not pass through the raw Executor answer.
|
||||
- `retry_context` is derived from verifier-identified missing evidence facts.
|
||||
@@ -0,0 +1,65 @@
|
||||
# MVP Demo Trace Acceptance
|
||||
|
||||
## Result
|
||||
|
||||
Accepted for implementation scope.
|
||||
|
||||
## Verification
|
||||
|
||||
### Static Verification
|
||||
|
||||
- Command: `mvn -q -DskipTests compile`
|
||||
- Result: passed
|
||||
- Notes: New trace controller, service, DTO, profile, verifier fallback, and test sources compile with the project.
|
||||
|
||||
### Script Verification
|
||||
|
||||
- Command: `mvn -q "-Dtest=DiagnosisTraceServiceTest,ChatServiceSupervisorAgentTest" test`
|
||||
- Result: passed
|
||||
- Notes: Covers successful trace aggregation, missing-session 404 path via `SessionNotFoundException`, low-confidence no-retry behavior, method-tool injection, and verifier fallback when Supervisor skips `chat_verifier`.
|
||||
|
||||
### OpenSpec Verification
|
||||
|
||||
- Command: `openspec validate mvp-demo-trace-acceptance --strict`
|
||||
- Result: passed
|
||||
|
||||
### GitNexus Verification
|
||||
|
||||
- Result: skipped by user decision
|
||||
- Notes: User requested subsequent project flow to bypass GitNexus.
|
||||
|
||||
### Manual / Runtime Verification
|
||||
|
||||
- Steps: Follow `mvp/demo/README.md` with `--spring.profiles.active=mvp-demo`.
|
||||
- Result: passed
|
||||
- Notes:
|
||||
- Session `mvp-demo-payment-timeout-20260703-rerun2` completed as `SUCCESS`.
|
||||
- Chat request returned `code=200`, `success=true`, and the same `sessionId`.
|
||||
- Chat duration was `96316 ms`; persisted session duration was `95028 ms`.
|
||||
- Trace API returned `code=200`, `returnedSteps=13`, `returnedTools=12`, `hasVerifier=true`, and `verifierVerdict=LOW_CONFID`.
|
||||
- Trace agents included `planner,executor,verifier`.
|
||||
- Trace tools included `lookup_knowledge,query_logs,query_metrics`.
|
||||
- Feedback submission returned success, and a follow-up trace query showed `feedback=useful`.
|
||||
- MySQL verification confirmed `agent_step` count `13` with agents `executor,planner,verifier`.
|
||||
- MySQL verification confirmed `tool_invocation` count `12` with tools `lookup_knowledge,query_logs,query_metrics`.
|
||||
|
||||
## Completed Scope
|
||||
|
||||
- Added `GET /api/diagnosis/{sessionId}/trace`.
|
||||
- Added read-only trace aggregation from persisted diagnosis tables.
|
||||
- Added `mvp-demo` profile overlay.
|
||||
- Added payment-timeout demo acceptance documentation.
|
||||
- Added MVP note for interview storytelling.
|
||||
- Added verifier fallback so runtime trace remains complete when Supervisor returns without `verifier_output`.
|
||||
|
||||
## Known Limits
|
||||
|
||||
- `mvp-demo` is not a fully offline mock runtime.
|
||||
- Runtime still depends on available MySQL, Redis, Milvus/Zilliz, model, and embedding configuration.
|
||||
- Sensitive configuration cleanup remains intentionally deferred.
|
||||
- Supervisor can still make inefficient routing choices inside a single round; `ChatService` now invokes `chat_verifier` as a fallback when Supervisor returns without `verifier_output`, so trace completeness is preserved for the MVP demo.
|
||||
|
||||
## Handoff
|
||||
|
||||
- Runtime demo passed with current infrastructure.
|
||||
- OpenSpec archive confirmation: requested by user after successful rerun.
|
||||
@@ -0,0 +1,35 @@
|
||||
# MVP Demo Trace Acceptance Brief
|
||||
|
||||
## Background
|
||||
|
||||
- User goal: make the MVP runnable, observable, and explainable for an Agent Engineer interview.
|
||||
- Current problem: the system can execute diagnosis, but reviewers need a simple way to replay one session from final answer back to agent steps and tool evidence.
|
||||
- Associated OpenSpec: `openspec/changes/mvp-demo-trace-acceptance/`
|
||||
- Devflow scale: standard-light.
|
||||
|
||||
## Scope
|
||||
|
||||
- In scope:
|
||||
- `mvp-demo` Spring profile overlay.
|
||||
- `GET /api/diagnosis/{sessionId}/trace` read-only API.
|
||||
- Trace aggregation DTO/service/controller.
|
||||
- Focused service tests.
|
||||
- Demo and acceptance documentation.
|
||||
- Out of scope:
|
||||
- Sensitive configuration cleanup.
|
||||
- Full offline LLM/vector/database mock runtime.
|
||||
- Database schema migration.
|
||||
- Changes to chat execution, verifier routing, upload, or feedback behavior.
|
||||
- Impact area:
|
||||
- `src/main/java/com/superbiz/agent/controller`
|
||||
- `src/main/java/com/superbiz/agent/service`
|
||||
- `src/main/java/com/superbiz/agent/dto`
|
||||
- `src/main/resources/application-mvp-demo.yml`
|
||||
- `mvp/demo`
|
||||
- `mvp/notes`
|
||||
|
||||
## OpenSpec Alignment
|
||||
|
||||
- proposal coverage: covered
|
||||
- specs coverage: covered
|
||||
- tasks coverage: covered
|
||||
@@ -0,0 +1,87 @@
|
||||
# MVP Demo Trace Acceptance Decisions
|
||||
|
||||
## Clarify
|
||||
|
||||
- Entry summary: continue the MVP toward a runnable and explainable demo by adding an `mvp-demo` profile, an end-to-end acceptance case, and a trace query API.
|
||||
- Slug: `mvp-demo-trace-acceptance`
|
||||
- Devflow scale: standard-light. The change adds a public read-only API and documentation, but does not alter core chat execution or persistence schemas.
|
||||
|
||||
## Context
|
||||
|
||||
- `devflow/index.md` was checked. Relevant history includes `session-storage`, `confidence-feedback`, `executor-action-memory-relevance`, and `chat-verifier-agent`.
|
||||
- `mvp/notes/agent-engineering-decisions.md` already recommends the next phase as "可复现 MVP Demo", including `mvp-demo` profile, fixed diagnosis case, one-click request, and `GET /api/diagnosis/{sessionId}/trace`.
|
||||
- `mvp/issues/ISS-003-mvp-design-implementation-review.md` identifies test stability, session traceability, verifier evidence chain, upload path, and SupervisorAgent consistency as recent MVP concerns. Security cleanup is intentionally deferred by user decision.
|
||||
|
||||
## Question Pool
|
||||
|
||||
| # | Dimension | Question | Mode | Status |
|
||||
|---|---|---|---|---|
|
||||
| Q1 | Terminology | Should "trace" mean persisted diagnosis execution evidence instead of transient frontend chat history? | evidence-driven | Resolved |
|
||||
| Q2 | Boundary | Should this change modify chat execution or only expose existing persisted evidence? | evidence-driven | Resolved |
|
||||
| Q3 | Acceptance | What proves the MVP flow is end-to-end enough for demo/interview use? | evidence-driven | Resolved |
|
||||
| Q4 | Interface | What is the API impact level for `GET /api/diagnosis/{sessionId}/trace`? | evidence-driven | Resolved |
|
||||
|
||||
## Evidence-driven
|
||||
|
||||
| Conclusion | Evidence Source | Reported To User |
|
||||
|---|---|---|
|
||||
| Trace should aggregate persisted diagnosis evidence, not Redis-only chat history. | `DiagnosisSession`, `AgentStep`, `ToolInvocation` entities and repositories | Reported in progress update |
|
||||
| Core chat execution does not need to change for this slice. | Existing unified chat path and SupervisorAgent commits; requested scope is demo/profile/trace/acceptance | Reported in progress update |
|
||||
| End-to-end acceptance should cover start -> chat -> trace -> feedback. | `ChatController`, `FeedbackController`, traceable session id decision in MVP notes | Reported in progress update |
|
||||
| Trace API is additive L3 because it is a new HTTP API for frontend/demo consumers. | sm-flow interface impact rules | Recorded in OpenSpec design |
|
||||
|
||||
## User-interview
|
||||
|
||||
| Question | User Words | Confirmation | OpenSpec Writeback |
|
||||
|---|---|---|---|
|
||||
| Should security/sensitive config cleanup be included? | "安全问题先不考虑"; "敏感配置先不做" | Confirmed | Non-goal |
|
||||
| Should this be implemented under sm-flow? | "按照 sm-flow 的流程来实现吧" | Confirmed | This change follows sm-flow artifacts |
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- Decision: Add a new trace API instead of embedding trace details in `/api/chat`.
|
||||
- Reason: Chat execution and observability should stay decoupled.
|
||||
- Impact: Demo can query trace after any successful chat request using the same session id.
|
||||
- Risk accepted: Response shape is new and should be treated as demo-facing contract.
|
||||
|
||||
- Decision: Keep `mvp-demo` profile as configuration overlay, not a fully mocked standalone runtime.
|
||||
- Reason: The current MVP still depends on real DB/Redis/Milvus/LLM for full chat execution; this change avoids inventing a fake runtime that hides integration behavior.
|
||||
- Impact: Demo profile improves repeatability for logs/metrics, while docs remain explicit about required external services.
|
||||
- Risk accepted: End-to-end acceptance may still require valid infrastructure and keys.
|
||||
|
||||
## Cross-Artifact Alignment
|
||||
|
||||
| Upstream -> Downstream | Check | Status |
|
||||
|---|---|---|
|
||||
| brief/prd -> proposal | Goal, scope, non-goals, and acceptance expectation are in proposal | Aligned |
|
||||
| proposal -> design | Scope, constraints, and API impact are in design | Aligned |
|
||||
| design -> specs/tasks | Trace DTO, controller/service, demo profile, and docs are represented | Aligned |
|
||||
| specs -> tasks | Observable behavior is covered by executable tasks | Aligned |
|
||||
|
||||
## Architecture Audit
|
||||
|
||||
- Data path: HTTP trace request -> controller -> trace service -> repositories -> aggregate DTO -> `Result.success`.
|
||||
- The service is read-only and does not mutate diagnosis, step, tool, or feedback state.
|
||||
- No schema change is needed because all required fields already exist in `diagnosis_session`, `agent_step`, and `tool_invocation`.
|
||||
- Main risk is response size for large sessions; MVP mitigates by returning previews already persisted by tools rather than raw external logs.
|
||||
- The additive API is acceptable for MVP because old callers remain unaffected.
|
||||
|
||||
## Pre-apply Research
|
||||
|
||||
- Reference implementations read:
|
||||
- `ChatController` for `/api` controller conventions.
|
||||
- `FeedbackController` for simple API controller shape.
|
||||
- `GlobalExceptionHandler` and `SessionNotFoundException` for 404 handling.
|
||||
- `DiagnosisSessionRepository`, `AgentStepRepository`, `ToolInvocationRepository` for available queries.
|
||||
- `DiagnosisSession`, `AgentStep`, `ToolInvocation` for fields.
|
||||
- Impact analysis:
|
||||
- `DiagnosisSessionRepository`: LOW, direct imports in service/controller paths.
|
||||
- `AgentStepRepository`: HIGH because it participates in chat/AiOps flows. This change only consumes existing query methods and does not modify the repository.
|
||||
- `ToolInvocationRepository`: LOW.
|
||||
|
||||
## Commit Gate
|
||||
|
||||
- OpenSpec proposal/design/specs/tasks exist.
|
||||
- API impact: L3 additive collaboration API, documented in design and spec.
|
||||
- User-confirmed non-goal: sensitive configuration cleanup remains out of scope.
|
||||
- No unresolved user-interview questions remain for this slice.
|
||||
@@ -0,0 +1,25 @@
|
||||
# MVP Demo Trace Acceptance Evidence
|
||||
|
||||
## Evidence
|
||||
|
||||
| Source | Evidence | Conclusion | Reported |
|
||||
|---|---|---|---|
|
||||
| `DiagnosisSessionRepository` | Existing `findBySessionId(String)` query | Trace can locate the session without new repository methods | Yes |
|
||||
| `AgentStepRepository` | Existing `findBySessionIdOrderByStepIndex(String)` query | Agent steps can be returned in execution order | Yes |
|
||||
| `ToolInvocationRepository` | Existing `findBySessionIdOrderByIdAsc(String)` query | Tool evidence can be returned in persisted order | Yes |
|
||||
| `GlobalExceptionHandler` | Handles `SessionNotFoundException` as HTTP 404 with `Result.error(404, ...)` | Missing trace can reuse existing error contract | Yes |
|
||||
| `mvn -q "-Dtest=DiagnosisTraceServiceTest" test` | Command passed | Trace aggregation behavior is covered offline | Yes |
|
||||
| `mvn -q -DskipTests compile` | Command passed | New code compiles with the full project | Yes |
|
||||
| `gitnexus detect-changes --repo SuperBizAgent-java` | Command completed with `No changes detected` and line-ending warnings | Required GitNexus check ran; output likely does not capture newly added files | Yes |
|
||||
|
||||
## Evidence-driven Conclusions
|
||||
|
||||
- Conclusion: No database migration is required.
|
||||
- Evidence: All trace fields are available from existing `diagnosis_session`, `agent_step`, and `tool_invocation` entities.
|
||||
- Risk: Response shape becomes a new API contract.
|
||||
- User confirmation: Not required; additive L3 API recorded in OpenSpec.
|
||||
|
||||
- Conclusion: Trace aggregation can be tested without external infrastructure.
|
||||
- Evidence: `DiagnosisTraceServiceTest` uses mocked repositories and an `ObjectMapper`.
|
||||
- Risk: Runtime integration still depends on configured infrastructure.
|
||||
- User confirmation: Not required; limitation recorded in acceptance docs.
|
||||
@@ -0,0 +1,94 @@
|
||||
# MVP Demo Runbook
|
||||
|
||||
This demo proves the MVP flow from user question to persisted diagnosis trace.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- MySQL, Redis, Milvus/Zilliz, and LLM/embedding configuration are available through the current project configuration.
|
||||
- Security and secret cleanup are intentionally out of scope for this MVP slice.
|
||||
- The `mvp-demo` profile enables mock Prometheus and CLS providers so log and metric tools can return repeatable evidence.
|
||||
|
||||
## Start
|
||||
|
||||
```powershell
|
||||
mvn spring-boot:run "-Dspring-boot.run.profiles=mvp-demo"
|
||||
```
|
||||
|
||||
The service listens on:
|
||||
|
||||
```text
|
||||
http://localhost:9900
|
||||
```
|
||||
|
||||
## 1. Run Chat Diagnosis
|
||||
|
||||
```powershell
|
||||
$sessionId = "mvp-demo-payment-timeout-001"
|
||||
$body = @{
|
||||
Id = $sessionId
|
||||
Question = "支付接口最近出现超时，请结合知识库、日志和指标判断可能原因，并给出修复建议。"
|
||||
} | ConvertTo-Json
|
||||
|
||||
Invoke-RestMethod `
|
||||
-Method Post `
|
||||
-Uri "http://localhost:9900/api/chat" `
|
||||
-ContentType "application/json" `
|
||||
-Body $body
|
||||
```
|
||||
|
||||
Expected result:
|
||||
|
||||
- `data.success` is `true`.
|
||||
- `data.sessionId` equals `mvp-demo-payment-timeout-001`.
|
||||
- `data.answer` contains a diagnosis answer.
|
||||
|
||||
## 2. Query Trace
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod `
|
||||
-Method Get `
|
||||
-Uri "http://localhost:9900/api/diagnosis/$sessionId/trace"
|
||||
```
|
||||
|
||||
Expected result:
|
||||
|
||||
- `code` is `200`.
|
||||
- `data.session.sessionId` equals the chat session id.
|
||||
- `data.steps` contains planner/executor/verifier records for complex questions.
|
||||
- `data.toolInvocations` contains evidence tool calls such as `lookup_knowledge`, `query_logs`, or `query_metrics`.
|
||||
- `data.session.selfEvaluation` contains verifier or rule evaluation when available.
|
||||
|
||||
## 3. Submit Feedback
|
||||
|
||||
```powershell
|
||||
$feedback = @{
|
||||
sessionId = $sessionId
|
||||
feedback = "useful"
|
||||
} | ConvertTo-Json
|
||||
|
||||
Invoke-RestMethod `
|
||||
-Method Post `
|
||||
-Uri "http://localhost:9900/api/feedback" `
|
||||
-ContentType "application/json" `
|
||||
-Body $feedback
|
||||
```
|
||||
|
||||
Expected result:
|
||||
|
||||
- `success` is `true`.
|
||||
- A later trace query shows `data.session.feedback` as `useful`.
|
||||
|
||||
## Demo Story
|
||||
|
||||
The important interview story is:
|
||||
|
||||
```text
|
||||
one session id
|
||||
-> user question
|
||||
-> multi-agent execution
|
||||
-> evidence tools
|
||||
-> verifier/self-evaluation
|
||||
-> final answer
|
||||
-> feedback
|
||||
-> trace API for replay and audit
|
||||
```
|
||||
@@ -0,0 +1,39 @@
|
||||
# Payment Timeout Acceptance Case
|
||||
|
||||
## Goal
|
||||
|
||||
Validate that the MVP can diagnose a payment timeout incident and expose the complete trace for replay.
|
||||
|
||||
## Input
|
||||
|
||||
- Session id: `mvp-demo-payment-timeout-001`
|
||||
- Question: `支付接口最近出现超时，请结合知识库、日志和指标判断可能原因，并给出修复建议。`
|
||||
- Profile: `mvp-demo`
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
1. Chat returns a successful answer with the same session id.
|
||||
2. Trace API returns session metadata, final answer, ordered agent steps, and ordered tool invocations.
|
||||
3. Trace contains enough evidence to explain which tools were used and whether verifier/self-evaluation was persisted.
|
||||
4. Feedback can be submitted for the same session id.
|
||||
5. A follow-up trace query shows the persisted feedback value.
|
||||
|
||||
## Trace Fields To Inspect
|
||||
|
||||
- `data.session.query`
|
||||
- `data.session.answer`
|
||||
- `data.session.selfEvaluation`
|
||||
- `data.session.feedback`
|
||||
- `data.steps[*].agentName`
|
||||
- `data.steps[*].thought`
|
||||
- `data.toolInvocations[*].toolName`
|
||||
- `data.toolInvocations[*].inputParams`
|
||||
- `data.toolInvocations[*].outputPreview`
|
||||
- `data.toolInvocations[*].retrievalDetails`
|
||||
- `data.summary`
|
||||
|
||||
## Known Limits
|
||||
|
||||
- This case is not a full offline test. It still requires valid infrastructure for chat, persistence, vector search, and model calls.
|
||||
- Mock logs and metrics are enabled by the `mvp-demo` profile to make those evidence tools repeatable.
|
||||
- Sensitive configuration cleanup is deferred by current MVP priority.
|
||||
@@ -0,0 +1,170 @@
|
||||
# ISS-003 MVP 设计与实现 Review 收敛
|
||||
|
||||
**状态**：待规划
|
||||
**严重程度**：高
|
||||
**发现时间**：2026-07-03
|
||||
**来源**：MVP 版本设计与实现 review
|
||||
|
||||
---
|
||||
|
||||
## 背景
|
||||
|
||||
当前 MVP 已具备 Chat、Planner/Executor/Verifier、知识检索、诊断会话落库、反馈与 case library 等主线能力，但设计文档、运行时实现和可验证性之间仍存在明显偏差。
|
||||
|
||||
本 issue 用来收敛本次 review 的主要风险，方便后续拆 OpenSpec change 或工程任务。
|
||||
|
||||
---
|
||||
|
||||
## 核心问题
|
||||
|
||||
### P0：敏感配置直接提交到仓库
|
||||
|
||||
`src/main/resources/application.yml` 中包含真实基础设施地址、数据库密码、Redis 密码、Milvus token、LLM API key。
|
||||
|
||||
`src/test/java/com/superbiz/agent/service/SimpleMilvusTest.java` 中也硬编码了 Milvus/Zilliz token。
|
||||
|
||||
**影响**：
|
||||
|
||||
- 密钥泄漏后需要立即轮换。
|
||||
- 合并 worktree 后会扩大泄漏面。
|
||||
- `show-sql: true` 与 DEBUG 日志可能进一步暴露业务数据。
|
||||
|
||||
**建议**：
|
||||
|
||||
- 立即轮换已提交的 token/password/api-key。
|
||||
- 将敏感配置改为环境变量或本地 profile 覆盖。
|
||||
- 提交 `application-example.yml` 或 `.env.example`，不要提交真实值。
|
||||
|
||||
### P1：测试体系不能稳定离线运行
|
||||
|
||||
`mvn test` 编译阶段通过，但 surefire 阶段大量失败，主要原因是测试直接依赖外部 MySQL、Redis、Milvus、LLM/Embedding 服务。
|
||||
|
||||
典型失败：
|
||||
|
||||
- MySQL/Flyway 连接失败导致 repository、Redis、Spring context 测试失败。
|
||||
- Milvus 连接测试出现 `DEADLINE_EXCEEDED`。
|
||||
- 当前环境下 Mockito inline mock maker self-attach 失败。
|
||||
|
||||
**影响**：
|
||||
|
||||
- 无法在合并前获得可靠的回归信号。
|
||||
- 实现变更与环境故障混在一起，问题定位成本高。
|
||||
|
||||
**建议**：
|
||||
|
||||
- 将纯单测、H2/JPA slice、外部集成测试分离。
|
||||
- 用 Maven profile 或 JUnit tag 区分 `unit` / `integration`。
|
||||
- 默认 `mvn test` 只跑不依赖外部服务的测试。
|
||||
|
||||
### P1：会话管理设计与实现不一致
|
||||
|
||||
`mvp/architecture/session-management.md` 设计 Redis 作为主会话存储，带 `session:{session_id}` 和 TTL。
|
||||
|
||||
实际 `/api/chat` 在 `ChatController` 中使用 JVM 内存 `ConcurrentHashMap` 管理历史消息，`RedisSessionManager` 虽然存在但没有接入 controller。
|
||||
|
||||
**影响**：
|
||||
|
||||
- 应用重启后会话历史丢失。
|
||||
- 多实例部署时会话不一致。
|
||||
- Redis TTL 与设计中的生命周期不生效。
|
||||
- 前端 chat session id 与后端 diagnosis session id 存在分裂。
|
||||
|
||||
**建议**：
|
||||
|
||||
- 明确 MVP 阶段是否接受内存会话。
|
||||
- 如果接受，需要同步更新文档并标注限制。
|
||||
- 如果不接受，应将 `ChatController` 接入 `SessionManager`，统一 session id 与 diagnosis session id 的关系。
|
||||
|
||||
### P1：Verifier 证据链仍不完整
|
||||
|
||||
`ToolTraceSummaryService` 期望从 `tool_invocation` 汇总 `lookup_knowledge`、`query_logs`、`query_metrics`、`query_order` 等证据工具。
|
||||
|
||||
当前只有 `LookupKnowledgeTool` 主动写入 `tool_invocation`。`QueryMetricsTools` 和 `QueryLogsTools` 返回 JSON，但没有落库。
|
||||
|
||||
**影响**：
|
||||
|
||||
- verifier 无法稳定审计日志、指标、订单等非知识库工具事实。
|
||||
- `thought` 或模型输出中看起来做了很多推理，但可追溯工具调用证据不足。
|
||||
- 用户侧可观测性仍然偏低。
|
||||
|
||||
**建议**：
|
||||
|
||||
- 抽象统一的 `ToolInvocationRecorder`。
|
||||
- 所有 evidence tool 都必须记录 input、output preview、success、duration、trace id。
|
||||
- verifier 只消费结构化 trace summary，不依赖模型自由文本回忆工具调用。
|
||||
|
||||
### P1：上传文档路径存在重复拼接风险
|
||||
|
||||
`DocumentManagementService.saveToLocal()` 返回的是包含 `knowledge_base` 前缀的本地路径。
|
||||
|
||||
`KnowledgeIndexService.readDocument()` 又执行 `Paths.get(knowledgeBasePath, filePath)`。
|
||||
|
||||
**影响**：
|
||||
|
||||
- 上传文档进入 L0 索引后，命中时读取原文可能拼成 `knowledge_base/knowledge_base/...`。
|
||||
- 这会降低 L0 命中后的答案质量，并造成“命中但读不到原文”的隐性故障。
|
||||
|
||||
**建议**：
|
||||
|
||||
- 统一 `filePath` 语义：要么存相对 `knowledge.base-path` 的路径，要么存绝对路径。
|
||||
- `readDocument()` 对 absolute path、已带 base path 的 relative path 做兼容。
|
||||
- 增加上传文档后 L0 命中并读取原文的回归测试。
|
||||
|
||||
### P2：SupervisorAgent 构建后未使用
|
||||
|
||||
`ChatService.executeChatComplex()` 中创建了 `SupervisorAgent`，但实际仍通过 `callAgent(planner/executor/verifier)` 手写顺序编排。
|
||||
|
||||
**影响**：
|
||||
|
||||
- 代码与设计文档中的 multi-agent 编排表述不一致。
|
||||
- 后续维护者容易误判当前已由 Supervisor 执行调度。
|
||||
|
||||
**建议**：
|
||||
|
||||
- 删除未使用的 `SupervisorAgent` 构建，明确当前是手写编排。
|
||||
- 或真正切到 Spring AI Alibaba SupervisorAgent flow，并补充行为验证。
|
||||
|
||||
### P2：生产安全边界偏弱
|
||||
|
||||
`SessionConfiguration` 使用 `activateDefaultTyping + LaissezFaireSubTypeValidator` 配置 Redis JSON 反序列化。
|
||||
|
||||
`WebMvcConfig` 对所有路径放开 CORS。
|
||||
|
||||
**影响**：
|
||||
|
||||
- Redis 若被非可信写入，存在多态反序列化风险。
|
||||
- CORS 全放开适合本地 MVP，不适合公开环境。
|
||||
|
||||
**建议**：
|
||||
|
||||
- Redis value 使用明确 DTO 类型或受限 subtype validator。
|
||||
- CORS 改为按 profile 配置允许域名。
|
||||
|
||||
---
|
||||
|
||||
## 优先级建议
|
||||
|
||||
1. 先处理敏感配置和密钥轮换，避免合并后扩大泄漏范围。
|
||||
2. 建立可离线运行的单测基线，让默认 `mvn test` 可用于合并门禁。
|
||||
3. 统一 session id 与 session storage，解决前后端、Redis、diagnosis session 的语义分裂。
|
||||
4. 补齐所有 evidence tool 的 `tool_invocation` 落库，提升 verifier 可追溯性。
|
||||
5. 修正上传文档路径语义，并补回归测试。
|
||||
6. 清理或真正启用 `SupervisorAgent`，避免设计和实现长期漂移。
|
||||
|
||||
---
|
||||
|
||||
## 相关文件
|
||||
|
||||
- `src/main/resources/application.yml`
|
||||
- `src/test/java/com/superbiz/agent/service/SimpleMilvusTest.java`
|
||||
- `src/main/java/com/superbiz/agent/controller/ChatController.java`
|
||||
- `src/main/java/com/superbiz/agent/service/session/impl/RedisSessionManager.java`
|
||||
- `src/main/java/com/superbiz/agent/service/ToolTraceSummaryService.java`
|
||||
- `src/main/java/com/superbiz/agent/tool/LookupKnowledgeTool.java`
|
||||
- `src/main/java/com/superbiz/agent/agent/tool/QueryMetricsTools.java`
|
||||
- `src/main/java/com/superbiz/agent/agent/tool/QueryLogsTools.java`
|
||||
- `src/main/java/com/superbiz/agent/service/DocumentManagementService.java`
|
||||
- `src/main/java/com/superbiz/agent/service/KnowledgeIndexService.java`
|
||||
- `src/main/java/com/superbiz/agent/service/ChatService.java`
|
||||
- `src/main/java/com/superbiz/agent/config/SessionConfiguration.java`
|
||||
- `src/main/java/com/superbiz/agent/config/WebMvcConfig.java`
|
||||
@@ -0,0 +1,59 @@
|
||||
# ISS-004 Executor 域级检索水位控制（Phase 2）
|
||||
|
||||
**状态**：待规划
|
||||
**严重程度**：低
|
||||
**发现时间**：2026-07-01
|
||||
**关联**：ISS-002（Executor 无约束重复调用 lookup_knowledge）
|
||||
|
||||
---
|
||||
|
||||
## 现象
|
||||
|
||||
ISS-002 修复后，`lookup_knowledge` 调用已经从 20+ 次收敛到约 10 次，但仍存在同一批 domain 之间反复横跳的冗余调用。
|
||||
|
||||
当前文档级去重能阻止重复内容进入上下文，但不能阻止 LLM 继续发起相似检索请求。
|
||||
|
||||
---
|
||||
|
||||
## 根因
|
||||
|
||||
Prompt 软约束依赖 LLM 自觉遵守。在 ReactAgent 自主决策模式下，模型倾向于“再确认一步”，而不是信任已有信息。
|
||||
|
||||
---
|
||||
|
||||
## 影响
|
||||
|
||||
- 不影响核心答案正确性。
|
||||
- 增加每轮检索耗时和 token 消耗。
|
||||
- 长会话中冗余调用会随 session 继续累积。
|
||||
|
||||
---
|
||||
|
||||
## 建议方案
|
||||
|
||||
在代码层增加域级检索水位控制，而不是只依赖 prompt。
|
||||
|
||||
水位指标可以包括：
|
||||
|
||||
- 当前 session 内 `lookup_knowledge` 调用次数。
|
||||
- 当前 session 已检索 domain 数量。
|
||||
- 当前 session token 消耗。
|
||||
- 最近一次检索结果的 `relevanceLevel`。
|
||||
|
||||
决策矩阵示例：
|
||||
|
||||
| 水位 | PRECISE | HIGHLY_RELEVANT | REFERENCE | DEDUPED |
|
||||
|---|---|---|---|---|
|
||||
| 低 | 可继续 | 可继续 | 可定向补充 | 停止 |
|
||||
| 中 | 可继续 | 建议停止 | 可定向补充 | 停止 |
|
||||
| 高 | 停止 | 停止 | 停止 | 停止 |
|
||||
|
||||
---
|
||||
|
||||
## 相关文件
|
||||
|
||||
- `src/main/java/com/superbiz/agent/dto/LookupResult.java`
|
||||
- `src/main/java/com/superbiz/agent/tool/RetrievedDocTracker.java`
|
||||
- `src/main/java/com/superbiz/agent/tool/LookupKnowledgeTool.java`
|
||||
- `src/main/resources/prompts/chat-executor-prompt.md`
|
||||
- `mvp/architecture/action-memory-relevance.md`
|
||||
@@ -4,3 +4,5 @@
|
||||
|---|---|---|---|---|
|
||||
| ISS-001 | Executor 重复召回同一文档 | 中 | 已修复 | [ISS-001-duplicate-retrieval.md](ISS-001-duplicate-retrieval.md) |
|
||||
| ISS-002 | Executor 无约束重复调用 lookup_knowledge | 中 | 已修复 | [ISS-002-executor-unconstrained-lookup.md](ISS-002-executor-unconstrained-lookup.md) |
|
||||
| ISS-003 | MVP 设计与实现 Review 收敛 | 高 | 待规划 | [ISS-003-mvp-design-implementation-review.md](ISS-003-mvp-design-implementation-review.md) |
|
||||
| ISS-004 | Executor 域级检索水位控制（Phase 2） | 低 | 待规划 | [ISS-004-executor-domain-hard-limit.md](ISS-004-executor-domain-hard-limit.md) |
|
||||
|
||||
@@ -0,0 +1,268 @@
|
||||
# MVP Agent 工程决策记录
|
||||
|
||||
本文记录 MVP 实现过程中已经落地的一些关键修复、取舍和工程判断。目标不是写流水账，而是沉淀面试时可以讲清楚的 Agent 工程思路。
|
||||
|
||||
---
|
||||
|
||||
## 1. 统一流式与非流式 Chat 主链路
|
||||
|
||||
### 背景
|
||||
|
||||
早期 `/api/chat` 和 `/api/chat_stream` 是两条不同实现：
|
||||
|
||||
- 非流式接口会走复杂度判断，并可能进入 Planner / Executor / Verifier 多 Agent 流程。
|
||||
- 流式接口直接创建单个 ReactAgent，然后 `agent.stream()` 输出 token。
|
||||
|
||||
这导致两个接口表面都是 chat，实际能力不一致：流式接口不会进入 verifier、不会沉淀完整诊断链路，也不容易和 `diagnosis_session`、`tool_invocation` 对齐。
|
||||
|
||||
### 决策
|
||||
|
||||
将两个接口统一到同一条核心链路：
|
||||
|
||||
```text
|
||||
getOrCreateSession
|
||||
-> 读取会话历史
|
||||
-> ChatService.executeChatWithStrategy(...)
|
||||
-> 写回会话历史
|
||||
```
|
||||
|
||||
接口差异只保留在传输层：
|
||||
|
||||
- `/api/chat` 返回完整 JSON。
|
||||
- `/api/chat_stream` 通过 SSE 分块发送最终答案。
|
||||
|
||||
### 取舍
|
||||
|
||||
这样会牺牲原来的 token 级实时流式体验，但换来业务行为一致、诊断链路一致、Verifier 和 evidence trace 一致。
|
||||
|
||||
对 MVP 来说，优先保证“同一个问题不因接口不同而进入不同智能链路”，比 token 级流式更重要。
|
||||
|
||||
---
|
||||
|
||||
## 2. 会话 ID 与诊断链路统一
|
||||
|
||||
### 背景
|
||||
|
||||
原实现中：
|
||||
|
||||
- `ChatController` 用前端传入的 `Id` 在 JVM 内存里维护历史消息。
|
||||
- `ChatService` 每次执行又生成新的 8 位 sessionId，作为 `diagnosis_session` 和工具调用追踪 ID。
|
||||
|
||||
这会造成前端会话、后端诊断会话、工具证据链三者分裂。
|
||||
|
||||
### 决策
|
||||
|
||||
将前端 chat session id 作为后端诊断链路的主 session id：
|
||||
|
||||
- Redis `SessionContext` 保存聊天历史。
|
||||
- `diagnosis_session.session_id` 复用同一个 id。
|
||||
- `RunnableConfig.metadata.sessionId` 和 `SessionContextHolder` 也使用同一个 id。
|
||||
- `tool_invocation`、`agent_step`、verifier evaluation 都可按同一 session id 串起来。
|
||||
|
||||
### 企业级意义
|
||||
|
||||
Agent 系统最怕“答得出来但查不清”。统一 session id 后，一次用户请求可以完整追踪：
|
||||
|
||||
```text
|
||||
用户问题 -> Agent 步骤 -> 工具调用 -> Verifier 判断 -> 最终答案 -> 用户反馈
|
||||
```
|
||||
|
||||
这是可观测、可审计、可复盘的基础。
|
||||
|
||||
---
|
||||
|
||||
## 3. 引入统一 ToolInvocationRecorder
|
||||
|
||||
### 背景
|
||||
|
||||
Verifier 需要结构化证据链，但原实现只有 `lookup_knowledge` 主动写入 `tool_invocation`。
|
||||
|
||||
`query_logs`、`query_metrics` 虽然返回 JSON，但没有统一落库，导致 verifier 看不到日志、指标等 evidence tool 的稳定记录。
|
||||
|
||||
### 决策
|
||||
|
||||
新增 `ToolInvocationRecorder`，作为所有 evidence tool 的统一落库入口。
|
||||
|
||||
当前接入：
|
||||
|
||||
- `lookup_knowledge`
|
||||
- `query_logs`
|
||||
- `query_metrics`
|
||||
|
||||
记录字段包括：
|
||||
|
||||
- tool name
|
||||
- input params
|
||||
- output preview
|
||||
- output length
|
||||
- success
|
||||
- error message
|
||||
- duration
|
||||
- trace id / domain details
|
||||
|
||||
### 企业级意义
|
||||
|
||||
这一步把 Agent 从“模型说它查过”推进到“系统能证明它查过”。
|
||||
|
||||
后续 verifier 不应该依赖模型自由文本回忆工具调用，而应该消费结构化 trace summary。
|
||||
|
||||
---
|
||||
|
||||
## 4. Verifier 作为事实约束层
|
||||
|
||||
### 背景
|
||||
|
||||
普通 Agent 很容易在工具调用后直接生成答案，但企业场景更关心：
|
||||
|
||||
- 关键结论有没有证据
|
||||
- 证据是直接证据还是间接支持
|
||||
- 哪些事实缺口需要人工介入
|
||||
- 工具失败时是否诚实降级
|
||||
|
||||
### 决策
|
||||
|
||||
保留 Planner / Executor / Verifier 三角色：
|
||||
|
||||
- Planner 负责拆解问题。
|
||||
- Executor 负责执行查询与形成初稿。
|
||||
- Verifier 负责基于 `tool_trace_summary` 做事实核查。
|
||||
|
||||
Verifier 输出结构化 JSON，包括：
|
||||
|
||||
- verdict
|
||||
- groundedness_score
|
||||
- critical_fact_count
|
||||
- facts_checked
|
||||
- rationale
|
||||
|
||||
### 取舍
|
||||
|
||||
Verifier 会增加一次模型调用成本，但换来可解释性和质量约束。对企业级 Agent 来说，这是值得的。
|
||||
|
||||
---
|
||||
|
||||
## 5. 从手写编排切换到 SupervisorAgent
|
||||
|
||||
### 背景
|
||||
|
||||
之前 `ChatService.executeChatComplex()` 中构建了 `SupervisorAgent`，但实际仍然手写调用：
|
||||
|
||||
```text
|
||||
planner -> executor -> verifier
|
||||
```
|
||||
|
||||
这会造成代码与设计不一致，维护者容易误以为当前已经由 Supervisor 调度。
|
||||
|
||||
### 决策
|
||||
|
||||
复杂问题真正切换到 `SupervisorAgent.invoke(...)`。
|
||||
|
||||
Supervisor 负责路由：
|
||||
|
||||
```text
|
||||
chat_supervisor -> chat_planner
|
||||
chat_supervisor -> chat_executor
|
||||
chat_supervisor -> chat_verifier
|
||||
chat_supervisor -> FINISH
|
||||
```
|
||||
|
||||
外层仍保留：
|
||||
|
||||
- verifier 输出解析
|
||||
- PASS / LOW_CONFID / REJECT 判定
|
||||
- retry context
|
||||
- fallback
|
||||
- evaluation 入库
|
||||
|
||||
### 验证
|
||||
|
||||
新增离线专项测试 `ChatServiceSupervisorAgentTest`，使用 scripted `ChatModel` 验证真实 SupervisorAgent 路由顺序，不依赖真实 LLM、MySQL、Redis。
|
||||
|
||||
### 企业级意义
|
||||
|
||||
这让项目不只是“自己写 if/else 多 Agent”，而是使用框架原生 multi-agent orchestration，同时保留业务层的质量门控。
|
||||
|
||||
---
|
||||
|
||||
## 6. 文档上传路径语义统一
|
||||
|
||||
### 背景
|
||||
|
||||
上传文档时，`DocumentManagementService.saveToLocal()` 返回带 `knowledge_base` 前缀的路径。
|
||||
|
||||
而 `KnowledgeIndexService.readDocument()` 又执行：
|
||||
|
||||
```java
|
||||
Paths.get(knowledgeBasePath, filePath)
|
||||
```
|
||||
|
||||
这可能拼出：
|
||||
|
||||
```text
|
||||
knowledge_base/knowledge_base/...
|
||||
```
|
||||
|
||||
最终表现为 L0 命中文档，但读取原文失败。
|
||||
|
||||
### 决策
|
||||
|
||||
统一路径语义：
|
||||
|
||||
- 新上传文档存相对 `knowledge.base-path` 的路径，例如 `payment/runbook.md`。
|
||||
- `readDocument()` 兼容新旧路径：
|
||||
- 相对路径
|
||||
- 已带 base path 的旧相对路径
|
||||
- 绝对路径
|
||||
|
||||
### 企业级意义
|
||||
|
||||
知识库检索不能只看“命中”，还要保证命中后的内容可读、可引用、可追踪。
|
||||
|
||||
这是 RAG / Agent 系统里很典型的工程细节：检索质量问题不一定来自模型，也可能来自路径、元数据、索引和原文之间的语义不一致。
|
||||
|
||||
---
|
||||
|
||||
## 7. MVP 阶段的优先级取舍
|
||||
|
||||
当前主动暂缓的问题：
|
||||
|
||||
- 敏感配置外置与密钥轮换
|
||||
- CORS / Redis 反序列化安全边界
|
||||
- 默认 `mvn test` 离线化
|
||||
|
||||
原因不是这些不重要，而是当前目标是先跑通并讲清楚 MVP Agent 工程闭环。
|
||||
|
||||
短期优先目标：
|
||||
|
||||
```text
|
||||
可演示 -> 可观测 -> 可验证 -> 可复盘
|
||||
```
|
||||
|
||||
安全和完整测试体系属于企业落地必须项，但可以在 MVP 主链路稳定后作为下一阶段补齐。
|
||||
|
||||
---
|
||||
|
||||
## 8. 后续建议
|
||||
|
||||
下一阶段建议聚焦“可复现 MVP Demo”：
|
||||
|
||||
1. 增加 `local-demo` 或 `mvp-demo` profile。
|
||||
2. 准备固定诊断 case，例如“支付接口超时”。
|
||||
3. 提供一键初始化知识库样例。
|
||||
4. 提供一键触发复杂诊断请求的脚本。
|
||||
5. 增加 trace 查询接口：
|
||||
|
||||
```text
|
||||
GET /api/diagnosis/{sessionId}/trace
|
||||
```
|
||||
|
||||
该接口聚合：
|
||||
|
||||
- diagnosis_session
|
||||
- agent_step
|
||||
- tool_invocation
|
||||
- verifier evaluation
|
||||
- final answer
|
||||
- feedback
|
||||
|
||||
这样 MVP 就能从“功能实现”升级为“企业级 Agent 工程作品”。
|
||||
@@ -0,0 +1,39 @@
|
||||
# MVP Demo Profile 与 Trace 查询接口
|
||||
|
||||
## 背景
|
||||
|
||||
MVP 已经能跑多 Agent 诊断、工具调用、Verifier 和反馈，但对外展示时仍然缺少一个稳定的复盘入口。面试官或评审如果想确认一次 Agent 回答是否可信，不能只看最终答案，还需要看到用户原始问题、Agent 步骤顺序、工具调用证据、Verifier / self-evaluation、最终答案和用户反馈。
|
||||
|
||||
## 决策
|
||||
|
||||
新增 `mvp-demo` profile 和 trace 查询接口：
|
||||
|
||||
```text
|
||||
GET /api/diagnosis/{sessionId}/trace
|
||||
```
|
||||
|
||||
接口聚合：
|
||||
|
||||
- `diagnosis_session`
|
||||
- `agent_step`
|
||||
- `tool_invocation`
|
||||
- `self_evaluation`
|
||||
- `feedback`
|
||||
|
||||
同时在 `mvp/demo` 下沉淀端到端验收 case，把启动、提问、查 trace、提交 feedback 串成一条可演示路径。
|
||||
|
||||
## 取舍
|
||||
|
||||
`mvp-demo` profile 不是完整离线 mock 环境，仍然复用当前真实 DB / Redis / Milvus / LLM 配置，只显式打开日志和指标 mock。原因是当前阶段目标是展示企业级 Agent 工程闭环，不是隐藏真实集成复杂度。
|
||||
|
||||
这让 MVP 的讲述从“我实现了一个聊天接口”升级为：
|
||||
|
||||
```text
|
||||
我实现了一条可执行、可观测、可验收、可复盘的 Agent 诊断链路。
|
||||
```
|
||||
|
||||
## 面试表达
|
||||
|
||||
- 我没有把 trace 塞进 chat 返回值，而是做成独立只读观测接口，保持执行链路和观测链路解耦。
|
||||
- Trace API 复用已经沉淀的 `diagnosis_session`、`agent_step`、`tool_invocation` 三张表，没有引入新的 schema 风险。
|
||||
- Demo profile 只做最小 overlay，让日志和指标工具可重复，保留真实基础设施集成，方便说明 MVP 与生产化之间的差距。
|
||||
@@ -0,0 +1 @@
|
||||
ready
|
||||
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"id": "chat-verifier-agent",
|
||||
"metadata": {
|
||||
"status": "archived",
|
||||
"created_at": "2026-07-02",
|
||||
"updated_at": "2026-07-03",
|
||||
"archive_readiness": "archived",
|
||||
"implementation_status": "archived"
|
||||
},
|
||||
"summary": "Add a verifier agent to the complex chat path and persist auditable verifier decisions with evidence traceability.",
|
||||
"artifacts": {
|
||||
"proposal": "proposal.md",
|
||||
"design": "design.md",
|
||||
"tasks": "tasks.md",
|
||||
"specs": [
|
||||
"specs/chat-verifier-agent/spec.md"
|
||||
],
|
||||
"devflow": "devflow/projects/2026-07-02-chat-verifier-agent"
|
||||
},
|
||||
"tasks": [
|
||||
"Verifier prompt",
|
||||
"VerifierInputHook explicit payload",
|
||||
"ChatService planner-executor-verifier orchestration",
|
||||
"Verdict routing and fixed user output templates",
|
||||
"Tool trace summary and evidence_refs traceability",
|
||||
"self_evaluation merge semantics",
|
||||
"Compile and runtime verification"
|
||||
],
|
||||
"verification": [
|
||||
{
|
||||
"type": "script",
|
||||
"command": "mvn -q -DskipTests compile",
|
||||
"result": "passed"
|
||||
},
|
||||
{
|
||||
"type": "runtime",
|
||||
"command": "POST /api/chat",
|
||||
"session_id": "9138f064",
|
||||
"result": "planner, executor, and verifier executed; verifier_evaluation contains evidence_refs and tool_trace_summary source_invocation_ids"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,349 @@
|
||||
## Context
|
||||
|
||||
Chat 多 Agent 链路当前由 ChatService 驱动 Planner → Executor，答案输出前无质量门禁。Verifier Agent 作为 Executor 后置质量门禁，在 Executor 输出后做事实核查。
|
||||
|
||||
前序 change `executor-action-memory-relevance` 已在 Executor 侧构建了行动记忆和检索质量归一化，Verifier 不需要重复验证检索质量。
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Verifier 作为无工具 ReactAgent，由 ChatService 显式调用
|
||||
- Verifier 输出 verdict (PASS/LOW_CONFID/REJECT) + groundedness_score + facts_checked
|
||||
- ChatService 负责单轮显式编排：Planner → Executor → Verifier
|
||||
- ChatService 外层根据 Verifier 判决做轮次路由：PASS→输出，LOW_CONFID≥0.5→带声明输出，LOW_CONFID<0.5→补充一轮，REJECT→降级
|
||||
- Verifier 判决写入 diagnosis_session.self_evaluation JSON 容器中的 `verifier_evaluation` 槽位做可观测
|
||||
|
||||
**Non-Goals:**
|
||||
- Verifier 不调用工具
|
||||
- 不改动单 Agent 链路
|
||||
- 不修改 Executor 的输出内容
|
||||
- 不涉及数据库表结构变更
|
||||
- Verifier 不继承 Executor 的中间推理过程（通过 MessagesModelHook 过滤）
|
||||
|
||||
## Decisions
|
||||
|
||||
| 决策 | 选择 | 放弃方案 | 原因 |
|
||||
|------|------|---------|------|
|
||||
| Verifier 是否有工具 | 无工具 ReactAgent | 有工具的 Agent | 职责单一，只核查不检索 |
|
||||
| 判决分类 | PASS / LOW_CONFID / REJECT | PASS / FAIL 二分类 | LOW_CONFID 提供了弹性输出路径 |
|
||||
| 回调机制 | ChatService 外层控制最多两轮 | 全交给 Supervisor / 不回调 | 轮次上限需要硬控制，不能只靠 prompt 记忆 |
|
||||
| 可观测方案 | 写入 self_evaluation JSON 容器 | agent_step / tool_invocation / 新表 | 不改表结构，同时避免与 evidence_score 覆盖冲突 |
|
||||
| 输入隔离 | 显式状态输入 + MessagesModelHook 裁剪噪音 | 仅靠原始消息过滤 / 数据库注入 | Verifier 需要稳定读取 query、工具摘要、最终答案，不能依赖消息格式猜测 |
|
||||
| 阈值配置 | yml 配置化 | 硬编码 | 方便运维调整，不需改代码 |
|
||||
|
||||
## Verifier 输入契约
|
||||
|
||||
Verifier 的业务输入由 `ChatService` 显式组装，不依赖原始 conversation messages 的隐式结构。
|
||||
|
||||
### 必选输入
|
||||
|
||||
- `original_query`：用户原始问题
|
||||
- `executor_final_answer`：本轮 Executor 最终答案
|
||||
- `tool_trace_summary`：由工具调用事实整理出的半结构化摘要
|
||||
|
||||
### 条件输入
|
||||
|
||||
- `retry_context`：仅第二轮注入，描述上一轮 verifier 发现的证据缺口和补充约束
|
||||
|
||||
### tool_trace_summary 最小结构
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"tool_name": "lookup_knowledge",
|
||||
"success": true,
|
||||
"input_summary": "查询 ERR_TIMEOUT",
|
||||
"output_summary": "命中 payment/errors.md，返回错误码定义",
|
||||
"evidence_level": "direct"
|
||||
},
|
||||
{
|
||||
"tool_name": "query_logs",
|
||||
"success": false,
|
||||
"input_summary": "按 traceId 查询日志",
|
||||
"output_summary": "日志服务超时",
|
||||
"evidence_level": "none"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
约束：
|
||||
|
||||
- `tool_trace_summary` 只纳入证据型工具调用，不纳入纯辅助或无业务事实意义的工具
|
||||
- `tool_trace_summary` 来源于工具调用事实，不直接透传原始日志全文
|
||||
- Verifier 基于摘要做事实核查，不直接读取数据库
|
||||
- 若某工具调用失败，仍需记录在摘要中，供 Verifier 判断证据缺口
|
||||
|
||||
### 证据型工具边界
|
||||
|
||||
默认纳入 `tool_trace_summary` 的工具：
|
||||
|
||||
- `lookup_knowledge`
|
||||
- `query_logs`
|
||||
- `query_metrics`
|
||||
- `query_order` 或其他业务事实查询类工具
|
||||
- 其他只读、能提供客观事实的工具
|
||||
|
||||
默认不纳入：
|
||||
|
||||
- `getCurrentDateTime`
|
||||
- 纯格式化、转换、控制类工具
|
||||
- 与事实核查无关的辅助工具
|
||||
|
||||
### 摘要压缩规则
|
||||
|
||||
- 每次调用只保留“最小证据摘要”，不透传原始返回全文
|
||||
- `output_summary` 控制为 1-3 句，重点描述“这次调用证明了什么 / 没能证明什么”
|
||||
- 失败调用必须保留，但统一标记：
|
||||
- `success=false`
|
||||
- `evidence_level=none`
|
||||
- 同一工具、同一主题域、同一轮次的重复调用可以折叠为一条合并摘要
|
||||
- 合并摘要至少保留：
|
||||
- 首次有效命中结果
|
||||
- 额外重复次数 / 未命中次数 / 失败次数
|
||||
|
||||
### 截断优先级
|
||||
|
||||
若 `tool_trace_summary` 过长，优先保留：
|
||||
|
||||
1. 被 `executor_final_answer` 直接引用的证据
|
||||
2. 支撑根因结论的证据
|
||||
3. 支撑修复结论的证据
|
||||
4. 与上一轮 `retry_context` 缺口直接相关的证据
|
||||
|
||||
低优先级、与最终答案无关的辅助性工具摘要可被截断。
|
||||
|
||||
### retry_context 最小结构
|
||||
|
||||
```json
|
||||
{
|
||||
"round": 1,
|
||||
"missing_evidence_facts": [
|
||||
"“根因是连接池耗尽”缺少直接证据",
|
||||
"“错误码 ERR_TIMEOUT 来自支付网关”只有间接支持"
|
||||
],
|
||||
"instruction": "仅补充以上断言相关证据，不要重复已完成检索"
|
||||
}
|
||||
```
|
||||
|
||||
### MessagesModelHook 职责边界
|
||||
|
||||
- 可以：移除 Planner/Executor 中间推理、无关闲聊和冗余 message
|
||||
- 不可以：作为 Verifier 核心业务输入的唯一来源
|
||||
- 目标：降噪，而非拼装业务事实
|
||||
|
||||
## Verifier 判决矩阵
|
||||
|
||||
Verifier 先提取并校验 `facts_checked`，再依据矩阵生成 verdict，避免只靠模型主观判断。
|
||||
|
||||
### facts_checked 分类
|
||||
|
||||
每条事实仅允许以下四类之一：
|
||||
|
||||
- `direct_evidence`：工具结果中有明确直接证据
|
||||
- `indirect_support`：可由工具结果合理推导，但不是直接陈述
|
||||
- `no_evidence`：工具结果中没有足够信息支撑
|
||||
- `contradicted`：工具结果与该事实冲突，或该事实编造了不存在的关键实体/错误码/结论
|
||||
|
||||
### 关键事实范围
|
||||
|
||||
Verifier 优先校验关键事实，至少包括：
|
||||
|
||||
- 根因结论（root cause）
|
||||
- 错误码 / 接口 / 组件归属
|
||||
- 证据来源陈述（如“日志显示”“文档说明”）
|
||||
- 明确修复结论
|
||||
|
||||
一般性建议、风险提示、非事实性表述默认不纳入关键事实，除非答案明确声称“已被证据证明”。
|
||||
|
||||
### verdict 规则
|
||||
|
||||
- `REJECT`
|
||||
- 任意关键事实为 `contradicted`
|
||||
- 或答案编造了工具/日志/文档中不存在的关键实体、错误码、结论
|
||||
|
||||
- `PASS`
|
||||
- 所有关键事实均为 `direct_evidence` 或 `indirect_support`
|
||||
- 且至少一条关键事实为 `direct_evidence`
|
||||
- 且不存在 `contradicted`
|
||||
|
||||
- `LOW_CONFID`
|
||||
- 不存在 `contradicted`
|
||||
- 但存在关键事实为 `no_evidence`
|
||||
- 或所有关键事实都只有 `indirect_support`，缺少直接锚点
|
||||
|
||||
一句话归纳：
|
||||
|
||||
- `REJECT` = 有冲突
|
||||
- `LOW_CONFID` = 无冲突但缺关键证据
|
||||
- `PASS` = 无冲突且关键事实均有支撑
|
||||
|
||||
### groundedness_score 计算
|
||||
|
||||
`groundedness_score` 不由模型自由打分，而由关键事实分类映射得到：
|
||||
|
||||
```text
|
||||
direct_evidence = 1.0
|
||||
indirect_support = 0.6
|
||||
no_evidence = 0.0
|
||||
contradicted = 0.0
|
||||
```
|
||||
|
||||
规则：
|
||||
|
||||
- 仅对关键事实计分
|
||||
- 取平均值后截断到 `[0.0, 1.0]`
|
||||
- 若存在任意关键事实为 `contradicted`，直接 verdict=`REJECT`，且 `groundedness_score=0.0`
|
||||
|
||||
### 第二轮补证据范围
|
||||
|
||||
第二轮 `retry_context` 仅回灌以下关键缺口：
|
||||
|
||||
- 关键事实为 `no_evidence`
|
||||
- 关键事实为 `indirect_support`，但仍缺直接证据锚点
|
||||
|
||||
`REJECT` 不进入第二轮补证据，直接降级输出。
|
||||
|
||||
## 用户侧输出协议
|
||||
|
||||
Verifier 的内部判决与用户侧最终输出类型分离：
|
||||
|
||||
- `PASS` → `NORMAL`
|
||||
- `LOW_CONFID` → `LOW_CONFID_WITH_DISCLAIMER`
|
||||
- `REJECT` → `DEGRADED`
|
||||
|
||||
### LOW_CONFID_WITH_DISCLAIMER
|
||||
|
||||
适用场景：
|
||||
|
||||
- 第一轮 `LOW_CONFID` 且 `groundedness_score >= threshold`
|
||||
- 第二轮后仍为 `LOW_CONFID`
|
||||
|
||||
输出规则：
|
||||
|
||||
- 使用固定免责声明前缀
|
||||
- 免责声明后拼接 `executor_final_answer`
|
||||
- 可选附加“当前证据缺口”列表，但来源必须是 verifier 的关键缺口，不得自由扩写
|
||||
|
||||
建议模板：
|
||||
|
||||
```text
|
||||
以下结论基于当前已获取证据，仍存在部分证据缺口，请谨慎参考。
|
||||
|
||||
{executor_final_answer}
|
||||
|
||||
当前缺口：
|
||||
- ...
|
||||
- ...
|
||||
```
|
||||
|
||||
### DEGRADED
|
||||
|
||||
适用场景：
|
||||
|
||||
- 任意一轮 `REJECT`
|
||||
- 系统无法基于现有证据形成可靠结论
|
||||
|
||||
输出规则：
|
||||
|
||||
- 不透传原始 `executor_final_answer`
|
||||
- 使用固定降级模板
|
||||
- 仅允许包含：
|
||||
- 已确认信息
|
||||
- 证据缺口
|
||||
- 下一步建议
|
||||
|
||||
建议模板：
|
||||
|
||||
```text
|
||||
当前无法基于已获取证据生成可靠结论，建议人工介入。
|
||||
|
||||
已确认信息：
|
||||
- ...
|
||||
|
||||
证据缺口：
|
||||
- ...
|
||||
|
||||
建议下一步：
|
||||
- ...
|
||||
```
|
||||
|
||||
### 输出边界
|
||||
|
||||
- `LOW_CONFID_WITH_DISCLAIMER` 可以带出原始答案，但必须加固定免责声明
|
||||
- `DEGRADED` 不得透传未经验证的原始答案
|
||||
- 用户侧输出模板由代码层拼装，不依赖 Verifier 自由生成
|
||||
|
||||
## self_evaluation 存储约定
|
||||
|
||||
`diagnosis_session.self_evaluation` 统一定义为 JSON 容器对象，而不是单一评估结果：
|
||||
|
||||
```json
|
||||
{
|
||||
"rule_evaluation": {
|
||||
"evidence_score": 65,
|
||||
"source": "rule",
|
||||
"factors": []
|
||||
},
|
||||
"verifier_evaluation": {
|
||||
"verdict": "LOW_CONFID",
|
||||
"groundedness_score": 0.42,
|
||||
"facts_checked": [],
|
||||
"rationale": "...",
|
||||
"round": 1
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
写入约束：
|
||||
|
||||
- `EvaluationService` 只负责写 `rule_evaluation`
|
||||
- `ChatService` 只负责写 `verifier_evaluation`
|
||||
- 两侧都必须使用 read-modify-write，保留另一侧已有内容
|
||||
- 禁止整段覆盖 `self_evaluation`，除非初始化为空对象
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Risk] Verifier 误判导致好答案被降级 → Mitigation: REJECT 仅用于明显编造场景，LOW_CONFID 为主要输出路径
|
||||
- [Risk] callback Planner 后新答案质量不一定提升 → Mitigation: 仅回调一次，Token 成本可控
|
||||
- [Risk] 第二轮仍可能产出 REJECT → Mitigation: 第二轮 REJECT 仍降级，不透传
|
||||
- [Risk] `self_evaluation` 被异步 evidence_score 覆盖 → Mitigation: 定义 JSON 容器槽位，统一 read-modify-write
|
||||
- [Risk] Verifier 增加 Token 消耗 → Mitigation: 单次轻量 LLM 调用，估算 <500 token
|
||||
- [Risk] 消息过滤可能导致输入契约漂移 → Mitigation: 主输入由显式状态输入提供，Hook 仅用于剔除中间推理和无关噪音
|
||||
- [Risk] 判决边界主观化，导致不同模型输出不稳定 → Mitigation: 用 facts_checked 分类 + verdict 矩阵 + 映射分数约束输出
|
||||
- [Risk] 最终用户文案随模型漂移，导致产品行为不稳定 → Mitigation: LOW_CONFID/DEGRADED 使用固定输出协议和模板
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
1. 创建 `chat-verifier-prompt.md`
|
||||
2. 新建 `VerifierInputHook.java`（MessagesModelHook 实现，BEFORE_MODEL 时裁剪 messages，只保留必要上下文）
|
||||
3. `ChatService.java` 新增 `buildChatVerifierAgent()` 方法（ReactAgent，无工具，带 hook）
|
||||
4. 添加 `verifier.low-confidence-threshold: 0.5` 到 application.yml
|
||||
5. 在 `ChatService.executeChatComplex()` 中显式调用 `Planner → Executor → Verifier`
|
||||
6. 保留 `SupervisorAgent` 构造作为 legacy residue，不再依赖 prompt-only supervisor sequencing 保证 Verifier 执行
|
||||
7. 在 `ChatService.executeChatComplex()` 外层实现最多两轮调用控制
|
||||
8. 组装 Verifier 显式状态输入：`original_query` / `executor_final_answer` / `tool_trace_summary` / `retry_context`
|
||||
9. 将 `self_evaluation` 升级为 JSON 容器读写：`rule_evaluation` / `verifier_evaluation`
|
||||
10. 读取 Verifier 判决写入 `verifier_evaluation`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Explicit orchestration
|
||||
|
||||
The final implementation uses `ChatService` to call `planner -> executor -> verifier` directly in each outer round. This replaces the earlier prompt-only dependency on `SupervisorAgent` for verifier execution. The supervisor construction remains in the code as legacy residue, but runtime correctness is driven by explicit `callAgent(...)` ordering.
|
||||
|
||||
### Traceability model
|
||||
|
||||
The implemented verifier input and persisted evaluation include an evidence index:
|
||||
|
||||
- `tool_trace_summary[*].trace_ref`
|
||||
- `tool_trace_summary[*].source_invocation_ids`
|
||||
- `tool_trace_summary[*].query_samples`
|
||||
- `tool_trace_summary[*].retrieval_layers`
|
||||
- `tool_trace_summary[*].relevance_levels`
|
||||
- `tool_trace_summary[*].source_documents`
|
||||
|
||||
Each verifier fact may carry `facts_checked[*].evidence_refs`, which points back to `trace_ref` and the underlying `tool_invocation` ids. This closes the audit gap where verifier could list many checked facts but the reviewer could not tell which facts related to which tool calls.
|
||||
|
||||
### Observability adjustment
|
||||
|
||||
`agent_step.thought` is now intentionally concise for verifier steps. Full verifier judgment belongs in `diagnosis_session.self_evaluation.verifier_evaluation`, with `model_output` retaining the model output snapshot.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Proposal: chat-verifier-agent
|
||||
|
||||
## Why
|
||||
|
||||
Chat 多 Agent 链路缺少出口质量门禁。Executor 输出答案后会直接返回给用户，无法在返回前拦截缺证据、低置信或明显编造的结论。
|
||||
|
||||
## What Changes
|
||||
|
||||
- 新增无工具 Verifier Agent，在 Executor 输出后读取答案和工具调用证据摘要，产出 `PASS` / `LOW_CONFID` / `REJECT` 判决。
|
||||
- `ChatService` 显式编排 `Planner -> Executor -> Verifier`，并根据 Verifier 判决控制最终输出或最多一次补充轮次。
|
||||
- Verifier 输入使用显式状态块：`original_query`、`executor_final_answer`、`tool_trace_summary`、第二轮可选 `retry_context`。
|
||||
- `diagnosis_session.self_evaluation` 作为 JSON 容器保存 `rule_evaluation` 与 `verifier_evaluation`，避免异步评分覆盖 Verifier 结果。
|
||||
- Verifier 结果增加可追溯证据引用：`tool_trace_summary[*].trace_ref`、`source_invocation_ids` 与 `facts_checked[*].evidence_refs`。
|
||||
- `LOW_CONFID` 和 `REJECT` 用户侧输出使用固定协议，`REJECT` 不透传未经验证的原始答案。
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `chat-verifier-agent`: Chat 多 Agent 出口事实核查、判决路由、观测存储和证据可追溯能力。
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- None.
|
||||
|
||||
## Impact
|
||||
|
||||
- Affected code: `ChatService`, chat verifier prompt, verifier input assembly, self-evaluation persistence, multi-agent runtime orchestration.
|
||||
- Affected runtime behavior: complex chat path now runs a Verifier gate after Executor and may perform one bounded retry for low-confidence evidence gaps.
|
||||
- No database schema change is required; `self_evaluation` remains the persistence container.
|
||||
- Non-goals: Verifier 不调用工具、不改写 Executor 答案、不影响单 Agent 链路、不支持超过两轮的补充编排。
|
||||
+189
@@ -0,0 +1,189 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Verifier SHALL fact-check Executor answers
|
||||
The system SHALL have a Verifier Agent that reads the Executor's answer and the tool call history, then produces a structured verdict.
|
||||
|
||||
#### Scenario: PASS verdict when all claims have evidence
|
||||
- **WHEN** all critical facts in the Executor's answer have direct or indirect support in tool call results
|
||||
- **AND** at least one critical fact has direct evidence
|
||||
- **AND** no critical fact is contradicted
|
||||
- **THEN** the Verifier SHALL output verdict="PASS" with groundedness_score ≥ 0.5
|
||||
|
||||
#### Scenario: LOW_CONFID verdict with partial evidence
|
||||
- **WHEN** no critical fact contradicts the tool results
|
||||
- **AND** some critical facts have no supporting evidence
|
||||
- **THEN** the Verifier SHALL output verdict="LOW_CONFID"
|
||||
|
||||
#### Scenario: LOW_CONFID verdict with only indirect support
|
||||
- **WHEN** no critical fact contradicts the tool results
|
||||
- **AND** all critical facts are only indirectly supported
|
||||
- **THEN** the Verifier SHALL output verdict="LOW_CONFID"
|
||||
|
||||
#### Scenario: REJECT verdict when claims contradict evidence
|
||||
- **WHEN** any critical fact in the Executor's answer contradicts tool call results
|
||||
- **OR** the answer fabricates a key entity, error code, or conclusion that does not exist in the tool evidence
|
||||
- **THEN** the Verifier SHALL output verdict="REJECT"
|
||||
|
||||
### Requirement: Verifier SHALL output structured JSON
|
||||
The Verifier SHALL output a JSON object with verdict, groundedness_score, facts_checked array, and rationale.
|
||||
|
||||
#### Scenario: Output format validation
|
||||
- **WHEN** the Verifier completes its analysis
|
||||
- **THEN** the output SHALL contain "verdict", "groundedness_score", "facts_checked", and "rationale" fields
|
||||
- **AND** groundedness_score SHALL be a float between 0.0 and 1.0
|
||||
- **AND** verdict SHALL be one of "PASS", "LOW_CONFID", or "REJECT"
|
||||
|
||||
#### Scenario: strict schema output
|
||||
- **WHEN** the Verifier returns its result
|
||||
- **THEN** it SHALL output exactly one JSON object
|
||||
- **AND** it SHALL NOT output Markdown, code fences, or explanatory text outside the JSON object
|
||||
- **AND** the JSON object SHALL include `critical_fact_count`
|
||||
- **AND** each `facts_checked` item SHALL include `fact`, `is_critical`, `verification`, and `detail`
|
||||
|
||||
### Requirement: facts_checked SHALL use a fixed classification set
|
||||
Each checked fact SHALL be labeled using a fixed evidence classification.
|
||||
|
||||
#### Scenario: fact classification values
|
||||
- **WHEN** the Verifier emits `facts_checked`
|
||||
- **THEN** each fact SHALL use one of `direct_evidence`, `indirect_support`, `no_evidence`, or `contradicted`
|
||||
|
||||
### Requirement: groundedness_score SHALL be derived from fact classifications
|
||||
The groundedness score SHALL be computed from critical fact classifications instead of being freely chosen by the model.
|
||||
|
||||
#### Scenario: contradicted fact forces reject
|
||||
- **WHEN** any critical fact is labeled `contradicted`
|
||||
- **THEN** the Verifier SHALL output verdict="REJECT"
|
||||
- **AND** groundedness_score SHALL be `0.0`
|
||||
|
||||
#### Scenario: score derived from supported facts
|
||||
- **WHEN** no critical fact is contradicted
|
||||
- **THEN** groundedness_score SHALL be computed from the mapped values of critical facts
|
||||
- **AND** the implementation SHALL use the fixed mapping `direct_evidence=1.0`, `indirect_support=0.6`, `no_evidence=0.0`
|
||||
- **AND** the result SHALL be clamped into `[0.0, 1.0]`
|
||||
|
||||
### Requirement: ChatService SHALL route based on Verifier verdict
|
||||
The system SHALL use ChatService for explicit single-round `Planner → Executor → Verifier` orchestration and SHALL use ChatService to control whether an additional round is allowed.
|
||||
|
||||
#### Scenario: PASS → direct output
|
||||
- **WHEN** Verifier outputs verdict="PASS"
|
||||
- **THEN** the system SHALL output the Executor's answer directly
|
||||
|
||||
#### Scenario: LOW_CONFID score≥0.5 → output with disclaimer
|
||||
- **WHEN** Verifier outputs verdict="LOW_CONFID" with groundedness_score ≥ 0.5
|
||||
- **THEN** the system SHALL output the Executor's answer prefixed with a fixed confidence disclaimer
|
||||
|
||||
#### Scenario: LOW_CONFID score<0.5 → trigger one additional round
|
||||
- **WHEN** Verifier outputs verdict="LOW_CONFID" with groundedness_score < 0.5 and this is the first callback
|
||||
- **THEN** the ChatService SHALL invoke one additional `Planner → Executor → Verifier` round to supplement evidence
|
||||
- **AND** after the second Verifier run, verdict="LOW_CONFID" SHALL be output with a confidence disclaimer
|
||||
- **AND** after the second Verifier run, verdict="REJECT" SHALL still produce a degraded output
|
||||
|
||||
#### Scenario: REJECT does not enter retry round
|
||||
- **WHEN** Verifier outputs verdict="REJECT"
|
||||
- **THEN** the system SHALL NOT start a retry round for evidence补充
|
||||
- **AND** it SHALL produce a degraded output directly
|
||||
|
||||
#### Scenario: REJECT → degraded output
|
||||
- **WHEN** Verifier outputs verdict="REJECT"
|
||||
- **THEN** the system SHALL output a degraded result indicating the answer cannot be reliably generated
|
||||
- **AND** it SHALL NOT pass through the raw Executor answer
|
||||
|
||||
### Requirement: User-facing verifier outputs SHALL follow fixed templates
|
||||
The system SHALL use fixed output protocols for LOW_CONFID and REJECT user-facing responses.
|
||||
|
||||
#### Scenario: LOW_CONFID uses disclaimer template
|
||||
- **WHEN** the final verdict is `LOW_CONFID`
|
||||
- **THEN** the user-facing response SHALL prepend a fixed disclaimer before the Executor answer
|
||||
- **AND** optional evidence gaps, if present, SHALL come only from verifier-identified critical gaps
|
||||
|
||||
#### Scenario: REJECT uses degraded template
|
||||
- **WHEN** the final verdict is `REJECT`
|
||||
- **THEN** the user-facing response SHALL use a degraded template
|
||||
- **AND** it SHALL include only confirmed facts, evidence gaps, and next-step suggestions
|
||||
- **AND** it SHALL NOT include unverified raw answer content
|
||||
|
||||
### Requirement: Verifier SHALL be observable
|
||||
The Verifier's verdict SHALL be persisted for observability.
|
||||
|
||||
#### Scenario: verdict written to self_evaluation
|
||||
- **WHEN** the Verifier produces a verdict
|
||||
- **THEN** the ChatService SHALL write the verdict data under `diagnosis_session.self_evaluation.verifier_evaluation`
|
||||
- **AND** existing `rule_evaluation` data SHALL be preserved
|
||||
|
||||
### Requirement: self_evaluation SHALL be a container object
|
||||
The `diagnosis_session.self_evaluation` field SHALL store multiple evaluation channels in one JSON object.
|
||||
|
||||
#### Scenario: rule evaluation stored separately
|
||||
- **WHEN** the rule-based evidence scoring completes
|
||||
- **THEN** the EvaluationService SHALL write the result under `rule_evaluation`
|
||||
- **AND** existing `verifier_evaluation` data SHALL be preserved
|
||||
|
||||
#### Scenario: verifier evaluation stored separately
|
||||
- **WHEN** the Verifier completes
|
||||
- **THEN** the ChatService SHALL write the result under `verifier_evaluation`
|
||||
- **AND** existing `rule_evaluation` data SHALL be preserved
|
||||
|
||||
#### Scenario: no whole-object overwrite after initialization
|
||||
- **WHEN** either evaluation channel updates `self_evaluation`
|
||||
- **THEN** the implementation SHALL use read-modify-write semantics
|
||||
- **AND** it SHALL NOT replace the whole JSON object except when initializing from null
|
||||
|
||||
### Requirement: Verifier SHALL consume explicit verification inputs
|
||||
The Verifier SHALL receive explicit verification inputs rather than inferring them only from raw conversation history.
|
||||
|
||||
#### Scenario: explicit input blocks available to Verifier
|
||||
- **WHEN** the Verifier starts
|
||||
- **THEN** the system SHALL provide `original_query`, `executor_final_answer`, and `tool_trace_summary` as explicit inputs
|
||||
- **AND** `retry_context` SHALL be provided on the second round only
|
||||
- **AND** message filtering MAY be used only to remove intermediate reasoning or unrelated noise
|
||||
|
||||
#### Scenario: tool trace summary derived from tool facts
|
||||
- **WHEN** the system prepares verifier inputs
|
||||
- **THEN** `tool_trace_summary` SHALL be generated from tool invocation facts
|
||||
- **AND** each summary item SHALL include tool name, success state, input summary, output summary, and evidence level
|
||||
- **AND** raw conversation history SHALL NOT be the only source of verifier evidence context
|
||||
|
||||
#### Scenario: tool trace summary preserves invocation references
|
||||
- **WHEN** the system prepares verifier inputs
|
||||
- **THEN** each summary item SHALL include a stable `trace_ref`
|
||||
- **AND** each summary item SHALL preserve `source_invocation_ids` for the tool invocation rows that contributed to the summary
|
||||
- **AND** each summary item SHOULD include query samples, retrieval layers, relevance levels, and source document labels when available
|
||||
|
||||
#### Scenario: only evidence-bearing tools included
|
||||
- **WHEN** the system generates `tool_trace_summary`
|
||||
- **THEN** it SHALL include only evidence-bearing tool invocations
|
||||
- **AND** non-evidence helper tools such as time or formatting tools SHALL be excluded by default
|
||||
|
||||
#### Scenario: failed evidence calls preserved as evidence gaps
|
||||
- **WHEN** an evidence-bearing tool invocation fails or returns no usable evidence
|
||||
- **THEN** the summary SHALL still include that invocation
|
||||
- **AND** it SHALL mark the entry as unsuccessful with an evidence level representing no evidence
|
||||
|
||||
#### Scenario: repeated tool calls may be compacted
|
||||
- **WHEN** repeated tool invocations concern the same tool, topic domain, and round
|
||||
- **THEN** the system MAY compact them into a merged summary entry
|
||||
- **AND** the merged entry SHALL preserve the first effective hit and the count of repeated, failed, or no-hit calls
|
||||
|
||||
#### Scenario: raw outputs not passed through in full
|
||||
- **WHEN** a tool invocation returns large raw content
|
||||
- **THEN** `tool_trace_summary` SHALL keep only a minimal evidence summary
|
||||
- **AND** the raw output SHALL NOT be passed through in full to the Verifier
|
||||
|
||||
#### Scenario: MessagesModelHook used only for noise reduction
|
||||
- **WHEN** a MessagesModelHook is used for the Verifier
|
||||
- **THEN** it MAY remove intermediate reasoning or irrelevant messages
|
||||
- **AND** it SHALL NOT be the primary source for assembling verifier business inputs
|
||||
|
||||
### Requirement: Verifier facts SHALL be auditable
|
||||
Verifier facts SHALL be linkable to the evidence summaries used during verification.
|
||||
|
||||
#### Scenario: facts_checked contains evidence refs
|
||||
- **WHEN** the Verifier emits `facts_checked`
|
||||
- **THEN** each fact SHALL include `evidence_refs`
|
||||
- **AND** each evidence ref SHALL point to an existing `tool_trace_summary.trace_ref`
|
||||
- **AND** each evidence ref SHALL preserve the relevant `source_invocation_ids` when available
|
||||
|
||||
#### Scenario: verifier evaluation persists traceability snapshot
|
||||
- **WHEN** the ChatService persists `verifier_evaluation`
|
||||
- **THEN** it SHALL include `traceability_version`
|
||||
- **AND** it SHALL include the `tool_trace_summary` snapshot used by the Verifier
|
||||
@@ -0,0 +1,58 @@
|
||||
# Tasks: chat-verifier-agent
|
||||
|
||||
## 1. Verifier Prompt
|
||||
|
||||
- [x] 1.1 Create `src/main/resources/prompts/chat-verifier-prompt.md`.
|
||||
- [x] 1.2 Define fixed fact classifications: `direct_evidence`, `indirect_support`, `no_evidence`, `contradicted`.
|
||||
- [x] 1.3 Define critical fact scope, verdict matrix, and `groundedness_score` mapping.
|
||||
- [x] 1.4 Define strict JSON output schema: `verdict`, `groundedness_score`, `critical_fact_count`, `facts_checked`, `rationale`.
|
||||
- [x] 1.5 Forbid Markdown, code fences, schema-extra fields, and text outside the JSON object.
|
||||
- [x] 1.6 Require `facts_checked[*].evidence_refs` for traceability to tool evidence.
|
||||
|
||||
## 2. Verifier Input Hook
|
||||
|
||||
- [x] 2.1 Add `VerifierInputHook.java` as a `MessagesModelHook` running at `BEFORE_MODEL`.
|
||||
- [x] 2.2 Replace raw verifier history with explicit payload fields: `original_query`, `executor_final_answer`, `tool_trace_summary`, `retry_context`.
|
||||
- [x] 2.3 Persist the current round `tool_trace_summary` in `VerifierContextHolder` for later verifier evaluation storage.
|
||||
|
||||
## 3. ChatService Integration
|
||||
|
||||
- [x] 3.1 Load `chatVerifierPrompt` and add `buildChatVerifierAgent()`.
|
||||
- [x] 3.2 Add configurable `verifier.low-confidence-threshold`.
|
||||
- [x] 3.3 Implement explicit per-round orchestration in `ChatService`: planner call, executor call, verifier call.
|
||||
- [x] 3.4 Keep max two outer rounds and inject `retry_context` only for the second round.
|
||||
- [x] 3.5 Parse verifier JSON directly and fall back to `LOW_CONFID` when verifier output is missing or invalid.
|
||||
- [x] 3.6 Keep `SupervisorAgent` construction as legacy residue only; runtime orchestration no longer depends on prompt-only supervisor sequencing.
|
||||
|
||||
## 4. Verdict Routing And User Output
|
||||
|
||||
- [x] 4.1 Route `PASS` to the executor answer.
|
||||
- [x] 4.2 Route `LOW_CONFID` to a fixed disclaimer plus executor answer.
|
||||
- [x] 4.3 Route `REJECT` to degraded output without passing through the raw unverified answer.
|
||||
- [x] 4.4 Build LOW_CONFID gap lists only from verifier-identified gaps.
|
||||
- [x] 4.5 Build DEGRADED confirmed facts, gaps, and next-step suggestions from verifier facts and trace summary.
|
||||
|
||||
## 5. Trace Summary And Observability
|
||||
|
||||
- [x] 5.1 Add `ToolTraceSummaryService` to build verifier evidence summaries from `tool_invocation`.
|
||||
- [x] 5.2 Include only evidence-bearing tools by default.
|
||||
- [x] 5.3 Compact repeated calls by tool and topic domain.
|
||||
- [x] 5.4 Preserve `source_invocation_ids`, `trace_ref`, query samples, retrieval layers, relevance levels, and source document labels.
|
||||
- [x] 5.5 Parse and persist `facts_checked[*].evidence_refs`.
|
||||
- [x] 5.6 Persist `verifier_evaluation.tool_trace_summary` and `traceability_version`.
|
||||
- [x] 5.7 Store concise verifier summaries in `agent_step.thought` while preserving fuller verifier output in `model_output` / `self_evaluation`.
|
||||
|
||||
## 6. self_evaluation Merge Semantics
|
||||
|
||||
- [x] 6.1 Add `SelfEvaluationMergeService`.
|
||||
- [x] 6.2 Write verifier results under `verifier_evaluation`.
|
||||
- [x] 6.3 Write rule scoring under `rule_evaluation`.
|
||||
- [x] 6.4 Preserve the other channel with read-modify-write semantics.
|
||||
|
||||
## 7. Verification
|
||||
|
||||
- [x] 7.1 Compile verification: `mvn -q -DskipTests compile`.
|
||||
- [x] 7.2 Runtime verification: `/api/chat` complex request reached `planner -> executor -> verifier`.
|
||||
- [x] 7.3 Runtime verification: session `9138f064` persisted `verifier_evaluation.facts_checked[*].evidence_refs`.
|
||||
- [x] 7.4 Runtime verification: session `9138f064` persisted `tool_trace_summary[*].source_invocation_ids`.
|
||||
- [x] 7.5 Runtime verification: LOW_CONFID user output included disclaimer and verifier-derived gaps.
|
||||
@@ -0,0 +1 @@
|
||||
mvp-demo-trace-acceptance committed on 2026-07-03
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-03
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"id": "mvp-demo-trace-acceptance",
|
||||
"metadata": {
|
||||
"status": "committed",
|
||||
"created_at": "2026-07-03",
|
||||
"updated_at": "2026-07-03",
|
||||
"implementation_status": "implemented"
|
||||
},
|
||||
"summary": "Add an MVP demo profile, a read-only diagnosis trace API, and an end-to-end acceptance case.",
|
||||
"artifacts": {
|
||||
"proposal": "proposal.md",
|
||||
"design": "design.md",
|
||||
"tasks": "tasks.md",
|
||||
"specs": [
|
||||
"specs/mvp-demo-trace-acceptance/spec.md"
|
||||
],
|
||||
"devflow": "devflow/projects/2026-07-03-mvp-demo-trace-acceptance"
|
||||
},
|
||||
"tasks": [
|
||||
"Add DiagnosisTraceResponse DTO",
|
||||
"Add DiagnosisTraceService aggregation",
|
||||
"Add DiagnosisTraceController endpoint",
|
||||
"Add mvp-demo profile",
|
||||
"Add MVP demo acceptance documentation",
|
||||
"Add focused trace service tests",
|
||||
"Run targeted verification and GitNexus change detection",
|
||||
"Update MVP notes and devflow acceptance"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
## Context
|
||||
|
||||
The MVP already persists diagnosis execution data across three tables:
|
||||
|
||||
- `diagnosis_session`: query, status, answer, counts, feedback, and `self_evaluation`.
|
||||
- `agent_step`: ordered agent execution records.
|
||||
- `tool_invocation`: evidence tool calls and retrieval metadata.
|
||||
|
||||
Recent work unified chat session ids and persisted tool invocations, so a single session id can now connect user input, agent steps, evidence tools, verifier evaluation, final answer, and feedback. The missing piece is a read-only aggregation API and a documented demo profile/workflow that a reviewer can run without reading database tables manually.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
|
||||
- Add a trace API that returns one aggregated view for a diagnosis session.
|
||||
- Keep the trace API read-only and based on existing persistence tables.
|
||||
- Add an `mvp-demo` profile that makes the demo intent explicit and keeps mock log/metric tools enabled.
|
||||
- Add a documented end-to-end acceptance case for start, chat, trace query, and feedback.
|
||||
- Add focused tests for trace aggregation.
|
||||
|
||||
**Non-Goals:**
|
||||
|
||||
- Do not clean up committed sensitive configuration in this change.
|
||||
- Do not add database migrations.
|
||||
- Do not alter `/api/chat`, `/api/chat_stream`, verifier routing, feedback, or document upload behavior.
|
||||
- Do not create a fully offline fake LLM runtime.
|
||||
|
||||
## Decisions
|
||||
|
||||
| Decision | Choice | Alternative Considered | Rationale |
|
||||
|---|---|---|---|
|
||||
| Trace API shape | Add `GET /api/diagnosis/{sessionId}/trace` | Extend `/api/chat` response | Trace is an observability concern and should not make chat responses larger or change chat clients. |
|
||||
| Aggregation ownership | New `DiagnosisTraceService` | Put aggregation in controller | Keeps controller thin and allows focused unit tests with mocked repositories. |
|
||||
| Response DTO | Dedicated nested DTO | Return raw entities or maps | DTO avoids leaking JPA entity details and gives a stable demo-facing contract. |
|
||||
| Missing session handling | Throw `SessionNotFoundException` and use existing global 404 handler | Return empty success payload | A missing trace is a real lookup miss and should be visible to callers. |
|
||||
| `self_evaluation` handling | Return raw JSON string and best-effort parsed JSON | Parse only, or ignore parse failures | Raw value preserves evidence even if JSON shape evolves; parsed value improves frontend/demo readability. |
|
||||
| Demo profile | Add `application-mvp-demo.yml` overlay | Change default `application.yml` | Overlay avoids disturbing current runtime and keeps demo choices explicit. |
|
||||
|
||||
## Interface Impact
|
||||
|
||||
- Level: L3 collaboration API.
|
||||
- Reason: This adds a new HTTP endpoint and response contract intended for frontend/demo/reviewer consumption.
|
||||
- Compatibility: Additive only. Existing callers do not need to change.
|
||||
- Documentation: The endpoint is documented in the MVP demo acceptance case.
|
||||
|
||||
## Data Structures
|
||||
|
||||
The trace response contains:
|
||||
|
||||
- `session`: session id, query, status, flow, counts, timing, created/updated time, final answer, raw self-evaluation JSON, parsed self-evaluation object, and feedback.
|
||||
- `steps`: ordered agent steps with step index, agent name, model input/output, thought, tool flag, duration, token count, and created time.
|
||||
- `toolInvocations`: ordered tool records with id, step id, tool name, input params, output preview, retrieval metadata, duration, success, error, and created time.
|
||||
- `summary`: counts derived from the returned collections and session fields.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Risk] Trace responses may become large for long sessions. -> Mitigation: the MVP returns persisted previews and structured metadata, not raw full external logs.
|
||||
- [Risk] `self_evaluation` JSON shape may evolve. -> Mitigation: return both raw and best-effort parsed forms.
|
||||
- [Risk] Demo profile still depends on real DB/Redis/Milvus/LLM. -> Mitigation: document prerequisites and keep mock logs/metrics enabled for repeatable tool evidence.
|
||||
- [Risk] New endpoint becomes a de facto frontend contract. -> Mitigation: use a dedicated DTO and document L3 additive API impact.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
- Deploying this change requires only application restart with the new code.
|
||||
- No database migration is required.
|
||||
- Rollback is deleting the new endpoint/profile/docs; persisted data remains unchanged.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- None for this slice. Security and full offline test profile remain deferred by explicit user decision.
|
||||
@@ -0,0 +1,29 @@
|
||||
## Why
|
||||
|
||||
The MVP can already execute multi-agent diagnosis, persist session traces, and collect feedback, but it is still hard to demonstrate as a complete enterprise-style workflow. A demo profile, a trace query API, and an explicit end-to-end acceptance case make the project runnable, observable, and explainable for interview and portfolio review.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add an `mvp-demo` Spring profile that keeps the existing external infrastructure contract but turns on mock log and metric providers for repeatable demonstrations.
|
||||
- Add a read-only trace query API: `GET /api/diagnosis/{sessionId}/trace`.
|
||||
- Aggregate `diagnosis_session`, `agent_step`, `tool_invocation`, verifier/self-evaluation, final answer, and feedback into one trace response.
|
||||
- Add an end-to-end MVP acceptance case that documents startup, chat request, trace query, and feedback submission.
|
||||
- Add focused service tests for trace aggregation without requiring MySQL, Redis, Milvus, or a real LLM.
|
||||
- Record the design decision in MVP notes for interview storytelling.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `mvp-demo-trace-acceptance`: Covers the MVP demo profile, trace query API, and end-to-end acceptance workflow for a reproducible agent diagnosis demo.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- None.
|
||||
|
||||
## Impact
|
||||
|
||||
- Affected code: new trace controller/service/DTOs, `application-mvp-demo.yml`, unit tests, MVP demo documentation.
|
||||
- Affected API: adds `GET /api/diagnosis/{sessionId}/trace`. This is an additive L3 collaboration API because it is intended for frontend, demo, and external reviewer consumption.
|
||||
- Affected runtime behavior: no change to chat execution, verifier, feedback, document upload, or persistence semantics.
|
||||
- Non-goals: no sensitive configuration cleanup, no database schema migration, no replacement of existing chat endpoints, no full offline mock LLM implementation.
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Diagnosis trace can be queried by session id
|
||||
The system SHALL expose a read-only HTTP endpoint `GET /api/diagnosis/{sessionId}/trace` that returns the persisted diagnosis trace for the requested session id.
|
||||
|
||||
#### Scenario: Existing session trace is returned
|
||||
- **WHEN** a caller requests trace data for a session id that exists in `diagnosis_session`
|
||||
- **THEN** the system returns a success response containing the session summary, ordered agent steps, ordered tool invocations, self-evaluation data, final answer, and feedback
|
||||
|
||||
#### Scenario: Missing session returns not found
|
||||
- **WHEN** a caller requests trace data for a session id that does not exist in `diagnosis_session`
|
||||
- **THEN** the system returns a 404 response using the existing session-not-found error contract
|
||||
|
||||
### Requirement: Trace aggregation is read-only
|
||||
The system MUST build trace output from existing persisted diagnosis tables and MUST NOT mutate diagnosis sessions, agent steps, tool invocations, feedback, or chat session state while serving the trace request.
|
||||
|
||||
#### Scenario: Trace query does not change persisted state
|
||||
- **WHEN** a caller requests `GET /api/diagnosis/{sessionId}/trace`
|
||||
- **THEN** the system reads `diagnosis_session`, `agent_step`, and `tool_invocation` records and returns an aggregate without saving any of those records
|
||||
|
||||
### Requirement: MVP demo profile is available
|
||||
The system SHALL provide an `mvp-demo` Spring profile that documents the demo runtime intent and keeps mock log and metric providers enabled for repeatable diagnosis demonstrations.
|
||||
|
||||
#### Scenario: Demo profile loads mock evidence providers
|
||||
- **WHEN** the application starts with `--spring.profiles.active=mvp-demo`
|
||||
- **THEN** `prometheus.mock-enabled` and `cls.mock-enabled` are enabled by profile configuration
|
||||
|
||||
### Requirement: End-to-end MVP acceptance case is documented
|
||||
The project SHALL include an end-to-end acceptance case that demonstrates start-up, chat diagnosis, trace query, and feedback submission using the same session id.
|
||||
|
||||
#### Scenario: Reviewer follows the acceptance case
|
||||
- **WHEN** a reviewer follows the documented MVP demo acceptance steps
|
||||
- **THEN** they can run the application, submit a diagnosis question, query the trace endpoint, and submit feedback for the same session id
|
||||
@@ -0,0 +1,25 @@
|
||||
## 1. Trace Query API
|
||||
|
||||
- [x] 1.1 Add a `DiagnosisTraceResponse` DTO that represents session summary, ordered agent steps, ordered tool invocations, and derived summary counts.
|
||||
- [x] 1.2 Add `DiagnosisTraceService` that loads `DiagnosisSession`, `AgentStep`, and `ToolInvocation` records by session id and builds the response.
|
||||
- [x] 1.3 Add `DiagnosisTraceController` with `GET /api/diagnosis/{sessionId}/trace`.
|
||||
- [x] 1.4 Return 404 through `SessionNotFoundException` when the requested diagnosis session does not exist.
|
||||
|
||||
## 2. Demo Profile And Acceptance Case
|
||||
|
||||
- [x] 2.1 Add `src/main/resources/application-mvp-demo.yml` with MVP demo profile overlays and mock logs/metrics enabled.
|
||||
- [x] 2.2 Add `mvp/demo/README.md` documenting prerequisites, startup, chat request, trace query, and feedback submission.
|
||||
- [x] 2.3 Add a concrete payment-timeout acceptance case with request/response expectations.
|
||||
|
||||
## 3. Tests And Verification
|
||||
|
||||
- [x] 3.1 Add focused unit tests for `DiagnosisTraceService` success and missing-session behavior.
|
||||
- [x] 3.2 Run targeted tests for the new trace service.
|
||||
- [x] 3.3 Run compile verification.
|
||||
- [x] 3.4 Run GitNexus change detection before commit or handoff.
|
||||
|
||||
## 4. Notes And Flow Records
|
||||
|
||||
- [x] 4.1 Update MVP engineering notes with the demo/trace decision.
|
||||
- [x] 4.2 Update OpenSpec tasks as work completes.
|
||||
- [x] 4.3 Record verification results in devflow acceptance notes.
|
||||
@@ -0,0 +1,192 @@
|
||||
# chat-verifier-agent Specification
|
||||
|
||||
## Purpose
|
||||
TBD - created by archiving change chat-verifier-agent. Update Purpose after archive.
|
||||
## Requirements
|
||||
### Requirement: Verifier SHALL fact-check Executor answers
|
||||
The system SHALL have a Verifier Agent that reads the Executor's answer and the tool call history, then produces a structured verdict.
|
||||
|
||||
#### Scenario: PASS verdict when all claims have evidence
|
||||
- **WHEN** all critical facts in the Executor's answer have direct or indirect support in tool call results
|
||||
- **AND** at least one critical fact has direct evidence
|
||||
- **AND** no critical fact is contradicted
|
||||
- **THEN** the Verifier SHALL output verdict="PASS" with groundedness_score ≥ 0.5
|
||||
|
||||
#### Scenario: LOW_CONFID verdict with partial evidence
|
||||
- **WHEN** no critical fact contradicts the tool results
|
||||
- **AND** some critical facts have no supporting evidence
|
||||
- **THEN** the Verifier SHALL output verdict="LOW_CONFID"
|
||||
|
||||
#### Scenario: LOW_CONFID verdict with only indirect support
|
||||
- **WHEN** no critical fact contradicts the tool results
|
||||
- **AND** all critical facts are only indirectly supported
|
||||
- **THEN** the Verifier SHALL output verdict="LOW_CONFID"
|
||||
|
||||
#### Scenario: REJECT verdict when claims contradict evidence
|
||||
- **WHEN** any critical fact in the Executor's answer contradicts tool call results
|
||||
- **OR** the answer fabricates a key entity, error code, or conclusion that does not exist in the tool evidence
|
||||
- **THEN** the Verifier SHALL output verdict="REJECT"
|
||||
|
||||
### Requirement: Verifier SHALL output structured JSON
|
||||
The Verifier SHALL output a JSON object with verdict, groundedness_score, facts_checked array, and rationale.
|
||||
|
||||
#### Scenario: Output format validation
|
||||
- **WHEN** the Verifier completes its analysis
|
||||
- **THEN** the output SHALL contain "verdict", "groundedness_score", "facts_checked", and "rationale" fields
|
||||
- **AND** groundedness_score SHALL be a float between 0.0 and 1.0
|
||||
- **AND** verdict SHALL be one of "PASS", "LOW_CONFID", or "REJECT"
|
||||
|
||||
#### Scenario: strict schema output
|
||||
- **WHEN** the Verifier returns its result
|
||||
- **THEN** it SHALL output exactly one JSON object
|
||||
- **AND** it SHALL NOT output Markdown, code fences, or explanatory text outside the JSON object
|
||||
- **AND** the JSON object SHALL include `critical_fact_count`
|
||||
- **AND** each `facts_checked` item SHALL include `fact`, `is_critical`, `verification`, and `detail`
|
||||
|
||||
### Requirement: facts_checked SHALL use a fixed classification set
|
||||
Each checked fact SHALL be labeled using a fixed evidence classification.
|
||||
|
||||
#### Scenario: fact classification values
|
||||
- **WHEN** the Verifier emits `facts_checked`
|
||||
- **THEN** each fact SHALL use one of `direct_evidence`, `indirect_support`, `no_evidence`, or `contradicted`
|
||||
|
||||
### Requirement: groundedness_score SHALL be derived from fact classifications
|
||||
The groundedness score SHALL be computed from critical fact classifications instead of being freely chosen by the model.
|
||||
|
||||
#### Scenario: contradicted fact forces reject
|
||||
- **WHEN** any critical fact is labeled `contradicted`
|
||||
- **THEN** the Verifier SHALL output verdict="REJECT"
|
||||
- **AND** groundedness_score SHALL be `0.0`
|
||||
|
||||
#### Scenario: score derived from supported facts
|
||||
- **WHEN** no critical fact is contradicted
|
||||
- **THEN** groundedness_score SHALL be computed from the mapped values of critical facts
|
||||
- **AND** the implementation SHALL use the fixed mapping `direct_evidence=1.0`, `indirect_support=0.6`, `no_evidence=0.0`
|
||||
- **AND** the result SHALL be clamped into `[0.0, 1.0]`
|
||||
|
||||
### Requirement: ChatService SHALL route based on Verifier verdict
|
||||
The system SHALL use ChatService for explicit single-round `Planner → Executor → Verifier` orchestration and SHALL use ChatService to control whether an additional round is allowed.
|
||||
|
||||
#### Scenario: PASS → direct output
|
||||
- **WHEN** Verifier outputs verdict="PASS"
|
||||
- **THEN** the system SHALL output the Executor's answer directly
|
||||
|
||||
#### Scenario: LOW_CONFID score≥0.5 → output with disclaimer
|
||||
- **WHEN** Verifier outputs verdict="LOW_CONFID" with groundedness_score ≥ 0.5
|
||||
- **THEN** the system SHALL output the Executor's answer prefixed with a fixed confidence disclaimer
|
||||
|
||||
#### Scenario: LOW_CONFID score<0.5 → trigger one additional round
|
||||
- **WHEN** Verifier outputs verdict="LOW_CONFID" with groundedness_score < 0.5 and this is the first callback
|
||||
- **THEN** the ChatService SHALL invoke one additional `Planner → Executor → Verifier` round to supplement evidence
|
||||
- **AND** after the second Verifier run, verdict="LOW_CONFID" SHALL be output with a confidence disclaimer
|
||||
- **AND** after the second Verifier run, verdict="REJECT" SHALL still produce a degraded output
|
||||
|
||||
#### Scenario: REJECT does not enter retry round
|
||||
- **WHEN** Verifier outputs verdict="REJECT"
|
||||
- **THEN** the system SHALL NOT start a retry round for evidence补充
|
||||
- **AND** it SHALL produce a degraded output directly
|
||||
|
||||
#### Scenario: REJECT → degraded output
|
||||
- **WHEN** Verifier outputs verdict="REJECT"
|
||||
- **THEN** the system SHALL output a degraded result indicating the answer cannot be reliably generated
|
||||
- **AND** it SHALL NOT pass through the raw Executor answer
|
||||
|
||||
### Requirement: User-facing verifier outputs SHALL follow fixed templates
|
||||
The system SHALL use fixed output protocols for LOW_CONFID and REJECT user-facing responses.
|
||||
|
||||
#### Scenario: LOW_CONFID uses disclaimer template
|
||||
- **WHEN** the final verdict is `LOW_CONFID`
|
||||
- **THEN** the user-facing response SHALL prepend a fixed disclaimer before the Executor answer
|
||||
- **AND** optional evidence gaps, if present, SHALL come only from verifier-identified critical gaps
|
||||
|
||||
#### Scenario: REJECT uses degraded template
|
||||
- **WHEN** the final verdict is `REJECT`
|
||||
- **THEN** the user-facing response SHALL use a degraded template
|
||||
- **AND** it SHALL include only confirmed facts, evidence gaps, and next-step suggestions
|
||||
- **AND** it SHALL NOT include unverified raw answer content
|
||||
|
||||
### Requirement: Verifier SHALL be observable
|
||||
The Verifier's verdict SHALL be persisted for observability.
|
||||
|
||||
#### Scenario: verdict written to self_evaluation
|
||||
- **WHEN** the Verifier produces a verdict
|
||||
- **THEN** the ChatService SHALL write the verdict data under `diagnosis_session.self_evaluation.verifier_evaluation`
|
||||
- **AND** existing `rule_evaluation` data SHALL be preserved
|
||||
|
||||
### Requirement: self_evaluation SHALL be a container object
|
||||
The `diagnosis_session.self_evaluation` field SHALL store multiple evaluation channels in one JSON object.
|
||||
|
||||
#### Scenario: rule evaluation stored separately
|
||||
- **WHEN** the rule-based evidence scoring completes
|
||||
- **THEN** the EvaluationService SHALL write the result under `rule_evaluation`
|
||||
- **AND** existing `verifier_evaluation` data SHALL be preserved
|
||||
|
||||
#### Scenario: verifier evaluation stored separately
|
||||
- **WHEN** the Verifier completes
|
||||
- **THEN** the ChatService SHALL write the result under `verifier_evaluation`
|
||||
- **AND** existing `rule_evaluation` data SHALL be preserved
|
||||
|
||||
#### Scenario: no whole-object overwrite after initialization
|
||||
- **WHEN** either evaluation channel updates `self_evaluation`
|
||||
- **THEN** the implementation SHALL use read-modify-write semantics
|
||||
- **AND** it SHALL NOT replace the whole JSON object except when initializing from null
|
||||
|
||||
### Requirement: Verifier SHALL consume explicit verification inputs
|
||||
The Verifier SHALL receive explicit verification inputs rather than inferring them only from raw conversation history.
|
||||
|
||||
#### Scenario: explicit input blocks available to Verifier
|
||||
- **WHEN** the Verifier starts
|
||||
- **THEN** the system SHALL provide `original_query`, `executor_final_answer`, and `tool_trace_summary` as explicit inputs
|
||||
- **AND** `retry_context` SHALL be provided on the second round only
|
||||
- **AND** message filtering MAY be used only to remove intermediate reasoning or unrelated noise
|
||||
|
||||
#### Scenario: tool trace summary derived from tool facts
|
||||
- **WHEN** the system prepares verifier inputs
|
||||
- **THEN** `tool_trace_summary` SHALL be generated from tool invocation facts
|
||||
- **AND** each summary item SHALL include tool name, success state, input summary, output summary, and evidence level
|
||||
- **AND** raw conversation history SHALL NOT be the only source of verifier evidence context
|
||||
|
||||
#### Scenario: tool trace summary preserves invocation references
|
||||
- **WHEN** the system prepares verifier inputs
|
||||
- **THEN** each summary item SHALL include a stable `trace_ref`
|
||||
- **AND** each summary item SHALL preserve `source_invocation_ids` for the tool invocation rows that contributed to the summary
|
||||
- **AND** each summary item SHOULD include query samples, retrieval layers, relevance levels, and source document labels when available
|
||||
|
||||
#### Scenario: only evidence-bearing tools included
|
||||
- **WHEN** the system generates `tool_trace_summary`
|
||||
- **THEN** it SHALL include only evidence-bearing tool invocations
|
||||
- **AND** non-evidence helper tools such as time or formatting tools SHALL be excluded by default
|
||||
|
||||
#### Scenario: failed evidence calls preserved as evidence gaps
|
||||
- **WHEN** an evidence-bearing tool invocation fails or returns no usable evidence
|
||||
- **THEN** the summary SHALL still include that invocation
|
||||
- **AND** it SHALL mark the entry as unsuccessful with an evidence level representing no evidence
|
||||
|
||||
#### Scenario: repeated tool calls may be compacted
|
||||
- **WHEN** repeated tool invocations concern the same tool, topic domain, and round
|
||||
- **THEN** the system MAY compact them into a merged summary entry
|
||||
- **AND** the merged entry SHALL preserve the first effective hit and the count of repeated, failed, or no-hit calls
|
||||
|
||||
#### Scenario: raw outputs not passed through in full
|
||||
- **WHEN** a tool invocation returns large raw content
|
||||
- **THEN** `tool_trace_summary` SHALL keep only a minimal evidence summary
|
||||
- **AND** the raw output SHALL NOT be passed through in full to the Verifier
|
||||
|
||||
#### Scenario: MessagesModelHook used only for noise reduction
|
||||
- **WHEN** a MessagesModelHook is used for the Verifier
|
||||
- **THEN** it MAY remove intermediate reasoning or irrelevant messages
|
||||
- **AND** it SHALL NOT be the primary source for assembling verifier business inputs
|
||||
|
||||
### Requirement: Verifier facts SHALL be auditable
|
||||
Verifier facts SHALL be linkable to the evidence summaries used during verification.
|
||||
|
||||
#### Scenario: facts_checked contains evidence refs
|
||||
- **WHEN** the Verifier emits `facts_checked`
|
||||
- **THEN** each fact SHALL include `evidence_refs`
|
||||
- **AND** each evidence ref SHALL point to an existing `tool_trace_summary.trace_ref`
|
||||
- **AND** each evidence ref SHALL preserve the relevant `source_invocation_ids` when available
|
||||
|
||||
#### Scenario: verifier evaluation persists traceability snapshot
|
||||
- **WHEN** the ChatService persists `verifier_evaluation`
|
||||
- **THEN** it SHALL include `traceability_version`
|
||||
- **AND** it SHALL include the `tool_trace_summary` snapshot used by the Verifier
|
||||
@@ -0,0 +1,37 @@
|
||||
## Purpose
|
||||
|
||||
Provide a repeatable MVP demo flow that can run a chat diagnosis, expose its persisted execution trace, and submit feedback for the same session id.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Diagnosis trace can be queried by session id
|
||||
The system SHALL expose a read-only HTTP endpoint `GET /api/diagnosis/{sessionId}/trace` that returns the persisted diagnosis trace for the requested session id.
|
||||
|
||||
#### Scenario: Existing session trace is returned
|
||||
- **WHEN** a caller requests trace data for a session id that exists in `diagnosis_session`
|
||||
- **THEN** the system returns a success response containing the session summary, ordered agent steps, ordered tool invocations, self-evaluation data, final answer, and feedback
|
||||
|
||||
#### Scenario: Missing session returns not found
|
||||
- **WHEN** a caller requests trace data for a session id that does not exist in `diagnosis_session`
|
||||
- **THEN** the system returns a 404 response using the existing session-not-found error contract
|
||||
|
||||
### Requirement: Trace aggregation is read-only
|
||||
The system MUST build trace output from existing persisted diagnosis tables and MUST NOT mutate diagnosis sessions, agent steps, tool invocations, feedback, or chat session state while serving the trace request.
|
||||
|
||||
#### Scenario: Trace query does not change persisted state
|
||||
- **WHEN** a caller requests `GET /api/diagnosis/{sessionId}/trace`
|
||||
- **THEN** the system reads `diagnosis_session`, `agent_step`, and `tool_invocation` records and returns an aggregate without saving any of those records
|
||||
|
||||
### Requirement: MVP demo profile is available
|
||||
The system SHALL provide an `mvp-demo` Spring profile that documents the demo runtime intent and keeps mock log and metric providers enabled for repeatable diagnosis demonstrations.
|
||||
|
||||
#### Scenario: Demo profile loads mock evidence providers
|
||||
- **WHEN** the application starts with `--spring.profiles.active=mvp-demo`
|
||||
- **THEN** `prometheus.mock-enabled` and `cls.mock-enabled` are enabled by profile configuration
|
||||
|
||||
### Requirement: End-to-end MVP acceptance case is documented
|
||||
The project SHALL include an end-to-end acceptance case that demonstrates start-up, chat diagnosis, trace query, and feedback submission using the same session id.
|
||||
|
||||
#### Scenario: Reviewer follows the acceptance case
|
||||
- **WHEN** a reviewer follows the documented MVP demo acceptance steps
|
||||
- **THEN** they can run the application, submit a diagnosis question, query the trace endpoint, and submit feedback for the same session id
|
||||
@@ -0,0 +1,82 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
通用 MySQL 查询脚本
|
||||
用法：
|
||||
python scripts/query_mysql.py "SELECT * FROM diagnosis_session ORDER BY created_at DESC LIMIT 5"
|
||||
python scripts/query_mysql.py # 交互模式
|
||||
依赖：pip install pymysql
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Windows 控制台 UTF-8 输出
|
||||
if sys.stdout.encoding and sys.stdout.encoding.lower() != 'utf-8':
|
||||
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
|
||||
|
||||
try:
|
||||
import pymysql
|
||||
import pymysql.cursors
|
||||
except ImportError:
|
||||
print("缺少依赖，请先执行: pip install pymysql")
|
||||
sys.exit(1)
|
||||
|
||||
# 从 application.yml 读取的连接信息
|
||||
DB_CONFIG = {
|
||||
"host": "119.29.78.52",
|
||||
"port": 33306,
|
||||
"user": "root",
|
||||
"password": "!Fucker123..",
|
||||
"database": "superbiz_agent",
|
||||
"charset": "utf8mb4",
|
||||
"cursorclass": pymysql.cursors.DictCursor,
|
||||
}
|
||||
|
||||
|
||||
def run_query(sql: str):
|
||||
conn = pymysql.connect(**DB_CONFIG)
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(sql)
|
||||
if sql.strip().upper().startswith("SELECT") or sql.strip().upper().startswith("SHOW"):
|
||||
rows = cur.fetchall()
|
||||
if not rows:
|
||||
print("(空结果)")
|
||||
return
|
||||
# 打印列头
|
||||
cols = list(rows[0].keys())
|
||||
col_widths = {c: max(len(c), max(len(str(r[c])) for r in rows)) for c in cols}
|
||||
header = " | ".join(c.ljust(col_widths[c]) for c in cols)
|
||||
print(header)
|
||||
print("-" * len(header))
|
||||
for row in rows:
|
||||
print(" | ".join(str(row[c]).ljust(col_widths[c]) for c in cols))
|
||||
print(f"\n({len(rows)} 行)")
|
||||
else:
|
||||
conn.commit()
|
||||
print(f"OK，影响行数: {cur.rowcount}")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) > 1:
|
||||
sql = " ".join(sys.argv[1:])
|
||||
run_query(sql)
|
||||
else:
|
||||
print("MySQL 交互模式（输入 exit 退出）")
|
||||
print(f"连接：{DB_CONFIG['user']}@{DB_CONFIG['host']}:{DB_CONFIG['port']}/{DB_CONFIG['database']}")
|
||||
print("-" * 50)
|
||||
while True:
|
||||
try:
|
||||
sql = input("sql> ").strip()
|
||||
if sql.lower() in ("exit", "quit", "q"):
|
||||
break
|
||||
if not sql:
|
||||
continue
|
||||
run_query(sql)
|
||||
except KeyboardInterrupt:
|
||||
break
|
||||
except Exception as e:
|
||||
print(f"错误: {e}")
|
||||
@@ -2,6 +2,7 @@ package com.superbiz.agent.agent.tool;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.service.ToolInvocationRecorder;
|
||||
import lombok.Data;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
@@ -34,6 +35,11 @@ public class QueryLogsTools {
|
||||
public static final String TOOL_GET_AVAILABLE_LOG_TOPICS = "getAvailableLogTopics";
|
||||
|
||||
private final ObjectMapper objectMapper = new ObjectMapper();
|
||||
private final ToolInvocationRecorder toolInvocationRecorder;
|
||||
|
||||
public QueryLogsTools(ToolInvocationRecorder toolInvocationRecorder) {
|
||||
this.toolInvocationRecorder = toolInvocationRecorder;
|
||||
}
|
||||
|
||||
@Value("${cls.mock-enabled:false}")
|
||||
private boolean mockEnabled;
|
||||
@@ -55,6 +61,7 @@ public class QueryLogsTools {
|
||||
"Call this tool first before querying logs to understand what log topics are available. " +
|
||||
"Returns a list of log topics with their names, descriptions, and example queries.")
|
||||
public String getAvailableLogTopics() {
|
||||
long startTime = System.currentTimeMillis();
|
||||
logger.info("获取可用的日志主题列表");
|
||||
|
||||
try {
|
||||
@@ -123,11 +130,15 @@ public class QueryLogsTools {
|
||||
|
||||
output.setMessage(String.format("共有 %d 个可用的日志主题。建议使用默认地域 'ap-guangzhou' 或省略 region 参数", topics.size()));
|
||||
|
||||
return objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(output);
|
||||
String response = objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(output);
|
||||
recordInvocation(startTime, "get_available_log_topics", null, null, null, response, true, null, "logs");
|
||||
return response;
|
||||
|
||||
} catch (Exception e) {
|
||||
logger.error("获取日志主题列表失败", e);
|
||||
return "{\"success\":false,\"message\":\"获取日志主题列表失败: " + e.getMessage() + "\"}";
|
||||
String response = "{\"success\":false,\"message\":\"获取日志主题列表失败: " + e.getMessage() + "\"}";
|
||||
recordInvocation(startTime, "get_available_log_topics", null, null, null, response, false, e.getMessage(), "logs");
|
||||
return response;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -164,6 +175,7 @@ public class QueryLogsTools {
|
||||
@ToolParam(description = "查询条件，支持 Lucene 语法，如 level:ERROR OR cpu_usage:>80；为空时返回该主题近 5 条核心日志") String query,
|
||||
@ToolParam(description = "返回日志条数，默认20，最大100") Integer limit) {
|
||||
|
||||
long startTime = System.currentTimeMillis();
|
||||
int actualLimit = (limit == null || limit <= 0) ? 20 : Math.min(limit, 100);
|
||||
|
||||
String safeQuery = query == null ? "" : query;
|
||||
@@ -178,7 +190,10 @@ public class QueryLogsTools {
|
||||
logger.info("使用 Mock 数据，返回 {} 条日志", logEntries.size());
|
||||
} else {
|
||||
// 真实模式：调用 CLS API（这里预留接口，后续实现）
|
||||
return buildErrorResponse("CLS 真实查询尚未实现，请启用 mock 模式进行测试");
|
||||
String response = buildErrorResponse("CLS 真实查询尚未实现，请启用 mock 模式进行测试");
|
||||
recordInvocation(startTime, safeQuery, region, logTopic, actualLimit, response, false,
|
||||
"CLS 真实查询尚未实现，请启用 mock 模式进行测试", normalizeTopicDomain(logTopic));
|
||||
return response;
|
||||
}
|
||||
|
||||
// 构建成功响应
|
||||
@@ -193,15 +208,51 @@ public class QueryLogsTools {
|
||||
|
||||
String jsonResult = objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(output);
|
||||
logger.info("日志查询完成: 找到 {} 条日志", logEntries.size());
|
||||
recordInvocation(startTime, safeQuery, region, logTopic, actualLimit, jsonResult,
|
||||
!logEntries.isEmpty(), logEntries.isEmpty() ? "未找到匹配的日志" : null,
|
||||
normalizeTopicDomain(logTopic));
|
||||
|
||||
return jsonResult;
|
||||
|
||||
} catch (Exception e) {
|
||||
logger.error("查询日志失败", e);
|
||||
return buildErrorResponse("查询失败: " + e.getMessage());
|
||||
String response = buildErrorResponse("查询失败: " + e.getMessage());
|
||||
recordInvocation(startTime, safeQuery, region, logTopic, actualLimit, response, false,
|
||||
e.getMessage(), normalizeTopicDomain(logTopic));
|
||||
return response;
|
||||
}
|
||||
}
|
||||
|
||||
private void recordInvocation(long startTime, String query, String region, String logTopic, Integer limit,
|
||||
String output, boolean success, String errorMessage, String topicDomain) {
|
||||
Map<String, Object> input = new HashMap<>();
|
||||
input.put("query", query == null || query.isBlank() ? "DEFAULT_QUERY" : query);
|
||||
if (region != null) {
|
||||
input.put("region", region);
|
||||
}
|
||||
if (logTopic != null) {
|
||||
input.put("log_topic", logTopic);
|
||||
}
|
||||
if (limit != null) {
|
||||
input.put("limit", limit);
|
||||
}
|
||||
input.put("mock_enabled", mockEnabled);
|
||||
|
||||
toolInvocationRecorder.recordEvidenceTool(
|
||||
"query_logs",
|
||||
input,
|
||||
output,
|
||||
success,
|
||||
startTime,
|
||||
errorMessage,
|
||||
topicDomain
|
||||
);
|
||||
}
|
||||
|
||||
private String normalizeTopicDomain(String logTopic) {
|
||||
return logTopic == null || logTopic.isBlank() ? "logs" : logTopic;
|
||||
}
|
||||
|
||||
/**
|
||||
* 构建 Mock 日志数据
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ package com.superbiz.agent.agent.tool;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.service.ToolInvocationRecorder;
|
||||
import lombok.Data;
|
||||
import okhttp3.OkHttpClient;
|
||||
import okhttp3.Request;
|
||||
@@ -30,6 +31,11 @@ public class QueryMetricsTools {
|
||||
public static final String TOOL_QUERY_PROMETHEUS_ALERTS = "queryPrometheusAlerts";
|
||||
|
||||
private final ObjectMapper objectMapper = new ObjectMapper();
|
||||
private final ToolInvocationRecorder toolInvocationRecorder;
|
||||
|
||||
public QueryMetricsTools(ToolInvocationRecorder toolInvocationRecorder) {
|
||||
this.toolInvocationRecorder = toolInvocationRecorder;
|
||||
}
|
||||
|
||||
@Value("${prometheus.base-url}")
|
||||
private String prometheusBaseUrl;
|
||||
@@ -59,6 +65,7 @@ public class QueryMetricsTools {
|
||||
"This tool retrieves all currently active/firing alerts including their labels, annotations, state, and values. " +
|
||||
"Use this tool when you need to check what alerts are currently firing, investigate alert conditions, or monitor alert status.")
|
||||
public String queryPrometheusAlerts() {
|
||||
long startTime = System.currentTimeMillis();
|
||||
logger.info("开始查询 Prometheus 活动告警, Mock模式: {}", mockEnabled);
|
||||
|
||||
try {
|
||||
@@ -73,7 +80,9 @@ public class QueryMetricsTools {
|
||||
PrometheusAlertsResult result = fetchPrometheusAlerts();
|
||||
|
||||
if (!"success".equals(result.getStatus())) {
|
||||
return buildErrorResponse("Prometheus API 返回非成功状态: " + result.getStatus(), result.getError());
|
||||
String response = buildErrorResponse("Prometheus API 返回非成功状态: " + result.getStatus(), result.getError());
|
||||
recordInvocation(startTime, response, false, result.getError());
|
||||
return response;
|
||||
}
|
||||
|
||||
// 转换为简化格式，对于相同的 alertname，只保留第一个
|
||||
@@ -110,14 +119,29 @@ public class QueryMetricsTools {
|
||||
|
||||
String jsonResult = objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(output);
|
||||
logger.info("Prometheus 告警查询完成: 找到 {} 个告警", simplifiedAlerts.size());
|
||||
recordInvocation(startTime, jsonResult, true, null);
|
||||
|
||||
return jsonResult;
|
||||
|
||||
} catch (Exception e) {
|
||||
logger.error("查询 Prometheus 告警失败", e);
|
||||
return buildErrorResponse("查询失败", e.getMessage());
|
||||
String response = buildErrorResponse("查询失败", e.getMessage());
|
||||
recordInvocation(startTime, response, false, e.getMessage());
|
||||
return response;
|
||||
}
|
||||
}
|
||||
|
||||
private void recordInvocation(long startTime, String output, boolean success, String errorMessage) {
|
||||
toolInvocationRecorder.recordEvidenceTool(
|
||||
"query_metrics",
|
||||
Map.of("query", "active_prometheus_alerts", "mock_enabled", mockEnabled),
|
||||
output,
|
||||
success,
|
||||
startTime,
|
||||
errorMessage,
|
||||
"prometheus_alerts"
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* 构建 Mock 告警数据
|
||||
|
||||
@@ -1,32 +1,30 @@
|
||||
package com.superbiz.agent.controller;
|
||||
|
||||
import com.alibaba.cloud.ai.graph.NodeOutput;
|
||||
import com.alibaba.cloud.ai.graph.OverAllState;
|
||||
import com.alibaba.cloud.ai.graph.agent.ReactAgent;
|
||||
import com.alibaba.cloud.ai.graph.streaming.OutputType;
|
||||
import com.alibaba.cloud.ai.graph.streaming.StreamingOutput;
|
||||
import lombok.Getter;
|
||||
import lombok.Setter;
|
||||
import com.superbiz.agent.domain.model.SessionContext;
|
||||
import com.superbiz.agent.service.AiOpsService;
|
||||
import com.superbiz.agent.service.ChatService;
|
||||
import com.superbiz.agent.service.session.SessionManager;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.chat.model.ChatModel;
|
||||
import org.springframework.ai.tool.ToolCallback;
|
||||
import org.springframework.ai.tool.ToolCallbackProvider;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.*;
|
||||
import org.springframework.web.servlet.mvc.method.annotation.SseEmitter;
|
||||
import reactor.core.publisher.Flux;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.time.LocalDateTime;
|
||||
import java.time.ZoneId;
|
||||
import java.util.*;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.locks.ReentrantLock;
|
||||
|
||||
/**
|
||||
* 统一 API 控制器
|
||||
@@ -44,17 +42,20 @@ public class ChatController {
|
||||
@Autowired
|
||||
private ChatService chatService;
|
||||
|
||||
@Autowired
|
||||
private SessionManager sessionManager;
|
||||
|
||||
@Autowired(required = false)
|
||||
private ToolCallbackProvider tools;
|
||||
|
||||
private final ExecutorService executor = Executors.newCachedThreadPool();
|
||||
|
||||
// 存储会话信息
|
||||
private final Map<String, SessionInfo> sessions = new ConcurrentHashMap<>();
|
||||
|
||||
// 最大历史消息窗口大小（成对计算：用户消息+AI回复=1对）
|
||||
private static final int MAX_WINDOW_SIZE = 6;
|
||||
|
||||
@Value("${session.ttl-seconds:3600}")
|
||||
private long sessionTtlSeconds;
|
||||
|
||||
/**
|
||||
* 普通对话接口（支持工具调用）
|
||||
* 与 /chat_react 逻辑一致，但直接返回完整结果而非流式输出
|
||||
@@ -71,10 +72,10 @@ public class ChatController {
|
||||
}
|
||||
|
||||
// 获取或创建会话
|
||||
SessionInfo session = getOrCreateSession(request.getId());
|
||||
SessionContext session = getOrCreateSession(request.getId());
|
||||
|
||||
// 获取历史消息
|
||||
List<Map<String, String>> history = session.getHistory();
|
||||
List<Map<String, String>> history = session.getMessageHistorySnapshot();
|
||||
logger.info("会话历史消息对数: {}", history.size() / 2);
|
||||
|
||||
// 获取注入的 ChatModel
|
||||
@@ -88,13 +89,14 @@ public class ChatController {
|
||||
// 根据问题复杂度自动选择单 Agent 或多 Agent
|
||||
logger.info("开始 ReactAgent 对话（支持自动工具调用）");
|
||||
ChatService.ChatResult result = chatService.executeChatWithStrategy(chatModel, toolCallbacks,
|
||||
request.getQuestion(), history);
|
||||
request.getQuestion(), history, session.getSessionId());
|
||||
String fullAnswer = result.answer();
|
||||
|
||||
// 更新会话历史
|
||||
session.addMessage(request.getQuestion(), fullAnswer);
|
||||
session.addChatMessagePair(request.getQuestion(), fullAnswer, MAX_WINDOW_SIZE);
|
||||
sessionManager.updateSession(session);
|
||||
logger.info("已更新会话历史 - SessionId: {}, 当前消息对数: {}",
|
||||
request.getId(), session.getMessagePairCount());
|
||||
session.getSessionId(), session.getMessagePairCount());
|
||||
|
||||
return ResponseEntity.ok(ApiResponse.success(ChatResponse.success(fullAnswer, result.sessionId())));
|
||||
|
||||
@@ -116,9 +118,11 @@ public class ChatController {
|
||||
return ResponseEntity.ok(ApiResponse.error("会话ID不能为空"));
|
||||
}
|
||||
|
||||
SessionInfo session = sessions.get(request.getId());
|
||||
if (session != null) {
|
||||
session.clearHistory();
|
||||
Optional<SessionContext> session = sessionManager.getSession(request.getId());
|
||||
if (session.isPresent()) {
|
||||
SessionContext context = session.get();
|
||||
context.clearMessageHistory();
|
||||
sessionManager.updateSession(context);
|
||||
return ResponseEntity.ok(ApiResponse.success("会话历史已清空"));
|
||||
} else {
|
||||
return ResponseEntity.ok(ApiResponse.error("会话不存在"));
|
||||
@@ -131,8 +135,8 @@ public class ChatController {
|
||||
}
|
||||
|
||||
/**
|
||||
* ReactAgent 对话接口（SSE 流式模式，支持多轮对话，支持自动工具调用，例如获取当前时间，查询日志，告警等）
|
||||
* 支持 session 管理，保留对话历史
|
||||
* 对话接口（SSE 流式模式）
|
||||
* 与 /chat 使用同一条 ChatService 策略链路，区别仅在于通过 SSE 分块返回最终答案。
|
||||
*/
|
||||
@PostMapping(value = "/chat_stream", produces = "text/event-stream;charset=UTF-8")
|
||||
public SseEmitter chatStream(@RequestBody ChatRequest request) {
|
||||
@@ -155,10 +159,10 @@ public class ChatController {
|
||||
logger.info("收到 ReactAgent 对话请求 - SessionId: {}, Question: {}", request.getId(), request.getQuestion());
|
||||
|
||||
// 获取或创建会话
|
||||
SessionInfo session = getOrCreateSession(request.getId());
|
||||
SessionContext session = getOrCreateSession(request.getId());
|
||||
|
||||
// 获取历史消息
|
||||
List<Map<String, String>> history = session.getHistory();
|
||||
List<Map<String, String>> history = session.getMessageHistorySnapshot();
|
||||
logger.info("ReactAgent 会话历史消息对数: {}", history.size() / 2);
|
||||
|
||||
// 获取注入的 ChatModel
|
||||
@@ -167,92 +171,25 @@ public class ChatController {
|
||||
// 记录可用工具
|
||||
chatService.logAvailableTools();
|
||||
|
||||
logger.info("开始 ReactAgent 流式对话（支持自动工具调用）");
|
||||
|
||||
// 构建系统提示词（包含历史消息）
|
||||
String systemPrompt = chatService.buildSystemPrompt(history);
|
||||
|
||||
// 创建 ReactAgent
|
||||
ReactAgent agent = chatService.createReactAgent(chatModel, systemPrompt);
|
||||
|
||||
// 用于累积完整答案
|
||||
StringBuilder fullAnswerBuilder = new StringBuilder();
|
||||
|
||||
// 使用 agent.stream() 进行流式对话
|
||||
Flux<NodeOutput> stream = agent.stream(request.getQuestion());
|
||||
|
||||
stream.subscribe(
|
||||
output -> {
|
||||
try {
|
||||
// 检查是否为 StreamingOutput 类型
|
||||
if (output instanceof StreamingOutput streamingOutput) {
|
||||
OutputType type = streamingOutput.getOutputType();
|
||||
|
||||
// 处理模型推理的流式输出
|
||||
if (type == OutputType.AGENT_MODEL_STREAMING) {
|
||||
// 流式增量内容，逐步显示
|
||||
String chunk = streamingOutput.message().getText();
|
||||
if (chunk != null && !chunk.isEmpty()) {
|
||||
fullAnswerBuilder.append(chunk);
|
||||
|
||||
// 实时发送到前端
|
||||
emitter.send(SseEmitter.event()
|
||||
.name("message")
|
||||
.data(SseMessage.content(chunk), MediaType.APPLICATION_JSON));
|
||||
|
||||
logger.info("发送流式内容: {}", chunk);
|
||||
}
|
||||
} else if (type == OutputType.AGENT_MODEL_FINISHED) {
|
||||
// 模型推理完成
|
||||
logger.info("模型输出完成");
|
||||
} else if (type == OutputType.AGENT_TOOL_FINISHED) {
|
||||
// 工具调用完成
|
||||
logger.info("工具调用完成: {}", output.node());
|
||||
} else if (type == OutputType.AGENT_HOOK_FINISHED) {
|
||||
// Hook 执行完成
|
||||
logger.debug("Hook 执行完成: {}", output.node());
|
||||
}
|
||||
}
|
||||
} catch (IOException e) {
|
||||
logger.error("发送流式消息失败", e);
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
},
|
||||
error -> {
|
||||
// 错误处理
|
||||
logger.error("ReactAgent 流式对话失败", error);
|
||||
try {
|
||||
emitter.send(SseEmitter.event()
|
||||
.name("message")
|
||||
.data(SseMessage.error(error.getMessage()), MediaType.APPLICATION_JSON));
|
||||
} catch (IOException ex) {
|
||||
logger.error("发送错误消息失败", ex);
|
||||
}
|
||||
emitter.completeWithError(error);
|
||||
},
|
||||
() -> {
|
||||
// 完成处理
|
||||
try {
|
||||
String fullAnswer = fullAnswerBuilder.toString();
|
||||
logger.info("ReactAgent 流式对话完成 - SessionId: {}, 答案长度: {}",
|
||||
request.getId(), fullAnswer.length());
|
||||
|
||||
// 更新会话历史
|
||||
session.addMessage(request.getQuestion(), fullAnswer);
|
||||
logger.info("已更新会话历史 - SessionId: {}, 当前消息对数: {}",
|
||||
request.getId(), session.getMessagePairCount());
|
||||
|
||||
// 发送完成标记
|
||||
emitter.send(SseEmitter.event()
|
||||
.name("message")
|
||||
.data(SseMessage.done(), MediaType.APPLICATION_JSON));
|
||||
emitter.complete();
|
||||
} catch (IOException e) {
|
||||
logger.error("发送完成消息失败", e);
|
||||
emitter.completeWithError(e);
|
||||
}
|
||||
}
|
||||
);
|
||||
ToolCallback[] toolCallbacks = tools != null ? tools.getToolCallbacks() : new ToolCallback[0];
|
||||
|
||||
logger.info("开始统一 ChatService 对话（SSE 分块返回）");
|
||||
ChatService.ChatResult result = chatService.executeChatWithStrategy(chatModel, toolCallbacks,
|
||||
request.getQuestion(), history, session.getSessionId());
|
||||
String fullAnswer = result.answer() == null ? "" : result.answer();
|
||||
logger.info("统一 ChatService 对话完成 - SessionId: {}, 答案长度: {}",
|
||||
result.sessionId(), fullAnswer.length());
|
||||
|
||||
session.addChatMessagePair(request.getQuestion(), fullAnswer, MAX_WINDOW_SIZE);
|
||||
sessionManager.updateSession(session);
|
||||
logger.info("已更新会话历史 - SessionId: {}, 当前消息对数: {}",
|
||||
session.getSessionId(), session.getMessagePairCount());
|
||||
|
||||
sendContentChunks(emitter, fullAnswer);
|
||||
emitter.send(SseEmitter.event()
|
||||
.name("message")
|
||||
.data(SseMessage.done(), MediaType.APPLICATION_JSON));
|
||||
emitter.complete();
|
||||
|
||||
} catch (Exception e) {
|
||||
logger.error("ReactAgent 对话初始化失败", e);
|
||||
@@ -365,12 +302,13 @@ public class ChatController {
|
||||
try {
|
||||
logger.info("收到获取会话信息请求 - SessionId: {}", sessionId);
|
||||
|
||||
SessionInfo session = sessions.get(sessionId);
|
||||
if (session != null) {
|
||||
Optional<SessionContext> session = sessionManager.getSession(sessionId);
|
||||
if (session.isPresent()) {
|
||||
SessionContext context = session.get();
|
||||
SessionInfoResponse response = new SessionInfoResponse();
|
||||
response.setSessionId(sessionId);
|
||||
response.setMessagePairCount(session.getMessagePairCount());
|
||||
response.setCreateTime(session.createTime);
|
||||
response.setMessagePairCount(context.getMessagePairCount());
|
||||
response.setCreateTime(toEpochMillis(context.getCreatedAt()));
|
||||
return ResponseEntity.ok(ApiResponse.success(response));
|
||||
} else {
|
||||
return ResponseEntity.ok(ApiResponse.error("会话不存在"));
|
||||
@@ -384,107 +322,39 @@ public class ChatController {
|
||||
|
||||
// ==================== 辅助方法 ====================
|
||||
|
||||
private SessionInfo getOrCreateSession(String sessionId) {
|
||||
if (sessionId == null || sessionId.isEmpty()) {
|
||||
sessionId = UUID.randomUUID().toString();
|
||||
}
|
||||
return sessions.computeIfAbsent(sessionId, SessionInfo::new);
|
||||
private SessionContext getOrCreateSession(String sessionId) {
|
||||
String resolvedSessionId = (sessionId == null || sessionId.isEmpty())
|
||||
? UUID.randomUUID().toString()
|
||||
: sessionId;
|
||||
return sessionManager.getSession(resolvedSessionId)
|
||||
.orElseGet(() -> {
|
||||
SessionContext context = SessionContext.builder()
|
||||
.sessionId(resolvedSessionId)
|
||||
.status("ACTIVE")
|
||||
.ttl(sessionTtlSeconds)
|
||||
.build();
|
||||
sessionManager.createSession(context, sessionTtlSeconds);
|
||||
return context;
|
||||
});
|
||||
}
|
||||
|
||||
// ==================== 内部类 ====================
|
||||
|
||||
/**
|
||||
* 会话信息
|
||||
* 管理单个会话的历史消息，支持自动清理和线程安全
|
||||
*/
|
||||
private static class SessionInfo {
|
||||
private final String sessionId;
|
||||
// 存储历史消息对：[{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]
|
||||
private final List<Map<String, String>> messageHistory;
|
||||
private final long createTime;
|
||||
private final ReentrantLock lock;
|
||||
|
||||
public SessionInfo(String sessionId) {
|
||||
this.sessionId = sessionId;
|
||||
this.messageHistory = new ArrayList<>();
|
||||
this.createTime = System.currentTimeMillis();
|
||||
this.lock = new ReentrantLock();
|
||||
private long toEpochMillis(LocalDateTime time) {
|
||||
if (time == null) {
|
||||
return 0L;
|
||||
}
|
||||
return time.atZone(ZoneId.systemDefault()).toInstant().toEpochMilli();
|
||||
}
|
||||
|
||||
/**
|
||||
* 添加一对消息（用户问题 + AI回复）
|
||||
* 自动管理历史消息窗口大小
|
||||
*/
|
||||
public void addMessage(String userQuestion, String aiAnswer) {
|
||||
lock.lock();
|
||||
try {
|
||||
// 添加用户消息
|
||||
Map<String, String> userMsg = new HashMap<>();
|
||||
userMsg.put("role", "user");
|
||||
userMsg.put("content", userQuestion);
|
||||
messageHistory.add(userMsg);
|
||||
|
||||
// 添加AI回复
|
||||
Map<String, String> assistantMsg = new HashMap<>();
|
||||
assistantMsg.put("role", "assistant");
|
||||
assistantMsg.put("content", aiAnswer);
|
||||
messageHistory.add(assistantMsg);
|
||||
|
||||
// 自动清理：保持最多 MAX_WINDOW_SIZE 对消息
|
||||
// 每对消息包含2条记录（user + assistant）
|
||||
int maxMessages = MAX_WINDOW_SIZE * 2;
|
||||
while (messageHistory.size() > maxMessages) {
|
||||
// 成对删除最旧的消息（删除前2条）
|
||||
messageHistory.remove(0); // 删除最旧的用户消息
|
||||
if (!messageHistory.isEmpty()) {
|
||||
messageHistory.remove(0); // 删除对应的AI回复
|
||||
}
|
||||
}
|
||||
|
||||
logger.debug("会话 {} 更新历史消息，当前消息对数: {}",
|
||||
sessionId, messageHistory.size() / 2);
|
||||
|
||||
} finally {
|
||||
lock.unlock();
|
||||
}
|
||||
private void sendContentChunks(SseEmitter emitter, String content) throws IOException {
|
||||
if (content == null || content.isEmpty()) {
|
||||
return;
|
||||
}
|
||||
|
||||
/**
|
||||
* 获取历史消息（线程安全）
|
||||
* 返回副本以避免并发修改
|
||||
*/
|
||||
public List<Map<String, String>> getHistory() {
|
||||
lock.lock();
|
||||
try {
|
||||
return new ArrayList<>(messageHistory);
|
||||
} finally {
|
||||
lock.unlock();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 清空历史消息
|
||||
*/
|
||||
public void clearHistory() {
|
||||
lock.lock();
|
||||
try {
|
||||
messageHistory.clear();
|
||||
logger.info("会话 {} 历史消息已清空", sessionId);
|
||||
} finally {
|
||||
lock.unlock();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 获取当前消息对数
|
||||
*/
|
||||
public int getMessagePairCount() {
|
||||
lock.lock();
|
||||
try {
|
||||
return messageHistory.size() / 2;
|
||||
} finally {
|
||||
lock.unlock();
|
||||
}
|
||||
int chunkSize = 80;
|
||||
for (int i = 0; i < content.length(); i += chunkSize) {
|
||||
int end = Math.min(i + chunkSize, content.length());
|
||||
emitter.send(SseEmitter.event()
|
||||
.name("message")
|
||||
.data(SseMessage.content(content.substring(i, end)), MediaType.APPLICATION_JSON));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
package com.superbiz.agent.controller;
|
||||
|
||||
import com.superbiz.agent.dto.DiagnosisTraceResponse;
|
||||
import com.superbiz.agent.dto.Result;
|
||||
import com.superbiz.agent.service.DiagnosisTraceService;
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.PathVariable;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
|
||||
@RestController
|
||||
@RequestMapping("/api/diagnosis")
|
||||
@RequiredArgsConstructor
|
||||
public class DiagnosisTraceController {
|
||||
|
||||
private final DiagnosisTraceService diagnosisTraceService;
|
||||
|
||||
@GetMapping("/{sessionId}/trace")
|
||||
public ResponseEntity<Result<DiagnosisTraceResponse>> getTrace(@PathVariable String sessionId) {
|
||||
return ResponseEntity.ok(Result.success(diagnosisTraceService.getTrace(sessionId)));
|
||||
}
|
||||
}
|
||||
@@ -8,7 +8,9 @@ import lombok.NoArgsConstructor;
|
||||
import java.io.Serializable;
|
||||
import java.time.LocalDateTime;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* 会话上下文数据类
|
||||
@@ -53,6 +55,12 @@ public class SessionContext implements Serializable {
|
||||
@Builder.Default
|
||||
private List<ToolCall> toolCalls = new ArrayList<>();
|
||||
|
||||
/**
|
||||
* 聊天消息历史：[{"role":"user","content":"..."}, {"role":"assistant","content":"..."}]
|
||||
*/
|
||||
@Builder.Default
|
||||
private List<Map<String, String>> messageHistory = new ArrayList<>();
|
||||
|
||||
/**
|
||||
* 会话创建时间
|
||||
*/
|
||||
@@ -79,6 +87,64 @@ public class SessionContext implements Serializable {
|
||||
this.lastActiveAt = LocalDateTime.now();
|
||||
}
|
||||
|
||||
/**
|
||||
* 添加一对聊天消息，并按消息对数裁剪窗口。
|
||||
*/
|
||||
public void addChatMessagePair(String userQuestion, String assistantAnswer, int maxPairCount) {
|
||||
if (this.messageHistory == null) {
|
||||
this.messageHistory = new ArrayList<>();
|
||||
}
|
||||
|
||||
Map<String, String> userMessage = new HashMap<>();
|
||||
userMessage.put("role", "user");
|
||||
userMessage.put("content", userQuestion);
|
||||
this.messageHistory.add(userMessage);
|
||||
|
||||
Map<String, String> assistantMessage = new HashMap<>();
|
||||
assistantMessage.put("role", "assistant");
|
||||
assistantMessage.put("content", assistantAnswer);
|
||||
this.messageHistory.add(assistantMessage);
|
||||
|
||||
int maxMessages = Math.max(maxPairCount, 0) * 2;
|
||||
while (maxMessages > 0 && this.messageHistory.size() > maxMessages) {
|
||||
this.messageHistory.remove(0);
|
||||
if (!this.messageHistory.isEmpty()) {
|
||||
this.messageHistory.remove(0);
|
||||
}
|
||||
}
|
||||
this.lastActiveAt = LocalDateTime.now();
|
||||
}
|
||||
|
||||
/**
|
||||
* 获取聊天历史副本，避免调用方直接修改内部列表。
|
||||
*/
|
||||
public List<Map<String, String>> getMessageHistorySnapshot() {
|
||||
if (this.messageHistory == null || this.messageHistory.isEmpty()) {
|
||||
return new ArrayList<>();
|
||||
}
|
||||
List<Map<String, String>> snapshot = new ArrayList<>();
|
||||
for (Map<String, String> message : this.messageHistory) {
|
||||
snapshot.add(new HashMap<>(message));
|
||||
}
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
/**
|
||||
* 清空聊天历史。
|
||||
*/
|
||||
public void clearMessageHistory() {
|
||||
if (this.messageHistory == null) {
|
||||
this.messageHistory = new ArrayList<>();
|
||||
} else {
|
||||
this.messageHistory.clear();
|
||||
}
|
||||
this.lastActiveAt = LocalDateTime.now();
|
||||
}
|
||||
|
||||
public int getMessagePairCount() {
|
||||
return this.messageHistory == null ? 0 : this.messageHistory.size() / 2;
|
||||
}
|
||||
|
||||
/**
|
||||
* 更新最后活跃时间
|
||||
*/
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
package com.superbiz.agent.dto;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Builder;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
import java.time.LocalDateTime;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
@Data
|
||||
@Builder
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class DiagnosisTraceResponse {
|
||||
|
||||
private SessionTrace session;
|
||||
private List<AgentStepTrace> steps;
|
||||
private List<ToolInvocationTrace> toolInvocations;
|
||||
private TraceSummary summary;
|
||||
|
||||
@Data
|
||||
@Builder
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public static class SessionTrace {
|
||||
private Long id;
|
||||
private String sessionId;
|
||||
private String query;
|
||||
private String status;
|
||||
private String agentFlow;
|
||||
private Integer totalDurationMs;
|
||||
private Integer totalTokenCount;
|
||||
private Integer stepCount;
|
||||
private Integer toolCallCount;
|
||||
private String answer;
|
||||
private String selfEvaluationRaw;
|
||||
private Map<String, Object> selfEvaluation;
|
||||
private String feedback;
|
||||
private LocalDateTime createdAt;
|
||||
private LocalDateTime updatedAt;
|
||||
}
|
||||
|
||||
@Data
|
||||
@Builder
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public static class AgentStepTrace {
|
||||
private Long id;
|
||||
private String sessionId;
|
||||
private Integer stepIndex;
|
||||
private String agentName;
|
||||
private String modelInput;
|
||||
private String modelOutput;
|
||||
private String thought;
|
||||
private Boolean hasToolCall;
|
||||
private Integer durationMs;
|
||||
private Integer tokenCount;
|
||||
private LocalDateTime createdAt;
|
||||
}
|
||||
|
||||
@Data
|
||||
@Builder
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public static class ToolInvocationTrace {
|
||||
private Long id;
|
||||
private String sessionId;
|
||||
private Long stepId;
|
||||
private String toolName;
|
||||
private String inputParamsRaw;
|
||||
private Map<String, Object> inputParams;
|
||||
private String outputPreview;
|
||||
private Integer outputLength;
|
||||
private String retrievalLayer;
|
||||
private Integer l0MatchCount;
|
||||
private Integer l1MatchCount;
|
||||
private Boolean truncated;
|
||||
private String relevanceLevel;
|
||||
private String dedupReason;
|
||||
private String retrievalDetailsRaw;
|
||||
private Map<String, Object> retrievalDetails;
|
||||
private Integer durationMs;
|
||||
private Boolean success;
|
||||
private String errorMessage;
|
||||
private LocalDateTime createdAt;
|
||||
}
|
||||
|
||||
@Data
|
||||
@Builder
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public static class TraceSummary {
|
||||
private int persistedStepCount;
|
||||
private int returnedStepCount;
|
||||
private int persistedToolCallCount;
|
||||
private int returnedToolCallCount;
|
||||
private boolean hasVerifierEvaluation;
|
||||
private boolean hasFeedback;
|
||||
}
|
||||
}
|
||||
@@ -1,25 +1,27 @@
|
||||
package com.superbiz.agent.hook;
|
||||
|
||||
import com.alibaba.cloud.ai.graph.agent.hook.messages.MessagesModelHook;
|
||||
import com.alibaba.cloud.ai.graph.agent.hook.messages.AgentCommand;
|
||||
import com.alibaba.cloud.ai.graph.RunnableConfig;
|
||||
import com.alibaba.cloud.ai.graph.agent.hook.HookPosition;
|
||||
import com.alibaba.cloud.ai.graph.agent.hook.HookPositions;
|
||||
import com.alibaba.cloud.ai.graph.RunnableConfig;
|
||||
import com.alibaba.cloud.ai.graph.agent.hook.messages.AgentCommand;
|
||||
import com.alibaba.cloud.ai.graph.agent.hook.messages.MessagesModelHook;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.domain.entity.AgentStep;
|
||||
import com.superbiz.agent.repository.AgentStepRepository;
|
||||
import com.superbiz.agent.util.SessionContextHolder;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.springframework.ai.chat.messages.Message;
|
||||
import org.springframework.ai.chat.messages.AssistantMessage;
|
||||
import org.springframework.ai.chat.messages.UserMessage;
|
||||
import org.springframework.ai.chat.messages.Message;
|
||||
import org.springframework.ai.chat.messages.ToolResponseMessage;
|
||||
import org.springframework.ai.chat.messages.UserMessage;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
/**
|
||||
* Agent 日志 Hook
|
||||
* 记录 Agent 的思考过程、消息流转 + 持久化 agent_step 到 DB
|
||||
* Persists per-agent model input/output snapshots into agent_step.
|
||||
*/
|
||||
@Slf4j
|
||||
@HookPositions({HookPosition.BEFORE_MODEL, HookPosition.AFTER_MODEL})
|
||||
@@ -27,11 +29,9 @@ public class AgentLoggingHook extends MessagesModelHook {
|
||||
|
||||
private final AgentStepRepository agentStepRepository;
|
||||
private final String agentName;
|
||||
private final ObjectMapper objectMapper = new ObjectMapper();
|
||||
|
||||
/** 每个 session 的步数计数器：sessionId → stepIndex */
|
||||
private final ConcurrentHashMap<String, Integer> stepCounters = new ConcurrentHashMap<>();
|
||||
|
||||
/** beforeModel → afterModel 中间状态：sessionId_stepIndex → {stepId, startTime} */
|
||||
private final ConcurrentHashMap<String, Map<String, Object>> pendingSteps = new ConcurrentHashMap<>();
|
||||
|
||||
public AgentLoggingHook(AgentStepRepository agentStepRepository, String agentName) {
|
||||
@@ -46,60 +46,44 @@ public class AgentLoggingHook extends MessagesModelHook {
|
||||
|
||||
@Override
|
||||
public AgentCommand beforeModel(List<Message> previousMessages, RunnableConfig config) {
|
||||
// 优先从 config.metadata 取 sessionId（线程安全），兜底 ThreadLocal
|
||||
String sessionId = config.metadata("sessionId")
|
||||
.map(Object::toString)
|
||||
.orElseGet(SessionContextHolder::getSessionId);
|
||||
|
||||
boolean hasSession = (sessionId != null);
|
||||
String sessionId = resolveSessionId(config);
|
||||
boolean hasSession = sessionId != null;
|
||||
|
||||
int stepIndex = 0;
|
||||
if (hasSession) {
|
||||
stepIndex = stepCounters.merge(sessionId, 0, (old, one) -> old + 1);
|
||||
stepIndex = stepCounters.merge(sessionId, 0, (oldValue, ignored) -> oldValue + 1);
|
||||
}
|
||||
|
||||
log.info("========================================");
|
||||
log.info("*** [Agent 思考] 第 {} 轮思考开始", (hasSession ? stepCounters.get(sessionId) : 0) + 1);
|
||||
log.info("*** [Agent 思考] 当前消息数量: {}", previousMessages.size());
|
||||
log.info("*** [AgentTrace] agent={}, phase=before_model, stepIndex={}", agentName, stepIndex);
|
||||
log.info("*** [AgentTrace] messageCount={}", previousMessages.size());
|
||||
|
||||
// 打印最后几条消息
|
||||
int lastN = Math.min(3, previousMessages.size());
|
||||
if (lastN > 0) {
|
||||
log.info("*** [Agent 思考] 最近 {} 条消息:", lastN);
|
||||
log.info("*** [AgentTrace] recentMessages={}", lastN);
|
||||
List<Message> recentMessages = previousMessages.subList(previousMessages.size() - lastN, previousMessages.size());
|
||||
for (int i = 0; i < recentMessages.size(); i++) {
|
||||
Message msg = recentMessages.get(i);
|
||||
String role = getMessageRole(msg);
|
||||
log.info(" [{}] 角色: {}, 类型: {}", i + 1, role, msg.getClass().getSimpleName());
|
||||
log.info(" [{}] role={}, type={}", i + 1, getMessageRole(msg), msg.getClass().getSimpleName());
|
||||
}
|
||||
}
|
||||
|
||||
log.info("*** [Agent 思考] 准备调用模型...");
|
||||
log.info("========================================");
|
||||
|
||||
// 持久化 agent_step（beforeModel：先创建，先记 model_input 摘要）
|
||||
if (sessionId != null) {
|
||||
try {
|
||||
String modelInputSummary = buildModelInputSummary(previousMessages);
|
||||
|
||||
AgentStep step = AgentStep.builder()
|
||||
.sessionId(sessionId)
|
||||
.stepIndex(stepIndex)
|
||||
.agentName(agentName)
|
||||
.modelInput(modelInputSummary)
|
||||
.modelInput(buildModelInputSummary(previousMessages))
|
||||
.build();
|
||||
AgentStep saved = agentStepRepository.save(step);
|
||||
|
||||
// 记录中间状态供 afterModel 使用
|
||||
pendingSteps.put(sessionId + "_" + stepIndex, Map.of(
|
||||
"stepId", saved.getId(),
|
||||
"startTime", System.currentTimeMillis()
|
||||
));
|
||||
|
||||
log.debug("agent_step 已创建: sessionId={}, stepIndex={}, id={}", sessionId, stepIndex, saved.getId());
|
||||
} catch (Exception e) {
|
||||
log.error("保存 agent_step 失败", e);
|
||||
// 不中断 Agent 执行
|
||||
log.error("Failed to persist agent_step before model", e);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -108,58 +92,38 @@ public class AgentLoggingHook extends MessagesModelHook {
|
||||
|
||||
@Override
|
||||
public AgentCommand afterModel(List<Message> previousMessages, RunnableConfig config) {
|
||||
String sessionId = SessionContextHolder.getSessionId();
|
||||
boolean hasSession = (sessionId != null);
|
||||
String sessionId = resolveSessionId(config);
|
||||
int stepIndex = sessionId == null ? 0 : stepCounters.getOrDefault(sessionId, 0);
|
||||
|
||||
log.info("========================================");
|
||||
log.info("*** [Agent 思考] 第 {} 轮思考完成", (hasSession ? stepCounters.getOrDefault(sessionId, 0) : 0));
|
||||
|
||||
// 查找最后一条 AssistantMessage（模型的回复）
|
||||
AssistantMessage lastAssistant = null;
|
||||
for (int i = previousMessages.size() - 1; i >= 0; i--) {
|
||||
if (previousMessages.get(i) instanceof AssistantMessage) {
|
||||
lastAssistant = (AssistantMessage) previousMessages.get(i);
|
||||
break;
|
||||
}
|
||||
}
|
||||
log.info("*** [AgentTrace] agent={}, phase=after_model, stepIndex={}", agentName, stepIndex);
|
||||
|
||||
AssistantMessage lastAssistant = findLastAssistant(previousMessages);
|
||||
boolean hasToolCall = false;
|
||||
|
||||
if (lastAssistant != null) {
|
||||
// 打印模型返回的文本内容
|
||||
String textContent = extractTextContent(lastAssistant);
|
||||
if (textContent != null && !textContent.isEmpty()) {
|
||||
log.info("*** [Agent 思考] 模型返回文本: {}",
|
||||
textContent.length() > 500
|
||||
? textContent.substring(0, 500) + "... (已截断，总长度: " + textContent.length() + ")"
|
||||
: textContent);
|
||||
log.info("*** [AgentTrace] text={}",
|
||||
textContent.length() > 500
|
||||
? textContent.substring(0, 500) + "... (len=" + textContent.length() + ")"
|
||||
: textContent);
|
||||
}
|
||||
|
||||
// 检查是否有工具调用
|
||||
if (lastAssistant.getToolCalls() != null && !lastAssistant.getToolCalls().isEmpty()) {
|
||||
hasToolCall = true;
|
||||
log.info("*** [Agent 思考] 模型决定调用 {} 个工具:",
|
||||
lastAssistant.getToolCalls().size());
|
||||
lastAssistant.getToolCalls().forEach(toolCall -> {
|
||||
log.info(" - 工具: {}, 参数: {}",
|
||||
toolCall.name(),
|
||||
toolCall.arguments());
|
||||
});
|
||||
log.info("*** [Agent 思考] 等待工具执行结果...");
|
||||
log.info("*** [AgentTrace] toolCalls={}", lastAssistant.getToolCalls().size());
|
||||
lastAssistant.getToolCalls().forEach(toolCall ->
|
||||
log.info(" - tool={}, arguments={}", toolCall.name(), toolCall.arguments()));
|
||||
} else {
|
||||
log.info("*** [Agent 思考] 模型决定不调用工具");
|
||||
log.info("*** [Agent 思考] 这是最终答案，准备返回给用户");
|
||||
log.info("*** [AgentTrace] no tool call");
|
||||
}
|
||||
}
|
||||
|
||||
log.info("========================================");
|
||||
|
||||
// 更新 agent_step（afterModel：补全 model_output、耗时等）
|
||||
if (sessionId != null) {
|
||||
int stepIndex = stepCounters.getOrDefault(sessionId, 0);
|
||||
String stepKey = sessionId + "_" + stepIndex;
|
||||
Map<String, Object> pending = pendingSteps.remove(stepKey);
|
||||
|
||||
if (pending != null) {
|
||||
try {
|
||||
Long stepId = (Long) pending.get("stepId");
|
||||
@@ -168,20 +132,12 @@ public class AgentLoggingHook extends MessagesModelHook {
|
||||
|
||||
AgentStep step = agentStepRepository.findById(stepId).orElse(null);
|
||||
if (step != null) {
|
||||
String thought = extractTextContent(lastAssistant);
|
||||
if (thought != null && thought.length() > 2000) {
|
||||
thought = thought.substring(0, 2000);
|
||||
}
|
||||
|
||||
step.setThought(thought);
|
||||
step.setThought(buildStoredThought(lastAssistant));
|
||||
step.setHasToolCall(hasToolCall);
|
||||
step.setDurationMs(durationMs);
|
||||
|
||||
if (lastAssistant != null) {
|
||||
String outputSummary = buildModelOutputSummary(lastAssistant);
|
||||
step.setModelOutput(outputSummary);
|
||||
|
||||
// 读取实际 token 用量（由 TokenTrackingChatModel 写入）
|
||||
step.setModelOutput(buildModelOutputSummary(lastAssistant));
|
||||
Integer tokenCount = TokenUsageHolder.get();
|
||||
if (tokenCount != null) {
|
||||
step.setTokenCount(tokenCount);
|
||||
@@ -189,24 +145,32 @@ public class AgentLoggingHook extends MessagesModelHook {
|
||||
}
|
||||
|
||||
agentStepRepository.save(step);
|
||||
log.debug("agent_step 已更新: sessionId={}, stepIndex={}, duration={}ms",
|
||||
sessionId, stepIndex, durationMs);
|
||||
}
|
||||
} catch (Exception e) {
|
||||
log.error("更新 agent_step 失败", e);
|
||||
log.error("Failed to update agent_step after model", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 清理 token 上下文
|
||||
TokenUsageHolder.clear();
|
||||
|
||||
return new AgentCommand(previousMessages);
|
||||
}
|
||||
|
||||
/**
|
||||
* 构建模型输入摘要（前 N 条消息的 role + 截断内容）
|
||||
*/
|
||||
private String resolveSessionId(RunnableConfig config) {
|
||||
return config.metadata("sessionId")
|
||||
.map(Object::toString)
|
||||
.orElseGet(SessionContextHolder::getSessionId);
|
||||
}
|
||||
|
||||
private AssistantMessage findLastAssistant(List<Message> previousMessages) {
|
||||
for (int i = previousMessages.size() - 1; i >= 0; i--) {
|
||||
if (previousMessages.get(i) instanceof AssistantMessage assistantMessage) {
|
||||
return assistantMessage;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
private String buildModelInputSummary(List<Message> messages) {
|
||||
StringBuilder sb = new StringBuilder();
|
||||
int maxMessages = Math.min(messages.size(), 5);
|
||||
@@ -226,25 +190,59 @@ public class AgentLoggingHook extends MessagesModelHook {
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* 构建模型输出摘要
|
||||
*/
|
||||
private String buildStoredThought(AssistantMessage message) {
|
||||
String text = extractTextContent(message);
|
||||
if (text == null || text.isBlank()) {
|
||||
return text;
|
||||
}
|
||||
if (!"verifier".equals(agentName)) {
|
||||
return truncate(text, 2000);
|
||||
}
|
||||
return summarizeVerifierThought(text);
|
||||
}
|
||||
|
||||
private String summarizeVerifierThought(String verifierOutput) {
|
||||
try {
|
||||
JsonNode root = objectMapper.readTree(verifierOutput);
|
||||
int factCount = root.path("facts_checked").isArray() ? root.path("facts_checked").size() : 0;
|
||||
int tracedFactCount = 0;
|
||||
if (root.path("facts_checked").isArray()) {
|
||||
for (JsonNode factNode : root.path("facts_checked")) {
|
||||
if (factNode.path("evidence_refs").isArray() && factNode.path("evidence_refs").size() > 0) {
|
||||
tracedFactCount++;
|
||||
}
|
||||
}
|
||||
}
|
||||
return "verdict=%s, score=%s, critical_fact_count=%s, facts_checked=%d, traced_facts=%d".formatted(
|
||||
root.path("verdict").asText("UNKNOWN"),
|
||||
root.path("groundedness_score").asText("0.0"),
|
||||
root.path("critical_fact_count").asText("0"),
|
||||
factCount,
|
||||
tracedFactCount
|
||||
);
|
||||
} catch (Exception e) {
|
||||
return truncate(verifierOutput, 300);
|
||||
}
|
||||
}
|
||||
|
||||
private String buildModelOutputSummary(AssistantMessage message) {
|
||||
String text = extractTextContent(message);
|
||||
if (text == null) {
|
||||
text = "";
|
||||
}
|
||||
if (text.length() > 500) {
|
||||
text = text.substring(0, 500) + "...";
|
||||
}
|
||||
int maxTextLength = "verifier".equals(agentName) ? 4000 : 500;
|
||||
text = truncate(text, maxTextLength);
|
||||
|
||||
StringBuilder sb = new StringBuilder();
|
||||
sb.append("{\"text\":\"").append(escapeJson(text)).append("\"");
|
||||
if (message.getToolCalls() != null && !message.getToolCalls().isEmpty()) {
|
||||
sb.append(",\"toolCalls\":[");
|
||||
for (int i = 0; i < message.getToolCalls().size(); i++) {
|
||||
if (i > 0) sb.append(",");
|
||||
if (i > 0) {
|
||||
sb.append(",");
|
||||
}
|
||||
sb.append("{\"name\":\"").append(escapeJson(message.getToolCalls().get(i).name()))
|
||||
.append("\",\"arguments\":").append(message.getToolCalls().get(i).arguments()).append("}");
|
||||
.append("\",\"arguments\":").append(message.getToolCalls().get(i).arguments()).append("}");
|
||||
}
|
||||
sb.append("]");
|
||||
}
|
||||
@@ -253,7 +251,9 @@ public class AgentLoggingHook extends MessagesModelHook {
|
||||
}
|
||||
|
||||
private String escapeJson(String s) {
|
||||
if (s == null) return "";
|
||||
if (s == null) {
|
||||
return "";
|
||||
}
|
||||
return s.replace("\\", "\\\\")
|
||||
.replace("\"", "\\\"")
|
||||
.replace("\n", "\\n")
|
||||
@@ -261,96 +261,70 @@ public class AgentLoggingHook extends MessagesModelHook {
|
||||
.replace("\t", "\\t");
|
||||
}
|
||||
|
||||
/**
|
||||
* 提取 AssistantMessage 的文本内容
|
||||
*/
|
||||
private String truncate(String text, int maxLength) {
|
||||
if (text == null || text.length() <= maxLength) {
|
||||
return text;
|
||||
}
|
||||
return text.substring(0, maxLength) + "...";
|
||||
}
|
||||
|
||||
private String extractTextContent(AssistantMessage message) {
|
||||
if (message == null) return null;
|
||||
if (message == null) {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
// 方法 1: 反射获取 text 字段
|
||||
try {
|
||||
java.lang.reflect.Field textField = message.getClass().getDeclaredField("text");
|
||||
textField.setAccessible(true);
|
||||
Object value = textField.get(message);
|
||||
if (value != null) {
|
||||
log.debug("通过 text 字段提取成功");
|
||||
return value.toString();
|
||||
return message.getText();
|
||||
} catch (Exception ignore) {
|
||||
// Fallback below.
|
||||
}
|
||||
|
||||
for (String fieldName : List.of("text", "content")) {
|
||||
try {
|
||||
java.lang.reflect.Field field = message.getClass().getDeclaredField(fieldName);
|
||||
field.setAccessible(true);
|
||||
Object value = field.get(message);
|
||||
if (value != null) {
|
||||
return value.toString();
|
||||
}
|
||||
} catch (NoSuchFieldException ignore) {
|
||||
// continue
|
||||
}
|
||||
} catch (NoSuchFieldException e) {
|
||||
// 尝试下一种方法
|
||||
}
|
||||
|
||||
// 方法 2: 反射获取 content 字段
|
||||
try {
|
||||
java.lang.reflect.Field contentField = message.getClass().getDeclaredField("content");
|
||||
contentField.setAccessible(true);
|
||||
Object value = contentField.get(message);
|
||||
if (value != null) {
|
||||
log.debug("通过 content 字段提取成功");
|
||||
return value.toString();
|
||||
for (String methodName : List.of("getText", "getContent")) {
|
||||
try {
|
||||
java.lang.reflect.Method method = message.getClass().getMethod(methodName);
|
||||
Object value = method.invoke(message);
|
||||
if (value != null) {
|
||||
return value.toString();
|
||||
}
|
||||
} catch (NoSuchMethodException ignore) {
|
||||
// continue
|
||||
}
|
||||
} catch (NoSuchFieldException e) {
|
||||
// 尝试下一种方法
|
||||
}
|
||||
|
||||
// 方法 3: 调用 getText() 方法
|
||||
try {
|
||||
java.lang.reflect.Method getTextMethod = message.getClass().getMethod("getText");
|
||||
Object value = getTextMethod.invoke(message);
|
||||
if (value != null) {
|
||||
log.debug("通过 getText() 方法提取成功");
|
||||
return value.toString();
|
||||
}
|
||||
} catch (NoSuchMethodException e) {
|
||||
// 尝试下一种方法
|
||||
String fallback = message.toString();
|
||||
if (fallback != null && !fallback.startsWith("AssistantMessage@")) {
|
||||
return fallback;
|
||||
}
|
||||
|
||||
// 方法 4: 调用 getContent() 方法
|
||||
try {
|
||||
java.lang.reflect.Method getContentMethod = message.getClass().getMethod("getContent");
|
||||
Object value = getContentMethod.invoke(message);
|
||||
if (value != null) {
|
||||
log.debug("通过 getContent() 方法提取成功");
|
||||
return value.toString();
|
||||
}
|
||||
} catch (NoSuchMethodException e) {
|
||||
// 方法不存在
|
||||
}
|
||||
|
||||
// 方法 5: 打印类结构信息
|
||||
log.warn("无法提取 AssistantMessage 文本内容，打印类信息:");
|
||||
log.warn("类名: {}", message.getClass().getName());
|
||||
log.warn("字段列表:");
|
||||
for (java.lang.reflect.Field field : message.getClass().getDeclaredFields()) {
|
||||
log.warn(" - {}: {}", field.getName(), field.getType().getSimpleName());
|
||||
}
|
||||
|
||||
// 方法 6: toString() 兜底
|
||||
String toString = message.toString();
|
||||
if (toString != null && !toString.startsWith("AssistantMessage@")) {
|
||||
log.debug("通过 toString() 提取");
|
||||
return toString;
|
||||
}
|
||||
|
||||
return null;
|
||||
} catch (Exception e) {
|
||||
log.error("提取 AssistantMessage 文本内容时出错", e);
|
||||
log.error("Failed to extract AssistantMessage text", e);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 获取消息角色
|
||||
*/
|
||||
private String getMessageRole(Message message) {
|
||||
if (message instanceof UserMessage) {
|
||||
return "User（用户）";
|
||||
} else if (message instanceof AssistantMessage) {
|
||||
return "Assistant（模型）";
|
||||
} else if (message instanceof ToolResponseMessage) {
|
||||
return "Tool（工具返回）";
|
||||
} else {
|
||||
return message.getClass().getSimpleName();
|
||||
return "user";
|
||||
}
|
||||
if (message instanceof AssistantMessage) {
|
||||
return "assistant";
|
||||
}
|
||||
if (message instanceof ToolResponseMessage) {
|
||||
return "tool";
|
||||
}
|
||||
return message.getClass().getSimpleName();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
package com.superbiz.agent.hook;
|
||||
|
||||
import com.alibaba.cloud.ai.graph.RunnableConfig;
|
||||
import com.alibaba.cloud.ai.graph.agent.hook.HookPosition;
|
||||
import com.alibaba.cloud.ai.graph.agent.hook.HookPositions;
|
||||
import com.alibaba.cloud.ai.graph.agent.hook.messages.AgentCommand;
|
||||
import com.alibaba.cloud.ai.graph.agent.hook.messages.MessagesModelHook;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.service.ToolTraceSummaryService;
|
||||
import com.superbiz.agent.util.SessionContextHolder;
|
||||
import com.superbiz.agent.util.VerifierContextHolder;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.springframework.ai.chat.messages.AssistantMessage;
|
||||
import org.springframework.ai.chat.messages.Message;
|
||||
import org.springframework.ai.chat.messages.UserMessage;
|
||||
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* Replaces verifier history with an explicit structured payload.
|
||||
*/
|
||||
@Slf4j
|
||||
@HookPositions(HookPosition.BEFORE_MODEL)
|
||||
public class VerifierInputHook extends MessagesModelHook {
|
||||
|
||||
private final ToolTraceSummaryService toolTraceSummaryService;
|
||||
private final ObjectMapper objectMapper = new ObjectMapper();
|
||||
|
||||
public VerifierInputHook(ToolTraceSummaryService toolTraceSummaryService) {
|
||||
this.toolTraceSummaryService = toolTraceSummaryService;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getName() {
|
||||
return "verifier_input_hook";
|
||||
}
|
||||
|
||||
@Override
|
||||
public AgentCommand beforeModel(List<Message> previousMessages, RunnableConfig config) {
|
||||
try {
|
||||
String sessionId = config.metadata("sessionId")
|
||||
.map(Object::toString)
|
||||
.orElseGet(SessionContextHolder::getSessionId);
|
||||
String executorFinalAnswer = VerifierContextHolder.getExecutorFinalAnswer();
|
||||
if (executorFinalAnswer == null || executorFinalAnswer.isBlank()) {
|
||||
executorFinalAnswer = extractLastAssistantText(previousMessages);
|
||||
}
|
||||
|
||||
List<Map<String, Object>> toolTraceSummary =
|
||||
toolTraceSummaryService.buildVerifierTraceSummary(sessionId, executorFinalAnswer);
|
||||
VerifierContextHolder.setToolTraceSummary(toolTraceSummary);
|
||||
|
||||
Map<String, Object> verifierInput = new LinkedHashMap<>();
|
||||
verifierInput.put("original_query", VerifierContextHolder.getOriginalQuery());
|
||||
verifierInput.put("executor_final_answer", executorFinalAnswer);
|
||||
verifierInput.put("tool_trace_summary", toolTraceSummary);
|
||||
verifierInput.put("retry_context", VerifierContextHolder.getRetryContext());
|
||||
|
||||
String payload = objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(verifierInput);
|
||||
return new AgentCommand(List.of(new UserMessage(payload)));
|
||||
} catch (Exception e) {
|
||||
log.error("Failed to build verifier input, fallback to original messages", e);
|
||||
return new AgentCommand(previousMessages);
|
||||
}
|
||||
}
|
||||
|
||||
private String extractLastAssistantText(List<Message> previousMessages) {
|
||||
for (int i = previousMessages.size() - 1; i >= 0; i--) {
|
||||
if (previousMessages.get(i) instanceof AssistantMessage assistantMessage) {
|
||||
String text = extractTextContent(assistantMessage);
|
||||
if (text != null && !text.isBlank()) {
|
||||
return text;
|
||||
}
|
||||
}
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
private String extractTextContent(AssistantMessage message) {
|
||||
try {
|
||||
try {
|
||||
return message.getText();
|
||||
} catch (Exception ignore) {
|
||||
// Fallback for older implementations.
|
||||
}
|
||||
|
||||
for (String methodName : List.of("getText", "getContent")) {
|
||||
try {
|
||||
var method = message.getClass().getMethod(methodName);
|
||||
Object value = method.invoke(message);
|
||||
if (value != null) {
|
||||
return value.toString();
|
||||
}
|
||||
} catch (NoSuchMethodException ignore) {
|
||||
// continue
|
||||
}
|
||||
}
|
||||
} catch (Exception e) {
|
||||
log.debug("Failed to extract verifier assistant text", e);
|
||||
}
|
||||
return message.toString();
|
||||
}
|
||||
}
|
||||
@@ -17,6 +17,11 @@ public interface ToolInvocationRepository extends JpaRepository<ToolInvocation,
|
||||
*/
|
||||
List<ToolInvocation> findBySessionId(String sessionId);
|
||||
|
||||
/**
|
||||
* 根据会话ID按创建顺序查询所有工具调用
|
||||
*/
|
||||
List<ToolInvocation> findBySessionIdOrderByIdAsc(String sessionId);
|
||||
|
||||
/**
|
||||
* 根据工具名查询所有调用
|
||||
*/
|
||||
|
||||
@@ -3,8 +3,10 @@ package com.superbiz.agent.service;
|
||||
import com.alibaba.cloud.ai.graph.OverAllState;
|
||||
import com.alibaba.cloud.ai.graph.RunnableConfig;
|
||||
import com.alibaba.cloud.ai.graph.agent.ReactAgent;
|
||||
import com.alibaba.cloud.ai.graph.agent.flow.agent.SupervisorAgent;
|
||||
import com.alibaba.cloud.ai.graph.agent.flow.agent.SequentialAgent;
|
||||
import com.alibaba.cloud.ai.graph.exception.GraphRunnerException;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.agent.tool.DateTimeTools;
|
||||
import com.superbiz.agent.agent.tool.InternalDocsTools;
|
||||
import com.superbiz.agent.agent.tool.QueryLogsTools;
|
||||
@@ -13,13 +15,14 @@ import com.superbiz.agent.domain.entity.DiagnosisSession;
|
||||
import com.superbiz.agent.hook.AgentLoggingHook;
|
||||
import com.superbiz.agent.hook.TokenTrackingChatModel;
|
||||
import com.superbiz.agent.hook.TokenUsageHolder;
|
||||
import com.superbiz.agent.hook.VerifierInputHook;
|
||||
import com.superbiz.agent.repository.AgentStepRepository;
|
||||
import com.superbiz.agent.repository.DiagnosisSessionRepository;
|
||||
import com.superbiz.agent.tool.LookupKnowledgeTool;
|
||||
import com.superbiz.agent.tool.RetrievedDocTracker;
|
||||
import com.superbiz.agent.util.QuestionComplexity;
|
||||
import com.superbiz.agent.util.SessionContextHolder;
|
||||
import com.superbiz.agent.service.KnowledgeDomainService;
|
||||
import com.superbiz.agent.util.VerifierContextHolder;
|
||||
|
||||
import jakarta.annotation.PostConstruct;
|
||||
import org.slf4j.Logger;
|
||||
@@ -29,11 +32,14 @@ import org.springframework.ai.chat.model.ChatModel;
|
||||
import org.springframework.ai.tool.ToolCallback;
|
||||
import org.springframework.ai.tool.ToolCallbackProvider;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.core.io.ClassPathResource;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
@@ -47,6 +53,8 @@ import java.util.UUID;
|
||||
public class ChatService {
|
||||
|
||||
private static final Logger logger = LoggerFactory.getLogger(ChatService.class);
|
||||
private static final String LOW_CONFID_DISCLAIMER = "以下结论基于当前已获取证据，仍存在部分证据缺口，请谨慎参考。";
|
||||
private static final String DEGRADED_PREFIX = "当前无法基于已获取证据生成可靠结论，建议人工介入。";
|
||||
|
||||
/** 封装 answer + 后端生成的 sessionId，用于 feedback 关联 */
|
||||
public record ChatResult(String answer, String sessionId) {}
|
||||
@@ -87,9 +95,23 @@ public class ChatService {
|
||||
@Autowired
|
||||
private KnowledgeDomainService knowledgeDomainService;
|
||||
|
||||
@Autowired
|
||||
private ToolTraceSummaryService toolTraceSummaryService;
|
||||
|
||||
@Autowired
|
||||
private SelfEvaluationMergeService selfEvaluationMergeService;
|
||||
|
||||
@Value("${verifier.low-confidence-threshold:0.5}")
|
||||
private double verifierLowConfidenceThreshold;
|
||||
|
||||
@Value("${chat.complex.retry-on-low-confidence:false}")
|
||||
private boolean retryOnLowConfidence;
|
||||
|
||||
/** 多 Agent Chat 的 Prompt */
|
||||
private String chatPlannerPrompt;
|
||||
private String chatExecutorPrompt;
|
||||
private String chatVerifierPrompt;
|
||||
private final ObjectMapper objectMapper = new ObjectMapper();
|
||||
|
||||
@PostConstruct
|
||||
public void init() {
|
||||
@@ -101,6 +123,9 @@ public class ChatService {
|
||||
chatExecutorPrompt = new String(
|
||||
new ClassPathResource("prompts/chat-executor-prompt.md").getInputStream().readAllBytes(),
|
||||
StandardCharsets.UTF_8);
|
||||
chatVerifierPrompt = new String(
|
||||
new ClassPathResource("prompts/chat-verifier-prompt.md").getInputStream().readAllBytes(),
|
||||
StandardCharsets.UTF_8);
|
||||
logger.info("Chat 多 Agent Prompts 加载成功");
|
||||
} catch (IOException e) {
|
||||
logger.error("加载 Chat Prompt 文件失败", e);
|
||||
@@ -189,16 +214,19 @@ public class ChatService {
|
||||
|
||||
/**
|
||||
* 动态构建方法工具数组
|
||||
* 根据 cls.mock-enabled 决定是否包含 QueryLogsTools
|
||||
* 根据已注入的 Bean 暴露本地工具，避免 mock/真实模式下漏注入。
|
||||
*/
|
||||
public Object[] buildMethodToolsArray() {
|
||||
List<Object> methodTools = new ArrayList<>();
|
||||
methodTools.add(dateTimeTools);
|
||||
methodTools.add(lookupKnowledgeTool);
|
||||
if (queryLogsTools != null) {
|
||||
// Mock 模式：包含 QueryLogsTools
|
||||
return new Object[]{dateTimeTools, lookupKnowledgeTool};
|
||||
} else {
|
||||
// 真实模式：不包含 QueryLogsTools（由 MCP 提供日志查询功能）
|
||||
return new Object[]{dateTimeTools, lookupKnowledgeTool, queryMetricsTools};
|
||||
methodTools.add(queryLogsTools);
|
||||
}
|
||||
if (queryMetricsTools != null) {
|
||||
methodTools.add(queryMetricsTools);
|
||||
}
|
||||
return methodTools.toArray();
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -250,19 +278,18 @@ public class ChatService {
|
||||
* @return ChatResult（answer + sessionId）
|
||||
*/
|
||||
public ChatResult executeChat(ReactAgent agent, String question) throws GraphRunnerException {
|
||||
return executeChat(agent, question, null);
|
||||
}
|
||||
|
||||
public ChatResult executeChat(ReactAgent agent, String question, String requestedSessionId) throws GraphRunnerException {
|
||||
logger.info("========================================");
|
||||
logger.info("📝 用户问题: {}", question);
|
||||
|
||||
String sessionId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String sessionId = resolveSessionId(requestedSessionId);
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
// 创建诊断会话
|
||||
DiagnosisSession session = DiagnosisSession.builder()
|
||||
.sessionId(sessionId)
|
||||
.query(question)
|
||||
.status("RUNNING")
|
||||
.agentFlow("CHAT")
|
||||
.build();
|
||||
// 创建或更新诊断会话
|
||||
DiagnosisSession session = startDiagnosisSession(sessionId, question);
|
||||
diagnosisSessionRepository.save(session);
|
||||
|
||||
// 设置 ThreadLocal 上下文（LookupKnowledgeTool 通过此获取 sessionId）
|
||||
@@ -313,62 +340,124 @@ public class ChatService {
|
||||
*/
|
||||
public ChatResult executeChatWithStrategy(ChatModel chatModel, ToolCallback[] toolCallbacks,
|
||||
String question, List<Map<String, String>> history) throws GraphRunnerException {
|
||||
return executeChatWithStrategy(chatModel, toolCallbacks, question, history, null);
|
||||
}
|
||||
|
||||
public ChatResult executeChatWithStrategy(ChatModel chatModel, ToolCallback[] toolCallbacks,
|
||||
String question, List<Map<String, String>> history,
|
||||
String requestedSessionId) throws GraphRunnerException {
|
||||
if (QuestionComplexity.isComplex(question)) {
|
||||
logger.info("📊 问题判定为复杂，使用多 Agent（Planner + Executor）执行");
|
||||
return executeChatComplex(chatModel, toolCallbacks, question, history);
|
||||
return executeChatComplex(chatModel, toolCallbacks, question, history, requestedSessionId);
|
||||
} else {
|
||||
logger.info("📊 问题判定为简单，使用单 Agent 执行");
|
||||
String systemPrompt = buildSystemPrompt(history);
|
||||
ReactAgent agent = createReactAgent(chatModel, systemPrompt);
|
||||
return executeChat(agent, question);
|
||||
return executeChat(agent, question, requestedSessionId);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 多 Agent 复杂对话执行（Planner + Executor + Supervisor）
|
||||
* 多 Agent 复杂对话执行（Planner -> Executor -> Verifier）
|
||||
*/
|
||||
public ChatResult executeChatComplex(ChatModel chatModel, ToolCallback[] toolCallbacks,
|
||||
String question, List<Map<String, String>> history) throws GraphRunnerException {
|
||||
String sessionId = UUID.randomUUID().toString().substring(0, 8);
|
||||
return executeChatComplex(chatModel, toolCallbacks, question, history, null);
|
||||
}
|
||||
|
||||
public ChatResult executeChatComplex(ChatModel chatModel, ToolCallback[] toolCallbacks,
|
||||
String question, List<Map<String, String>> history,
|
||||
String requestedSessionId) throws GraphRunnerException {
|
||||
String sessionId = resolveSessionId(requestedSessionId);
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
DiagnosisSession session = DiagnosisSession.builder()
|
||||
.sessionId(sessionId)
|
||||
.query(question)
|
||||
.status("RUNNING")
|
||||
.agentFlow("CHAT")
|
||||
.build();
|
||||
DiagnosisSession session = startDiagnosisSession(sessionId, question);
|
||||
diagnosisSessionRepository.save(session);
|
||||
|
||||
SessionContextHolder.setSessionId(sessionId);
|
||||
VerifierContextHolder.setOriginalQuery(question);
|
||||
VerifierContextHolder.setRetryContext(null);
|
||||
VerifierContextHolder.setExecutorFinalAnswer(null);
|
||||
|
||||
try {
|
||||
ReactAgent planner = buildChatPlannerAgent(chatModel, toolCallbacks, history);
|
||||
ReactAgent executor = buildChatExecutorAgent(chatModel, toolCallbacks, history);
|
||||
|
||||
SupervisorAgent supervisor = SupervisorAgent.builder()
|
||||
.name("chat_supervisor")
|
||||
.description("负责调度 Planner 与 Executor 的多 Agent 控制器")
|
||||
.model(chatModel)
|
||||
.systemPrompt("你是一个智能任务调度器。分析用户问题，调用 Planner 拆解步骤，调用 Executor 执行各步骤。")
|
||||
.subAgents(List.of(planner, executor))
|
||||
VerifierDecision finalDecision = null;
|
||||
String retryContext = null;
|
||||
String answer = null;
|
||||
RunnableConfig config = RunnableConfig.builder()
|
||||
.addMetadata("sessionId", sessionId)
|
||||
.build();
|
||||
|
||||
Optional<OverAllState> stateOptional = supervisor.invoke(question);
|
||||
long duration = System.currentTimeMillis() - startTime;
|
||||
for (int round = 1; round <= 2; round++) {
|
||||
VerifierContextHolder.setRetryContext(retryContext);
|
||||
VerifierContextHolder.setToolTraceSummary(null);
|
||||
|
||||
String answer = null;
|
||||
if (stateOptional.isPresent()) {
|
||||
// 从 state 中提取 Executor 的最终输出
|
||||
OverAllState state = stateOptional.get();
|
||||
Optional<AssistantMessage> executorOutput = state.value("executor_feedback")
|
||||
.filter(AssistantMessage.class::isInstance)
|
||||
.map(AssistantMessage.class::cast);
|
||||
if (executorOutput.isPresent()) {
|
||||
answer = executorOutput.get().getText();
|
||||
ReactAgent planner = buildChatPlannerAgent(chatModel, history, retryContext);
|
||||
ReactAgent executor = buildChatExecutorAgent(chatModel, toolCallbacks, history, retryContext);
|
||||
ReactAgent verifier = buildChatVerifierAgent(chatModel);
|
||||
|
||||
SequentialAgent workflow = SequentialAgent.builder()
|
||||
.name("chat_workflow")
|
||||
.description("按固定顺序执行 Planner、Executor、Verifier 的多 Agent 工作流")
|
||||
.subAgents(List.of(planner, executor, verifier))
|
||||
.build();
|
||||
|
||||
String workflowInput = buildWorkflowInput(question, retryContext);
|
||||
Optional<OverAllState> stateOptional = workflow.invoke(workflowInput, config);
|
||||
if (stateOptional.isEmpty()) {
|
||||
finalDecision = buildVerifierFallbackDecision(round, "workflow 未返回有效状态");
|
||||
answer = buildLowConfidenceOutput(answer, finalDecision);
|
||||
persistVerifierEvaluation(session, finalDecision, round);
|
||||
break;
|
||||
}
|
||||
|
||||
String plannerPlan = extractStateText(stateOptional, "planner_plan");
|
||||
answer = extractStateText(stateOptional, "executor_feedback");
|
||||
VerifierContextHolder.setExecutorFinalAnswer(answer);
|
||||
String verifierOutput = extractStateText(stateOptional, "verifier_output");
|
||||
if ((verifierOutput == null || verifierOutput.isBlank()) && answer != null && !answer.isBlank()) {
|
||||
verifierOutput = invokeVerifierFallback(verifier, question, round, config);
|
||||
}
|
||||
finalDecision = parseVerifierDecision(verifierOutput, round);
|
||||
logger.debug("Sequential workflow round {} finished: plannerPlanLength={}, answerLength={}, verifierOutputLength={}",
|
||||
round,
|
||||
plannerPlan != null ? plannerPlan.length() : 0,
|
||||
answer != null ? answer.length() : 0,
|
||||
verifierOutput != null ? verifierOutput.length() : 0);
|
||||
|
||||
if (finalDecision == null) {
|
||||
finalDecision = buildVerifierFallbackDecision(round, "verifier_output 缺失或无法解析");
|
||||
answer = buildLowConfidenceOutput(answer, finalDecision);
|
||||
persistVerifierEvaluation(session, finalDecision, round);
|
||||
break;
|
||||
}
|
||||
|
||||
if ("PASS".equals(finalDecision.verdict())) {
|
||||
answer = answer == null || answer.isBlank() ? "抱歉，多 Agent 分析未能生成有效结论。" : answer;
|
||||
persistVerifierEvaluation(session, finalDecision, round);
|
||||
break;
|
||||
}
|
||||
|
||||
if ("REJECT".equals(finalDecision.verdict())) {
|
||||
answer = buildDegradedOutput(finalDecision);
|
||||
persistVerifierEvaluation(session, finalDecision, round);
|
||||
break;
|
||||
}
|
||||
|
||||
boolean shouldRetry = retryOnLowConfidence
|
||||
&& finalDecision.groundednessScore() < verifierLowConfidenceThreshold
|
||||
&& round < 2;
|
||||
if (!shouldRetry) {
|
||||
answer = buildLowConfidenceOutput(answer, finalDecision);
|
||||
persistVerifierEvaluation(session, finalDecision, round);
|
||||
break;
|
||||
}
|
||||
|
||||
retryContext = buildRetryContext(finalDecision);
|
||||
persistVerifierEvaluation(session, finalDecision, round);
|
||||
}
|
||||
|
||||
long duration = System.currentTimeMillis() - startTime;
|
||||
|
||||
if (answer == null || answer.isBlank()) {
|
||||
answer = "抱歉，多 Agent 分析未能生成有效结论。";
|
||||
}
|
||||
@@ -394,11 +483,12 @@ public class ChatService {
|
||||
} finally {
|
||||
retrievedDocTracker.clearSession(sessionId);
|
||||
SessionContextHolder.clear();
|
||||
VerifierContextHolder.clear();
|
||||
}
|
||||
}
|
||||
|
||||
private ReactAgent buildChatPlannerAgent(ChatModel chatModel, ToolCallback[] toolCallbacks,
|
||||
List<Map<String, String>> history) {
|
||||
private ReactAgent buildChatPlannerAgent(ChatModel chatModel, List<Map<String, String>> history,
|
||||
String retryContext) {
|
||||
StringBuilder prompt = new StringBuilder(chatPlannerPrompt);
|
||||
|
||||
// 注入 knowledge map
|
||||
@@ -414,6 +504,9 @@ public class ChatService {
|
||||
}
|
||||
prompt.append("--- 对话历史结束 ---\n");
|
||||
}
|
||||
if (retryContext != null && !retryContext.isBlank()) {
|
||||
prompt.append("\n\n--- 本轮补证据约束 ---\n").append(retryContext).append("\n");
|
||||
}
|
||||
return ReactAgent.builder()
|
||||
.name("chat_planner")
|
||||
.description("负责拆解问题、规划步骤")
|
||||
@@ -424,8 +517,20 @@ public class ChatService {
|
||||
.build();
|
||||
}
|
||||
|
||||
private ReactAgent buildChatVerifierAgent(ChatModel chatModel) {
|
||||
return ReactAgent.builder()
|
||||
.name("chat_verifier")
|
||||
.description("负责验证 Executor 答案的事实准确性")
|
||||
.model(chatModel)
|
||||
.systemPrompt(chatVerifierPrompt)
|
||||
.hooks(new AgentLoggingHook(agentStepRepository, "verifier"),
|
||||
new VerifierInputHook(toolTraceSummaryService))
|
||||
.outputKey("verifier_output")
|
||||
.build();
|
||||
}
|
||||
|
||||
private ReactAgent buildChatExecutorAgent(ChatModel chatModel, ToolCallback[] toolCallbacks,
|
||||
List<Map<String, String>> history) {
|
||||
List<Map<String, String>> history, String retryContext) {
|
||||
StringBuilder prompt = new StringBuilder(chatExecutorPrompt);
|
||||
if (!history.isEmpty()) {
|
||||
prompt.append("\n\n--- 对话历史 ---\n");
|
||||
@@ -434,6 +539,9 @@ public class ChatService {
|
||||
}
|
||||
prompt.append("--- 对话历史结束 ---\n");
|
||||
}
|
||||
if (retryContext != null && !retryContext.isBlank()) {
|
||||
prompt.append("\n\n--- 本轮补证据约束 ---\n").append(retryContext).append("\n");
|
||||
}
|
||||
return ReactAgent.builder()
|
||||
.name("chat_executor")
|
||||
.description("负责执行具体步骤并及时反馈")
|
||||
@@ -446,6 +554,290 @@ public class ChatService {
|
||||
.build();
|
||||
}
|
||||
|
||||
private String resolveSessionId(String requestedSessionId) {
|
||||
if (requestedSessionId != null && !requestedSessionId.isBlank()) {
|
||||
return requestedSessionId;
|
||||
}
|
||||
return UUID.randomUUID().toString().substring(0, 8);
|
||||
}
|
||||
|
||||
private DiagnosisSession startDiagnosisSession(String sessionId, String question) {
|
||||
DiagnosisSession session = diagnosisSessionRepository.findBySessionId(sessionId)
|
||||
.orElseGet(() -> DiagnosisSession.builder()
|
||||
.sessionId(sessionId)
|
||||
.agentFlow("CHAT")
|
||||
.build());
|
||||
session.setQuery(question);
|
||||
session.setStatus("RUNNING");
|
||||
session.setAgentFlow("CHAT");
|
||||
session.setAnswer(null);
|
||||
session.setTotalDurationMs(null);
|
||||
session.setTotalTokenCount(null);
|
||||
session.setStepCount(null);
|
||||
session.setToolCallCount(null);
|
||||
return session;
|
||||
}
|
||||
|
||||
private String buildWorkflowInput(String question, String retryContext) {
|
||||
StringBuilder input = new StringBuilder();
|
||||
input.append("请按固定工作流完成本轮 Planner -> Executor -> Verifier。\n\n");
|
||||
input.append("--- 用户问题 ---\n").append(question);
|
||||
if (retryContext != null && !retryContext.isBlank()) {
|
||||
input.append("\n\n--- retry_context ---\n").append(retryContext);
|
||||
}
|
||||
input.append("\n\nVerifier 完成后由外层代码读取 verifier_output 并决定最终用户输出。");
|
||||
return input.toString();
|
||||
}
|
||||
|
||||
private String invokeVerifierFallback(ReactAgent verifier, String question, int round, RunnableConfig config) {
|
||||
try {
|
||||
logger.warn("Sequential workflow round {} finished without verifier_output, invoking chat_verifier fallback", round);
|
||||
return verifier.call("请基于 executor_final_answer 和 tool_trace_summary 输出 verifier JSON。原始问题：" + question, config)
|
||||
.getText();
|
||||
} catch (Exception e) {
|
||||
logger.error("chat_verifier fallback 执行失败", e);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private VerifierDecision parseVerifierDecision(String verifierOutput, int round) {
|
||||
if (verifierOutput == null || verifierOutput.isBlank()) {
|
||||
return null;
|
||||
}
|
||||
|
||||
try {
|
||||
JsonNode root = objectMapper.readTree(sanitizeJsonPayload(verifierOutput));
|
||||
List<Map<String, Object>> factsChecked = parseFactsChecked(root.path("facts_checked"));
|
||||
|
||||
return new VerifierDecision(
|
||||
root.path("verdict").asText("LOW_CONFID"),
|
||||
root.path("groundedness_score").asDouble(0.0),
|
||||
root.path("critical_fact_count").asInt(0),
|
||||
factsChecked,
|
||||
root.path("rationale").asText(""),
|
||||
round
|
||||
);
|
||||
} catch (Exception e) {
|
||||
logger.error("解析 verifier_output 失败: {}", verifierOutput, e);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private String sanitizeJsonPayload(String raw) {
|
||||
String trimmed = raw.trim();
|
||||
if (trimmed.startsWith("```")) {
|
||||
int firstNewline = trimmed.indexOf('\n');
|
||||
int lastFence = trimmed.lastIndexOf("```");
|
||||
if (firstNewline >= 0 && lastFence > firstNewline) {
|
||||
return trimmed.substring(firstNewline + 1, lastFence).trim();
|
||||
}
|
||||
}
|
||||
return trimmed;
|
||||
}
|
||||
|
||||
private List<Map<String, Object>> parseFactsChecked(JsonNode factsNode) {
|
||||
List<Map<String, Object>> factsChecked = new ArrayList<>();
|
||||
if (!factsNode.isArray()) {
|
||||
return factsChecked;
|
||||
}
|
||||
for (JsonNode factNode : factsNode) {
|
||||
Map<String, Object> fact = new LinkedHashMap<>();
|
||||
fact.put("fact", factNode.path("fact").asText(""));
|
||||
fact.put("is_critical", factNode.path("is_critical").asBoolean(false));
|
||||
fact.put("verification", factNode.path("verification").asText(""));
|
||||
fact.put("detail", factNode.path("detail").asText(""));
|
||||
fact.put("evidence_refs", parseEvidenceRefs(factNode.path("evidence_refs")));
|
||||
factsChecked.add(fact);
|
||||
}
|
||||
return factsChecked;
|
||||
}
|
||||
|
||||
private List<Map<String, Object>> parseEvidenceRefs(JsonNode evidenceRefsNode) {
|
||||
List<Map<String, Object>> evidenceRefs = new ArrayList<>();
|
||||
if (!evidenceRefsNode.isArray()) {
|
||||
return evidenceRefs;
|
||||
}
|
||||
for (JsonNode refNode : evidenceRefsNode) {
|
||||
Map<String, Object> evidenceRef = new LinkedHashMap<>();
|
||||
evidenceRef.put("trace_ref", refNode.path("trace_ref").asText(""));
|
||||
evidenceRef.put("tool_name", refNode.path("tool_name").asText(""));
|
||||
evidenceRef.put("topic_domain", refNode.path("topic_domain").asText(""));
|
||||
evidenceRef.put("note", refNode.path("note").asText(""));
|
||||
|
||||
List<Long> sourceInvocationIds = new ArrayList<>();
|
||||
JsonNode idsNode = refNode.path("source_invocation_ids");
|
||||
if (idsNode.isArray()) {
|
||||
for (JsonNode idNode : idsNode) {
|
||||
if (idNode.canConvertToLong()) {
|
||||
sourceInvocationIds.add(idNode.asLong());
|
||||
}
|
||||
}
|
||||
}
|
||||
evidenceRef.put("source_invocation_ids", sourceInvocationIds);
|
||||
evidenceRefs.add(evidenceRef);
|
||||
}
|
||||
return evidenceRefs;
|
||||
}
|
||||
|
||||
private VerifierDecision buildVerifierFallbackDecision(int round, String rationale) {
|
||||
return new VerifierDecision("LOW_CONFID", 0.0, 0, List.of(), rationale, round);
|
||||
}
|
||||
|
||||
private String extractStateText(Optional<OverAllState> stateOptional, String key) {
|
||||
if (stateOptional.isEmpty()) {
|
||||
return null;
|
||||
}
|
||||
return stateOptional.get().value(key)
|
||||
.map(value -> {
|
||||
if (value instanceof AssistantMessage assistantMessage) {
|
||||
return assistantMessage.getText();
|
||||
}
|
||||
return String.valueOf(value);
|
||||
})
|
||||
.orElse(null);
|
||||
}
|
||||
|
||||
private void persistVerifierEvaluation(DiagnosisSession session, VerifierDecision decision, int round) {
|
||||
if (decision == null) {
|
||||
return;
|
||||
}
|
||||
Map<String, Object> verifierEvaluation = new LinkedHashMap<>();
|
||||
verifierEvaluation.put("verdict", decision.verdict());
|
||||
verifierEvaluation.put("groundedness_score", decision.groundednessScore());
|
||||
verifierEvaluation.put("critical_fact_count", decision.criticalFactCount());
|
||||
verifierEvaluation.put("facts_checked", decision.factsChecked());
|
||||
verifierEvaluation.put("rationale", decision.rationale());
|
||||
verifierEvaluation.put("round", round);
|
||||
verifierEvaluation.put("traceability_version", "v1");
|
||||
verifierEvaluation.put("tool_trace_summary",
|
||||
Optional.ofNullable(VerifierContextHolder.getToolTraceSummary()).orElse(List.of()));
|
||||
|
||||
String merged = selfEvaluationMergeService.mergeVerifierEvaluation(session.getSelfEvaluation(), verifierEvaluation);
|
||||
session.setSelfEvaluation(merged);
|
||||
diagnosisSessionRepository.save(session);
|
||||
}
|
||||
|
||||
private String buildRetryContext(VerifierDecision decision) {
|
||||
try {
|
||||
List<String> missingFacts = extractEvidenceGaps(decision);
|
||||
Map<String, Object> retryContext = new LinkedHashMap<>();
|
||||
retryContext.put("round", decision.round());
|
||||
retryContext.put("missing_evidence_facts", missingFacts);
|
||||
retryContext.put("instruction", "仅补充以上断言相关证据，不要重复已完成检索");
|
||||
return objectMapper.writeValueAsString(retryContext);
|
||||
} catch (Exception e) {
|
||||
logger.error("构造 retry_context 失败", e);
|
||||
return "{\"round\":1,\"missing_evidence_facts\":[],\"instruction\":\"仅补充缺失证据\"}";
|
||||
}
|
||||
}
|
||||
|
||||
private String buildLowConfidenceOutput(String executorAnswer, VerifierDecision decision) {
|
||||
StringBuilder output = new StringBuilder(LOW_CONFID_DISCLAIMER);
|
||||
output.append("\n\n").append(executorAnswer == null ? "" : executorAnswer);
|
||||
|
||||
List<String> gaps = extractEvidenceGaps(decision);
|
||||
if (!gaps.isEmpty()) {
|
||||
output.append("\n\n当前缺口：");
|
||||
for (String gap : gaps) {
|
||||
output.append("\n- ").append(gap);
|
||||
}
|
||||
}
|
||||
return output.toString();
|
||||
}
|
||||
|
||||
private String buildDegradedOutput(VerifierDecision decision) {
|
||||
StringBuilder output = new StringBuilder(DEGRADED_PREFIX);
|
||||
|
||||
List<String> confirmedFacts = extractConfirmedFacts(decision);
|
||||
List<String> gaps = extractEvidenceGaps(decision);
|
||||
List<String> suggestions = buildNextStepSuggestions(decision);
|
||||
|
||||
output.append("\n\n已确认信息：");
|
||||
if (confirmedFacts.isEmpty()) {
|
||||
output.append("\n- 暂无可稳定确认的信息");
|
||||
} else {
|
||||
for (String fact : confirmedFacts) {
|
||||
output.append("\n- ").append(fact);
|
||||
}
|
||||
}
|
||||
|
||||
output.append("\n\n证据缺口：");
|
||||
if (gaps.isEmpty()) {
|
||||
output.append("\n- 当前缺少足够的直接证据支撑核心结论");
|
||||
} else {
|
||||
for (String gap : gaps) {
|
||||
output.append("\n- ").append(gap);
|
||||
}
|
||||
}
|
||||
|
||||
output.append("\n\n建议下一步：");
|
||||
for (String suggestion : suggestions) {
|
||||
output.append("\n- ").append(suggestion);
|
||||
}
|
||||
return output.toString();
|
||||
}
|
||||
|
||||
private List<String> extractConfirmedFacts(VerifierDecision decision) {
|
||||
List<String> confirmedFacts = new ArrayList<>();
|
||||
for (Map<String, Object> fact : decision.factsChecked()) {
|
||||
String verification = String.valueOf(fact.get("verification"));
|
||||
boolean critical = Boolean.TRUE.equals(fact.get("is_critical"));
|
||||
if (critical && ("direct_evidence".equals(verification) || "indirect_support".equals(verification))) {
|
||||
confirmedFacts.add(String.valueOf(fact.get("fact")));
|
||||
}
|
||||
}
|
||||
return confirmedFacts;
|
||||
}
|
||||
|
||||
private List<String> extractEvidenceGaps(VerifierDecision decision) {
|
||||
List<String> gaps = new ArrayList<>();
|
||||
for (Map<String, Object> fact : decision.factsChecked()) {
|
||||
String verification = String.valueOf(fact.get("verification"));
|
||||
boolean critical = Boolean.TRUE.equals(fact.get("is_critical"));
|
||||
if (critical && ("no_evidence".equals(verification) || "contradicted".equals(verification))) {
|
||||
gaps.add(String.valueOf(fact.get("fact")) + "：" + String.valueOf(fact.get("detail")));
|
||||
}
|
||||
}
|
||||
if (gaps.isEmpty() && "LOW_CONFID".equals(decision.verdict())) {
|
||||
for (Map<String, Object> fact : decision.factsChecked()) {
|
||||
String verification = String.valueOf(fact.get("verification"));
|
||||
boolean critical = Boolean.TRUE.equals(fact.get("is_critical"));
|
||||
if (critical && "indirect_support".equals(verification)) {
|
||||
gaps.add(String.valueOf(fact.get("fact")) + "：缺少直接证据锚点");
|
||||
}
|
||||
}
|
||||
}
|
||||
return gaps;
|
||||
}
|
||||
|
||||
private List<String> buildNextStepSuggestions(VerifierDecision decision) {
|
||||
List<String> suggestions = new ArrayList<>();
|
||||
List<Map<String, Object>> toolSummary = toolTraceSummaryService.buildVerifierTraceSummary(SessionContextHolder.getSessionId(), null);
|
||||
boolean hasKnowledgeTool = toolSummary.stream().anyMatch(item -> "lookup_knowledge".equals(item.get("tool_name")));
|
||||
boolean hasFailedEvidence = toolSummary.stream().anyMatch(item -> !Boolean.TRUE.equals(item.get("success")));
|
||||
|
||||
if (!hasKnowledgeTool) {
|
||||
suggestions.add("补充知识库或业务文档检索结果，建立可引用的证据锚点");
|
||||
}
|
||||
if (hasFailedEvidence) {
|
||||
suggestions.add("优先重试失败的证据型查询，补齐日志、指标或知识库侧证据");
|
||||
}
|
||||
if (suggestions.isEmpty()) {
|
||||
suggestions.add("围绕上述证据缺口补充只读查询，再由人工复核最终结论");
|
||||
}
|
||||
return suggestions;
|
||||
}
|
||||
|
||||
private record VerifierDecision(
|
||||
String verdict,
|
||||
double groundednessScore,
|
||||
int criticalFactCount,
|
||||
List<Map<String, Object>> factsChecked,
|
||||
String rationale,
|
||||
int round
|
||||
) {
|
||||
}
|
||||
|
||||
/** 从 agent_step 汇总 token、步数等指标回填 diagnosis_session */
|
||||
private void backfillSessionMetrics(DiagnosisSession session) {
|
||||
try {
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
package com.superbiz.agent.service;
|
||||
|
||||
import com.fasterxml.jackson.core.type.TypeReference;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.domain.entity.AgentStep;
|
||||
import com.superbiz.agent.domain.entity.DiagnosisSession;
|
||||
import com.superbiz.agent.domain.entity.ToolInvocation;
|
||||
import com.superbiz.agent.dto.DiagnosisTraceResponse;
|
||||
import com.superbiz.agent.exception.SessionNotFoundException;
|
||||
import com.superbiz.agent.repository.AgentStepRepository;
|
||||
import com.superbiz.agent.repository.DiagnosisSessionRepository;
|
||||
import com.superbiz.agent.repository.ToolInvocationRepository;
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
@Service
|
||||
@RequiredArgsConstructor
|
||||
public class DiagnosisTraceService {
|
||||
|
||||
private static final TypeReference<Map<String, Object>> JSON_MAP_TYPE = new TypeReference<>() {
|
||||
};
|
||||
|
||||
private final DiagnosisSessionRepository diagnosisSessionRepository;
|
||||
private final AgentStepRepository agentStepRepository;
|
||||
private final ToolInvocationRepository toolInvocationRepository;
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
public DiagnosisTraceResponse getTrace(String sessionId) {
|
||||
DiagnosisSession session = diagnosisSessionRepository.findBySessionId(sessionId)
|
||||
.orElseThrow(() -> new SessionNotFoundException(sessionId));
|
||||
List<AgentStep> steps = agentStepRepository.findBySessionIdOrderByStepIndex(sessionId);
|
||||
List<ToolInvocation> toolInvocations = toolInvocationRepository.findBySessionIdOrderByIdAsc(sessionId);
|
||||
|
||||
return DiagnosisTraceResponse.builder()
|
||||
.session(toSessionTrace(session))
|
||||
.steps(steps.stream().map(this::toAgentStepTrace).toList())
|
||||
.toolInvocations(toolInvocations.stream().map(this::toToolInvocationTrace).toList())
|
||||
.summary(toSummary(session, steps, toolInvocations))
|
||||
.build();
|
||||
}
|
||||
|
||||
private DiagnosisTraceResponse.SessionTrace toSessionTrace(DiagnosisSession session) {
|
||||
return DiagnosisTraceResponse.SessionTrace.builder()
|
||||
.id(session.getId())
|
||||
.sessionId(session.getSessionId())
|
||||
.query(session.getQuery())
|
||||
.status(session.getStatus())
|
||||
.agentFlow(session.getAgentFlow())
|
||||
.totalDurationMs(session.getTotalDurationMs())
|
||||
.totalTokenCount(session.getTotalTokenCount())
|
||||
.stepCount(session.getStepCount())
|
||||
.toolCallCount(session.getToolCallCount())
|
||||
.answer(session.getAnswer())
|
||||
.selfEvaluationRaw(session.getSelfEvaluation())
|
||||
.selfEvaluation(parseJsonObject(session.getSelfEvaluation()))
|
||||
.feedback(session.getFeedback())
|
||||
.createdAt(session.getCreatedAt())
|
||||
.updatedAt(session.getUpdatedAt())
|
||||
.build();
|
||||
}
|
||||
|
||||
private DiagnosisTraceResponse.AgentStepTrace toAgentStepTrace(AgentStep step) {
|
||||
return DiagnosisTraceResponse.AgentStepTrace.builder()
|
||||
.id(step.getId())
|
||||
.sessionId(step.getSessionId())
|
||||
.stepIndex(step.getStepIndex())
|
||||
.agentName(step.getAgentName())
|
||||
.modelInput(step.getModelInput())
|
||||
.modelOutput(step.getModelOutput())
|
||||
.thought(step.getThought())
|
||||
.hasToolCall(step.getHasToolCall())
|
||||
.durationMs(step.getDurationMs())
|
||||
.tokenCount(step.getTokenCount())
|
||||
.createdAt(step.getCreatedAt())
|
||||
.build();
|
||||
}
|
||||
|
||||
private DiagnosisTraceResponse.ToolInvocationTrace toToolInvocationTrace(ToolInvocation invocation) {
|
||||
return DiagnosisTraceResponse.ToolInvocationTrace.builder()
|
||||
.id(invocation.getId())
|
||||
.sessionId(invocation.getSessionId())
|
||||
.stepId(invocation.getStepId())
|
||||
.toolName(invocation.getToolName())
|
||||
.inputParamsRaw(invocation.getInputParams())
|
||||
.inputParams(parseJsonObject(invocation.getInputParams()))
|
||||
.outputPreview(invocation.getOutputPreview())
|
||||
.outputLength(invocation.getOutputLength())
|
||||
.retrievalLayer(invocation.getRetrievalLayer())
|
||||
.l0MatchCount(invocation.getL0MatchCount())
|
||||
.l1MatchCount(invocation.getL1MatchCount())
|
||||
.truncated(invocation.getIsTruncated())
|
||||
.relevanceLevel(invocation.getRelevanceLevel())
|
||||
.dedupReason(invocation.getDedupReason())
|
||||
.retrievalDetailsRaw(invocation.getRetrievalDetails())
|
||||
.retrievalDetails(parseJsonObject(invocation.getRetrievalDetails()))
|
||||
.durationMs(invocation.getDurationMs())
|
||||
.success(invocation.getSuccess())
|
||||
.errorMessage(invocation.getErrorMessage())
|
||||
.createdAt(invocation.getCreatedAt())
|
||||
.build();
|
||||
}
|
||||
|
||||
private DiagnosisTraceResponse.TraceSummary toSummary(
|
||||
DiagnosisSession session,
|
||||
List<AgentStep> steps,
|
||||
List<ToolInvocation> toolInvocations
|
||||
) {
|
||||
Map<String, Object> selfEvaluation = parseJsonObject(session.getSelfEvaluation());
|
||||
return DiagnosisTraceResponse.TraceSummary.builder()
|
||||
.persistedStepCount(defaultInt(session.getStepCount()))
|
||||
.returnedStepCount(steps.size())
|
||||
.persistedToolCallCount(defaultInt(session.getToolCallCount()))
|
||||
.returnedToolCallCount(toolInvocations.size())
|
||||
.hasVerifierEvaluation(selfEvaluation != null && selfEvaluation.containsKey("verifier_evaluation"))
|
||||
.hasFeedback(session.getFeedback() != null && !session.getFeedback().isBlank())
|
||||
.build();
|
||||
}
|
||||
|
||||
private Map<String, Object> parseJsonObject(String json) {
|
||||
if (json == null || json.isBlank()) {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
return objectMapper.readValue(json, JSON_MAP_TYPE);
|
||||
} catch (Exception ignored) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private int defaultInt(Integer value) {
|
||||
return value == null ? 0 : value;
|
||||
}
|
||||
}
|
||||
@@ -260,7 +260,8 @@ public class DocumentManagementService {
|
||||
private String saveToLocal(MultipartFile file, String fileName, String category) {
|
||||
try {
|
||||
// 1. 构建目标路径
|
||||
Path categoryDir = Paths.get(knowledgeBasePath, category);
|
||||
Path baseDir = Paths.get(knowledgeBasePath).normalize();
|
||||
Path categoryDir = baseDir.resolve(category).normalize();
|
||||
Files.createDirectories(categoryDir);
|
||||
|
||||
Path targetPath = categoryDir.resolve(fileName);
|
||||
@@ -268,8 +269,9 @@ public class DocumentManagementService {
|
||||
// 2. 保存文件
|
||||
file.transferTo(targetPath.toFile());
|
||||
|
||||
log.info("文件已保存到本地: {}", targetPath);
|
||||
return targetPath.toString();
|
||||
String relativePath = baseDir.relativize(targetPath.normalize()).toString().replace("\\", "/");
|
||||
log.info("文件已保存到本地: {}, storedPath={}", targetPath, relativePath);
|
||||
return relativePath;
|
||||
|
||||
} catch (IOException e) {
|
||||
throw new DocumentProcessException(
|
||||
@@ -287,7 +289,7 @@ public class DocumentManagementService {
|
||||
private void cleanupLocalFile(String localPath) {
|
||||
if (localPath != null) {
|
||||
try {
|
||||
Files.deleteIfExists(Paths.get(localPath));
|
||||
Files.deleteIfExists(resolveLocalPath(localPath));
|
||||
log.info("已清理本地文件: {}", localPath);
|
||||
} catch (IOException e) {
|
||||
log.warn("清理本地文件失败: {}", localPath, e);
|
||||
@@ -357,7 +359,7 @@ public class DocumentManagementService {
|
||||
// 删除本地文件
|
||||
if (doc.getFilePath() != null) {
|
||||
try {
|
||||
Files.deleteIfExists(Paths.get(doc.getFilePath()));
|
||||
Files.deleteIfExists(resolveLocalPath(doc.getFilePath()));
|
||||
log.info("本地文件已删除: {}", doc.getFilePath());
|
||||
} catch (IOException e) {
|
||||
log.warn("删除本地文件失败: {}", doc.getFilePath(), e);
|
||||
@@ -407,6 +409,26 @@ public class DocumentManagementService {
|
||||
return null;
|
||||
}
|
||||
|
||||
private Path resolveLocalPath(String filePath) {
|
||||
Path path = Paths.get(filePath).normalize();
|
||||
if (path.isAbsolute()) {
|
||||
return path;
|
||||
}
|
||||
|
||||
Path basePath = Paths.get(knowledgeBasePath).toAbsolutePath().normalize();
|
||||
Path baseName = basePath.getFileName();
|
||||
if (baseName != null && path.startsWith(baseName) && basePath.getParent() != null) {
|
||||
return basePath.getParent().resolve(path).normalize();
|
||||
}
|
||||
|
||||
Path pathFromWorkingDir = path.toAbsolutePath().normalize();
|
||||
if (pathFromWorkingDir.startsWith(basePath)) {
|
||||
return pathFromWorkingDir;
|
||||
}
|
||||
|
||||
return basePath.resolve(path).normalize();
|
||||
}
|
||||
|
||||
/**
|
||||
* 转换为响应 DTO
|
||||
*/
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
package com.superbiz.agent.service;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.domain.entity.DiagnosisSession;
|
||||
import com.superbiz.agent.domain.entity.ToolInvocation;
|
||||
import com.superbiz.agent.repository.DiagnosisSessionRepository;
|
||||
@@ -12,6 +11,7 @@ import org.springframework.scheduling.annotation.Async;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
@@ -32,17 +32,19 @@ public class EvaluationService {
|
||||
@Autowired
|
||||
private ToolInvocationRepository toolInvocationRepository;
|
||||
|
||||
private final ObjectMapper objectMapper = new ObjectMapper();
|
||||
@Autowired
|
||||
private SelfEvaluationMergeService selfEvaluationMergeService;
|
||||
|
||||
@Async
|
||||
public void evaluate(String sessionId, String answer) {
|
||||
diagnosisSessionRepository.findBySessionId(sessionId).ifPresent(session -> {
|
||||
try {
|
||||
List<ToolInvocation> toolInvocations = toolInvocationRepository.findBySessionId(sessionId);
|
||||
String selfEvaluation = evaluateWithRules(session, toolInvocations);
|
||||
session.setSelfEvaluation(selfEvaluation);
|
||||
Map<String, Object> ruleEvaluation = evaluateWithRules(session, toolInvocations);
|
||||
String merged = selfEvaluationMergeService.mergeRuleEvaluation(session.getSelfEvaluation(), ruleEvaluation);
|
||||
session.setSelfEvaluation(merged);
|
||||
diagnosisSessionRepository.save(session);
|
||||
logger.info("证据评分已写入: sessionId={}, result={}", sessionId, selfEvaluation);
|
||||
logger.info("证据评分已写入: sessionId={}, result={}", sessionId, merged);
|
||||
} catch (Exception e) {
|
||||
logger.error("评分失败: sessionId={}", sessionId, e);
|
||||
}
|
||||
@@ -53,7 +55,7 @@ public class EvaluationService {
|
||||
// 规则引擎（事实层）
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
private String evaluateWithRules(DiagnosisSession session, List<ToolInvocation> invocations) {
|
||||
private Map<String, Object> evaluateWithRules(DiagnosisSession session, List<ToolInvocation> invocations) {
|
||||
List<Map<String, Object>> factors = new ArrayList<>();
|
||||
|
||||
if ("FAILED".equals(session.getStatus())) {
|
||||
@@ -117,19 +119,12 @@ public class EvaluationService {
|
||||
return Map.of("name", name, "delta", delta, "description", description);
|
||||
}
|
||||
|
||||
private String buildResult(int score, List<Map<String, Object>> factors) {
|
||||
try {
|
||||
Map<String, Object> result = Map.of(
|
||||
"evidence_score", score,
|
||||
"source", "rule",
|
||||
"factors", factors
|
||||
// llm_opinion: null ← 预留字段，LLM 观点叠加时在此处扩展
|
||||
);
|
||||
return objectMapper.writeValueAsString(result);
|
||||
} catch (Exception e) {
|
||||
logger.error("序列化评分结果失败", e);
|
||||
return "{\"evidence_score\":0,\"source\":\"rule\",\"factors\":[]}";
|
||||
}
|
||||
private Map<String, Object> buildResult(int score, List<Map<String, Object>> factors) {
|
||||
Map<String, Object> result = new LinkedHashMap<>();
|
||||
result.put("evidence_score", score);
|
||||
result.put("source", "rule");
|
||||
result.put("factors", factors);
|
||||
return result;
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
@@ -160,7 +160,12 @@ public class KnowledgeIndexService {
|
||||
|
||||
public String readDocument(String filePath, int maxChars) {
|
||||
try {
|
||||
Path fullPath = Paths.get(knowledgeBasePath, filePath);
|
||||
Path fullPath = resolveDocumentPath(filePath);
|
||||
if (!Files.exists(fullPath)) {
|
||||
log.warn("读取文档失败，文件不存在: basePath={}, filePath={}, resolvedPath={}",
|
||||
knowledgeBasePath, filePath, fullPath);
|
||||
return null;
|
||||
}
|
||||
String content = Files.readString(fullPath);
|
||||
|
||||
if (content.length() > maxChars) {
|
||||
@@ -170,11 +175,35 @@ public class KnowledgeIndexService {
|
||||
return content;
|
||||
|
||||
} catch (IOException e) {
|
||||
log.error("读取文档失败: {}/{}", knowledgeBasePath, filePath, e);
|
||||
log.error("读取文档失败: basePath={}, filePath={}", knowledgeBasePath, filePath, e);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
Path resolveDocumentPath(String filePath) {
|
||||
if (filePath == null || filePath.isBlank()) {
|
||||
throw new IllegalArgumentException("filePath cannot be blank");
|
||||
}
|
||||
|
||||
Path path = Paths.get(filePath).normalize();
|
||||
if (path.isAbsolute()) {
|
||||
return path;
|
||||
}
|
||||
|
||||
Path basePath = Paths.get(knowledgeBasePath).toAbsolutePath().normalize();
|
||||
Path baseName = basePath.getFileName();
|
||||
if (baseName != null && path.startsWith(baseName) && basePath.getParent() != null) {
|
||||
return basePath.getParent().resolve(path).normalize();
|
||||
}
|
||||
|
||||
Path pathFromWorkingDir = path.toAbsolutePath().normalize();
|
||||
if (pathFromWorkingDir.startsWith(basePath)) {
|
||||
return pathFromWorkingDir;
|
||||
}
|
||||
|
||||
return basePath.resolve(path).normalize();
|
||||
}
|
||||
|
||||
public void addToIndex(KnowledgeEntry entry) {
|
||||
knowledgeIndex.add(entry);
|
||||
log.debug("文档已添加到 L0 索引: title={}", entry.getTitle());
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
package com.superbiz.agent.service;
|
||||
|
||||
import com.fasterxml.jackson.core.type.TypeReference;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* 统一维护 diagnosis_session.self_evaluation JSON 容器。
|
||||
*/
|
||||
@Slf4j
|
||||
@Service
|
||||
public class SelfEvaluationMergeService {
|
||||
|
||||
private static final TypeReference<LinkedHashMap<String, Object>> MAP_TYPE = new TypeReference<>() {};
|
||||
|
||||
private final ObjectMapper objectMapper = new ObjectMapper();
|
||||
|
||||
public String mergeRuleEvaluation(String existingJson, Map<String, Object> ruleEvaluation) {
|
||||
return merge(existingJson, "rule_evaluation", ruleEvaluation);
|
||||
}
|
||||
|
||||
public String mergeVerifierEvaluation(String existingJson, Map<String, Object> verifierEvaluation) {
|
||||
return merge(existingJson, "verifier_evaluation", verifierEvaluation);
|
||||
}
|
||||
|
||||
private String merge(String existingJson, String key, Map<String, Object> value) {
|
||||
try {
|
||||
Map<String, Object> root = parseRoot(existingJson);
|
||||
root.put(key, value);
|
||||
return objectMapper.writeValueAsString(root);
|
||||
} catch (Exception e) {
|
||||
log.error("合并 self_evaluation 失败: key={}", key, e);
|
||||
return fallbackJson(key, value);
|
||||
}
|
||||
}
|
||||
|
||||
private Map<String, Object> parseRoot(String existingJson) throws Exception {
|
||||
if (existingJson == null || existingJson.isBlank()) {
|
||||
return new LinkedHashMap<>();
|
||||
}
|
||||
|
||||
Map<String, Object> parsed = objectMapper.readValue(existingJson, MAP_TYPE);
|
||||
if (parsed.containsKey("rule_evaluation") || parsed.containsKey("verifier_evaluation")) {
|
||||
return new LinkedHashMap<>(parsed);
|
||||
}
|
||||
|
||||
LinkedHashMap<String, Object> wrapped = new LinkedHashMap<>();
|
||||
if (parsed.containsKey("evidence_score") || parsed.containsKey("source") || parsed.containsKey("factors")) {
|
||||
wrapped.put("rule_evaluation", parsed);
|
||||
return wrapped;
|
||||
}
|
||||
if (parsed.containsKey("verdict") || parsed.containsKey("groundedness_score") || parsed.containsKey("facts_checked")) {
|
||||
wrapped.put("verifier_evaluation", parsed);
|
||||
return wrapped;
|
||||
}
|
||||
return new LinkedHashMap<>(parsed);
|
||||
}
|
||||
|
||||
private String fallbackJson(String key, Map<String, Object> value) {
|
||||
try {
|
||||
return objectMapper.writeValueAsString(Map.of(key, value));
|
||||
} catch (Exception ex) {
|
||||
log.error("兜底序列化 self_evaluation 失败", ex);
|
||||
return "{}";
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
package com.superbiz.agent.service;
|
||||
|
||||
import com.fasterxml.jackson.core.JsonProcessingException;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.domain.entity.ToolInvocation;
|
||||
import com.superbiz.agent.repository.ToolInvocationRepository;
|
||||
import com.superbiz.agent.util.SessionContextHolder;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
|
||||
/**
|
||||
* Central persistence point for agent evidence tool invocations.
|
||||
*/
|
||||
@Slf4j
|
||||
@Service
|
||||
public class ToolInvocationRecorder {
|
||||
|
||||
private static final int OUTPUT_PREVIEW_LIMIT = 500;
|
||||
|
||||
private final ToolInvocationRepository toolInvocationRepository;
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
public ToolInvocationRecorder(ToolInvocationRepository toolInvocationRepository, ObjectMapper objectMapper) {
|
||||
this.toolInvocationRepository = toolInvocationRepository;
|
||||
this.objectMapper = objectMapper;
|
||||
}
|
||||
|
||||
public void save(ToolInvocation invocation) {
|
||||
try {
|
||||
if (invocation.getSessionId() == null || invocation.getSessionId().isBlank()) {
|
||||
invocation.setSessionId(SessionContextHolder.getSessionId());
|
||||
}
|
||||
if (invocation.getSessionId() == null || invocation.getSessionId().isBlank()) {
|
||||
log.debug("Skip tool_invocation without sessionId: tool={}", invocation.getToolName());
|
||||
return;
|
||||
}
|
||||
toolInvocationRepository.save(invocation);
|
||||
} catch (Exception e) {
|
||||
log.error("保存 tool_invocation 失败: tool={}", invocation.getToolName(), e);
|
||||
}
|
||||
}
|
||||
|
||||
public void recordEvidenceTool(String toolName,
|
||||
Map<String, Object> inputParams,
|
||||
String output,
|
||||
boolean success,
|
||||
long startTimeMillis,
|
||||
String errorMessage,
|
||||
String topicDomain) {
|
||||
String outputPreview = preview(output);
|
||||
Map<String, Object> details = new LinkedHashMap<>();
|
||||
details.put("trace_id", UUID.randomUUID().toString());
|
||||
if (topicDomain != null && !topicDomain.isBlank()) {
|
||||
details.put("retrieved_domains", List.of(topicDomain));
|
||||
}
|
||||
|
||||
ToolInvocation invocation = ToolInvocation.builder()
|
||||
.toolName(toolName)
|
||||
.inputParams(toJson(inputParams == null ? Map.of() : inputParams))
|
||||
.outputPreview(outputPreview)
|
||||
.outputLength(output == null ? 0 : output.length())
|
||||
.isTruncated(output != null && output.length() > OUTPUT_PREVIEW_LIMIT)
|
||||
.retrievalDetails(toJson(details))
|
||||
.durationMs((int) Math.max(0, System.currentTimeMillis() - startTimeMillis))
|
||||
.success(success)
|
||||
.errorMessage(errorMessage)
|
||||
.build();
|
||||
save(invocation);
|
||||
}
|
||||
|
||||
private String preview(String output) {
|
||||
if (output == null) {
|
||||
return null;
|
||||
}
|
||||
return output.length() <= OUTPUT_PREVIEW_LIMIT
|
||||
? output
|
||||
: output.substring(0, OUTPUT_PREVIEW_LIMIT) + "...";
|
||||
}
|
||||
|
||||
private String toJson(Map<String, Object> value) {
|
||||
try {
|
||||
return objectMapper.writeValueAsString(value);
|
||||
} catch (JsonProcessingException e) {
|
||||
log.debug("tool_invocation JSON 序列化失败", e);
|
||||
return "{}";
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,307 @@
|
||||
package com.superbiz.agent.service;
|
||||
|
||||
import com.fasterxml.jackson.core.type.TypeReference;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.domain.entity.ToolInvocation;
|
||||
import com.superbiz.agent.repository.ToolInvocationRepository;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Comparator;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
/**
|
||||
* Builds a verifier-facing evidence index from persisted tool invocations.
|
||||
*/
|
||||
@Slf4j
|
||||
@Service
|
||||
public class ToolTraceSummaryService {
|
||||
|
||||
private static final TypeReference<LinkedHashMap<String, Object>> MAP_TYPE = new TypeReference<>() {};
|
||||
private static final Set<String> EVIDENCE_TOOLS = Set.of("lookup_knowledge", "query_logs", "query_metrics", "query_order");
|
||||
|
||||
private final ToolInvocationRepository toolInvocationRepository;
|
||||
private final ObjectMapper objectMapper = new ObjectMapper();
|
||||
|
||||
public ToolTraceSummaryService(ToolInvocationRepository toolInvocationRepository) {
|
||||
this.toolInvocationRepository = toolInvocationRepository;
|
||||
}
|
||||
|
||||
public List<Map<String, Object>> buildVerifierTraceSummary(String sessionId, String executorFinalAnswer) {
|
||||
if (sessionId == null || sessionId.isBlank()) {
|
||||
return List.of();
|
||||
}
|
||||
|
||||
List<ToolInvocation> invocations = toolInvocationRepository.findBySessionIdOrderByIdAsc(sessionId);
|
||||
if (invocations.isEmpty()) {
|
||||
return List.of();
|
||||
}
|
||||
|
||||
Map<String, AggregateEntry> grouped = new LinkedHashMap<>();
|
||||
for (ToolInvocation invocation : invocations) {
|
||||
if (!EVIDENCE_TOOLS.contains(invocation.getToolName())) {
|
||||
continue;
|
||||
}
|
||||
String topicDomain = extractTopicDomain(invocation);
|
||||
String key = invocation.getToolName() + "|" + topicDomain;
|
||||
AggregateEntry entry = grouped.computeIfAbsent(
|
||||
key,
|
||||
ignored -> new AggregateEntry(invocation.getToolName(), topicDomain));
|
||||
entry.absorb(invocation);
|
||||
}
|
||||
|
||||
List<AggregateEntry> rankedEntries = grouped.values().stream()
|
||||
.sorted(Comparator.comparingInt((AggregateEntry entry) -> entry.relevanceScore(executorFinalAnswer)).reversed())
|
||||
.limit(8)
|
||||
.toList();
|
||||
|
||||
List<Map<String, Object>> summaries = new ArrayList<>();
|
||||
for (int i = 0; i < rankedEntries.size(); i++) {
|
||||
summaries.add(rankedEntries.get(i).toSummary("trace-" + (i + 1)));
|
||||
}
|
||||
return summaries;
|
||||
}
|
||||
|
||||
private String extractTopicDomain(ToolInvocation invocation) {
|
||||
try {
|
||||
if (invocation.getRetrievalDetails() != null && !invocation.getRetrievalDetails().isBlank()) {
|
||||
Map<String, Object> details = objectMapper.readValue(invocation.getRetrievalDetails(), MAP_TYPE);
|
||||
Object domains = details.get("retrieved_domains");
|
||||
if (domains instanceof List<?> domainList && !domainList.isEmpty()) {
|
||||
return String.valueOf(domainList.get(0));
|
||||
}
|
||||
}
|
||||
} catch (Exception e) {
|
||||
log.debug("Failed to parse retrieved_domains, fallback to general", e);
|
||||
}
|
||||
return "general";
|
||||
}
|
||||
|
||||
private String extractInputSummary(ToolInvocation invocation) {
|
||||
String query = extractQuery(invocation);
|
||||
if (query != null && !query.isBlank()) {
|
||||
return "query=" + truncate(query, 120);
|
||||
}
|
||||
return invocation.getToolName() + " invoked";
|
||||
}
|
||||
|
||||
private String extractQuery(ToolInvocation invocation) {
|
||||
try {
|
||||
if (invocation.getInputParams() != null && !invocation.getInputParams().isBlank()) {
|
||||
Map<String, Object> params = objectMapper.readValue(invocation.getInputParams(), MAP_TYPE);
|
||||
Object query = params.get("query");
|
||||
if (query != null) {
|
||||
return String.valueOf(query);
|
||||
}
|
||||
}
|
||||
} catch (Exception e) {
|
||||
log.debug("Failed to parse invocation query", e);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
private String extractOutputSummary(ToolInvocation invocation, String topicDomain) {
|
||||
if (!Boolean.TRUE.equals(invocation.getSuccess())) {
|
||||
if (invocation.getErrorMessage() != null && !invocation.getErrorMessage().isBlank()) {
|
||||
return "call failed: " + truncate(invocation.getErrorMessage(), 120);
|
||||
}
|
||||
return "no usable evidence returned";
|
||||
}
|
||||
|
||||
if ("lookup_knowledge".equals(invocation.getToolName())) {
|
||||
String relevance = invocation.getRelevanceLevel() != null ? invocation.getRelevanceLevel() : "UNKNOWN";
|
||||
String preview = invocation.getOutputPreview() != null && !invocation.getOutputPreview().isBlank()
|
||||
? truncate(invocation.getOutputPreview(), 160)
|
||||
: "no preview";
|
||||
return "matched domain=" + topicDomain + ", relevance=" + relevance + ", preview=" + preview;
|
||||
}
|
||||
|
||||
if (invocation.getOutputPreview() != null && !invocation.getOutputPreview().isBlank()) {
|
||||
return truncate(invocation.getOutputPreview(), 160);
|
||||
}
|
||||
return "evidence retrieved without preview";
|
||||
}
|
||||
|
||||
private String determineEvidenceLevel(ToolInvocation invocation) {
|
||||
if (!Boolean.TRUE.equals(invocation.getSuccess())) {
|
||||
return "none";
|
||||
}
|
||||
if ("PRECISE".equals(invocation.getRelevanceLevel()) || "HIGHLY_RELEVANT".equals(invocation.getRelevanceLevel())) {
|
||||
return "direct";
|
||||
}
|
||||
if ("REFERENCE".equals(invocation.getRelevanceLevel())) {
|
||||
return "indirect";
|
||||
}
|
||||
return "none";
|
||||
}
|
||||
|
||||
private List<String> extractStringList(Object value) {
|
||||
if (!(value instanceof List<?> list) || list.isEmpty()) {
|
||||
return List.of();
|
||||
}
|
||||
List<String> result = new ArrayList<>();
|
||||
for (Object item : list) {
|
||||
if (item != null) {
|
||||
result.add(String.valueOf(item));
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
private List<String> extractSourceDocuments(ToolInvocation invocation) {
|
||||
if (invocation.getRetrievalDetails() == null || invocation.getRetrievalDetails().isBlank()) {
|
||||
return List.of();
|
||||
}
|
||||
try {
|
||||
Map<String, Object> details = objectMapper.readValue(invocation.getRetrievalDetails(), MAP_TYPE);
|
||||
List<String> paths = extractStringList(details.get("l0_paths"));
|
||||
if (!paths.isEmpty()) {
|
||||
return paths;
|
||||
}
|
||||
List<String> titles = extractStringList(details.get("l0_titles"));
|
||||
if (!titles.isEmpty()) {
|
||||
return titles;
|
||||
}
|
||||
} catch (Exception e) {
|
||||
log.debug("Failed to parse source documents", e);
|
||||
}
|
||||
return List.of();
|
||||
}
|
||||
|
||||
private String truncate(String text, int maxLength) {
|
||||
if (text == null) {
|
||||
return "";
|
||||
}
|
||||
return text.length() <= maxLength ? text : text.substring(0, maxLength) + "...";
|
||||
}
|
||||
|
||||
private final class AggregateEntry {
|
||||
private final String toolName;
|
||||
private final String topicDomain;
|
||||
private String inputSummary;
|
||||
private String outputSummary;
|
||||
private boolean success;
|
||||
private String evidenceLevel = "none";
|
||||
private int invocationCount;
|
||||
private int failedCount;
|
||||
private int noHitCount;
|
||||
private final List<Long> sourceInvocationIds = new ArrayList<>();
|
||||
private final LinkedHashSet<String> querySamples = new LinkedHashSet<>();
|
||||
private final LinkedHashSet<String> retrievalLayers = new LinkedHashSet<>();
|
||||
private final LinkedHashSet<String> relevanceLevels = new LinkedHashSet<>();
|
||||
private final LinkedHashSet<String> sourceDocuments = new LinkedHashSet<>();
|
||||
|
||||
private AggregateEntry(String toolName, String topicDomain) {
|
||||
this.toolName = toolName;
|
||||
this.topicDomain = topicDomain;
|
||||
}
|
||||
|
||||
void absorb(ToolInvocation invocation) {
|
||||
invocationCount++;
|
||||
if (invocation.getId() != null) {
|
||||
sourceInvocationIds.add(invocation.getId());
|
||||
}
|
||||
|
||||
String query = extractQuery(invocation);
|
||||
if (query != null && !query.isBlank()) {
|
||||
querySamples.add(query);
|
||||
}
|
||||
if (invocation.getRetrievalLayer() != null && !invocation.getRetrievalLayer().isBlank()) {
|
||||
retrievalLayers.add(invocation.getRetrievalLayer());
|
||||
}
|
||||
if (invocation.getRelevanceLevel() != null && !invocation.getRelevanceLevel().isBlank()) {
|
||||
relevanceLevels.add(invocation.getRelevanceLevel());
|
||||
}
|
||||
sourceDocuments.addAll(extractSourceDocuments(invocation));
|
||||
|
||||
if (inputSummary == null || inputSummary.isBlank()) {
|
||||
inputSummary = extractInputSummary(invocation);
|
||||
}
|
||||
|
||||
boolean invocationSuccess = Boolean.TRUE.equals(invocation.getSuccess());
|
||||
if (!invocationSuccess) {
|
||||
failedCount++;
|
||||
return;
|
||||
}
|
||||
if (invocation.getDedupReason() != null) {
|
||||
noHitCount++;
|
||||
}
|
||||
|
||||
String invocationEvidenceLevel = determineEvidenceLevel(invocation);
|
||||
if (!success || evidenceRank(invocationEvidenceLevel) > evidenceRank(evidenceLevel)) {
|
||||
success = true;
|
||||
evidenceLevel = invocationEvidenceLevel;
|
||||
outputSummary = extractOutputSummary(invocation, topicDomain);
|
||||
}
|
||||
}
|
||||
|
||||
int relevanceScore(String answer) {
|
||||
int score = success ? 10 : 0;
|
||||
if ("direct".equals(evidenceLevel)) {
|
||||
score += 10;
|
||||
} else if ("indirect".equals(evidenceLevel)) {
|
||||
score += 5;
|
||||
}
|
||||
if (answer != null) {
|
||||
String normalized = answer.toLowerCase(Locale.ROOT);
|
||||
if (normalized.contains(topicDomain.toLowerCase(Locale.ROOT))) {
|
||||
score += 8;
|
||||
}
|
||||
if (normalized.contains(toolName.toLowerCase(Locale.ROOT))) {
|
||||
score += 3;
|
||||
}
|
||||
}
|
||||
return score;
|
||||
}
|
||||
|
||||
Map<String, Object> toSummary(String traceRef) {
|
||||
String mergedOutput = outputSummary == null ? "no summarized evidence" : outputSummary;
|
||||
if (invocationCount > 1) {
|
||||
StringBuilder builder = new StringBuilder(mergedOutput);
|
||||
builder.append(" (merged ").append(invocationCount).append(" invocations");
|
||||
if (failedCount > 0) {
|
||||
builder.append(", failed=").append(failedCount);
|
||||
}
|
||||
if (noHitCount > 0) {
|
||||
builder.append(", no_hit=").append(noHitCount);
|
||||
}
|
||||
builder.append(")");
|
||||
mergedOutput = builder.toString();
|
||||
}
|
||||
|
||||
Map<String, Object> summary = new LinkedHashMap<>();
|
||||
summary.put("trace_ref", traceRef);
|
||||
summary.put("tool_name", toolName);
|
||||
summary.put("success", success);
|
||||
summary.put("input_summary", inputSummary);
|
||||
summary.put("output_summary", mergedOutput);
|
||||
summary.put("evidence_level", evidenceLevel);
|
||||
summary.put("topic_domain", topicDomain);
|
||||
summary.put("source_invocation_ids", new ArrayList<>(sourceInvocationIds));
|
||||
summary.put("invocation_count", invocationCount);
|
||||
summary.put("failed_invocation_count", failedCount);
|
||||
summary.put("no_hit_invocation_count", noHitCount);
|
||||
summary.put("query_samples", new ArrayList<>(querySamples));
|
||||
summary.put("retrieval_layers", new ArrayList<>(retrievalLayers));
|
||||
summary.put("relevance_levels", new ArrayList<>(relevanceLevels));
|
||||
summary.put("source_documents", new ArrayList<>(sourceDocuments));
|
||||
return summary;
|
||||
}
|
||||
|
||||
private int evidenceRank(String level) {
|
||||
if ("direct".equals(level)) {
|
||||
return 2;
|
||||
}
|
||||
if ("indirect".equals(level)) {
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3,8 +3,8 @@ package com.superbiz.agent.tool;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.domain.entity.ToolInvocation;
|
||||
import com.superbiz.agent.dto.*;
|
||||
import com.superbiz.agent.repository.ToolInvocationRepository;
|
||||
import com.superbiz.agent.service.KnowledgeIndexService;
|
||||
import com.superbiz.agent.service.ToolInvocationRecorder;
|
||||
import com.superbiz.agent.service.VectorSearchService;
|
||||
import com.superbiz.agent.util.SessionContextHolder;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
@@ -49,7 +49,7 @@ public class LookupKnowledgeTool {
|
||||
private VectorSearchService vectorSearchService;
|
||||
|
||||
@Autowired
|
||||
private ToolInvocationRepository toolInvocationRepository;
|
||||
private ToolInvocationRecorder toolInvocationRecorder;
|
||||
|
||||
@Autowired
|
||||
private RetrievedDocTracker retrievedDocTracker;
|
||||
@@ -417,7 +417,7 @@ public class LookupKnowledgeTool {
|
||||
.success(true)
|
||||
.build();
|
||||
|
||||
toolInvocationRepository.save(inv);
|
||||
toolInvocationRecorder.save(inv);
|
||||
log.debug("tool_invocation 已保存: sessionId={}, layer={}, relevanceLevel={}, duration={}ms",
|
||||
sessionId, layer, result != null ? result.getRelevanceLevel() : null, duration);
|
||||
} catch (Exception e) {
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
package com.superbiz.agent.util;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* Thread-local verifier context shared across one planner/executor/verifier round.
|
||||
*/
|
||||
public final class VerifierContextHolder {
|
||||
|
||||
private static final ThreadLocal<String> ORIGINAL_QUERY = new ThreadLocal<>();
|
||||
private static final ThreadLocal<String> RETRY_CONTEXT = new ThreadLocal<>();
|
||||
private static final ThreadLocal<String> EXECUTOR_FINAL_ANSWER = new ThreadLocal<>();
|
||||
private static final ThreadLocal<List<Map<String, Object>>> TOOL_TRACE_SUMMARY = new ThreadLocal<>();
|
||||
|
||||
private VerifierContextHolder() {
|
||||
}
|
||||
|
||||
public static void setOriginalQuery(String originalQuery) {
|
||||
ORIGINAL_QUERY.set(originalQuery);
|
||||
}
|
||||
|
||||
public static String getOriginalQuery() {
|
||||
return ORIGINAL_QUERY.get();
|
||||
}
|
||||
|
||||
public static void setRetryContext(String retryContext) {
|
||||
RETRY_CONTEXT.set(retryContext);
|
||||
}
|
||||
|
||||
public static String getRetryContext() {
|
||||
return RETRY_CONTEXT.get();
|
||||
}
|
||||
|
||||
public static void setExecutorFinalAnswer(String executorFinalAnswer) {
|
||||
EXECUTOR_FINAL_ANSWER.set(executorFinalAnswer);
|
||||
}
|
||||
|
||||
public static String getExecutorFinalAnswer() {
|
||||
return EXECUTOR_FINAL_ANSWER.get();
|
||||
}
|
||||
|
||||
public static void setToolTraceSummary(List<Map<String, Object>> toolTraceSummary) {
|
||||
TOOL_TRACE_SUMMARY.set(toolTraceSummary);
|
||||
}
|
||||
|
||||
public static List<Map<String, Object>> getToolTraceSummary() {
|
||||
return TOOL_TRACE_SUMMARY.get();
|
||||
}
|
||||
|
||||
public static void clear() {
|
||||
ORIGINAL_QUERY.remove();
|
||||
RETRY_CONTEXT.remove();
|
||||
EXECUTOR_FINAL_ANSWER.remove();
|
||||
TOOL_TRACE_SUMMARY.remove();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
spring:
|
||||
config:
|
||||
activate:
|
||||
on-profile: mvp-demo
|
||||
|
||||
server:
|
||||
port: 9900
|
||||
|
||||
prometheus:
|
||||
mock-enabled: true
|
||||
timeout: 5
|
||||
|
||||
cls:
|
||||
mock-enabled: true
|
||||
|
||||
logging:
|
||||
level:
|
||||
root: INFO
|
||||
com.superbiz.agent: DEBUG
|
||||
com.alibaba.cloud: INFO
|
||||
|
||||
mvp:
|
||||
demo:
|
||||
name: payment-timeout-trace
|
||||
description: Repeatable MVP flow for chat diagnosis, tool evidence, verifier evaluation, trace query, and feedback.
|
||||
@@ -0,0 +1,142 @@
|
||||
你是质量闸 verifier。你的任务是对 `executor_final_answer` 做一次基于现有证据的事实校验。
|
||||
|
||||
边界约束：
|
||||
- 不做新的检索
|
||||
- 不做超出输入证据的推理扩写
|
||||
- 不补充输入中不存在的新事实
|
||||
- 只输出一个合法 JSON 对象，不输出 Markdown，不输出代码块，不输出额外说明
|
||||
|
||||
## 输入字段
|
||||
|
||||
- `original_query`：用户原始问题
|
||||
- `executor_final_answer`：本轮 Executor 最终答案
|
||||
- `tool_trace_summary`：基于真实工具调用整理出的证据索引。每一项都带有：
|
||||
- `trace_ref`
|
||||
- `tool_name`
|
||||
- `topic_domain`
|
||||
- `source_invocation_ids`
|
||||
- `input_summary`
|
||||
- `output_summary`
|
||||
- `evidence_level`
|
||||
- `retry_context`：第二轮可选输入；若为空，按首轮处理
|
||||
|
||||
## 任务步骤
|
||||
|
||||
### 步骤一：提取关键事实
|
||||
优先提取并校验 `executor_final_answer` 里的全部实质性结论。关键事实至少包括：
|
||||
- 每一个根因结论
|
||||
- 每一个错误码、接口、组件归属或语义判断
|
||||
- 每一个明确的修复建议、参数建议、排查步骤
|
||||
- 每一个“证据来源陈述”
|
||||
|
||||
覆盖要求：
|
||||
- 不允许只抽取一个总括性事实替代整段答案
|
||||
- 如果答案给出多个根因，必须逐条拆成多个 `fact`
|
||||
- 如果答案给出多条修复建议，必须逐条拆成多个 `fact`
|
||||
- 只有寒暄、流程衔接语、与结论无关的话，才可以不纳入 `facts_checked`
|
||||
|
||||
### 步骤二：逐条校验事实
|
||||
每条事实必须输出：
|
||||
- `fact`
|
||||
- `is_critical`
|
||||
- `verification`
|
||||
- `detail`
|
||||
- `evidence_refs`
|
||||
|
||||
`verification` 只允许以下四个值：
|
||||
- `direct_evidence`
|
||||
- `indirect_support`
|
||||
- `no_evidence`
|
||||
- `contradicted`
|
||||
|
||||
### 步骤三：补齐 evidence_refs
|
||||
`evidence_refs` 必须是数组，数组元素必须引用 `tool_trace_summary` 中真实存在的证据项。每个元素包含：
|
||||
- `trace_ref`
|
||||
- `tool_name`
|
||||
- `topic_domain`
|
||||
- `source_invocation_ids`
|
||||
- `note`
|
||||
|
||||
规则：
|
||||
- 有证据支撑时，必须引用支撑该事实的证据项
|
||||
- `no_evidence` 并不等于不引用
|
||||
- 如果工具确实查过相关方向，但证据不够，仍应引用对应 trace，并在 `note` 里说明“不足以支撑”
|
||||
- 只有当确实找不到相关 trace 时，`evidence_refs` 才允许为空数组
|
||||
- 不允许编造不存在的 `trace_ref` 或 `source_invocation_ids`
|
||||
|
||||
### 步骤四：生成 verdict
|
||||
严格使用以下判定矩阵：
|
||||
1. 若任一关键事实（`is_critical=true`）为 `contradicted`
|
||||
- `verdict = "REJECT"`
|
||||
- `groundedness_score = 0.0`
|
||||
|
||||
2. 否则，若所有关键事实均为 `direct_evidence` 或 `indirect_support`
|
||||
且至少一条关键事实为 `direct_evidence`
|
||||
- `verdict = "PASS"`
|
||||
|
||||
3. 否则，若不存在 `contradicted`
|
||||
且存在关键事实为 `no_evidence`
|
||||
或所有关键事实都只有 `indirect_support`
|
||||
- `verdict = "LOW_CONFID"`
|
||||
|
||||
### 步骤五：计算 groundedness_score
|
||||
只统计 `is_critical=true` 的事实，映射如下：
|
||||
- `direct_evidence = 1.0`
|
||||
- `indirect_support = 0.6`
|
||||
- `no_evidence = 0.0`
|
||||
- `contradicted = 0.0`
|
||||
|
||||
规则：
|
||||
- 若任一关键事实为 `contradicted`，分数固定为 `0.0`
|
||||
- 否则对关键事实取平均值
|
||||
- 保留 2 位小数
|
||||
- 分数范围必须在 `[0.0, 1.0]`
|
||||
|
||||
### 步骤六：PASS 前覆盖性自检
|
||||
在输出 `PASS` 前，必须再次检查：
|
||||
- `facts_checked` 是否覆盖了 `executor_final_answer` 的全部实质性结论
|
||||
- 是否遗漏了单独出现的根因、修复建议、参数建议、排查步骤
|
||||
|
||||
如有明显遗漏，即使已校验事实都有证据，也不得输出 `PASS`。
|
||||
|
||||
### 步骤七：处理 retry_context
|
||||
若 `retry_context` 不为空：
|
||||
- 优先检查上一轮缺失证据点是否已补足
|
||||
- 不要扩展与缺口无关的新事实
|
||||
- 不要因为存在 `retry_context` 就自动降低 verdict
|
||||
|
||||
## 输出协议
|
||||
|
||||
必须输出且只能输出以下 JSON 结构：
|
||||
|
||||
{
|
||||
"verdict": "PASS",
|
||||
"groundedness_score": 0.8,
|
||||
"critical_fact_count": 2,
|
||||
"facts_checked": [
|
||||
{
|
||||
"fact": "ERR_TIMEOUT 表示请求超时",
|
||||
"is_critical": true,
|
||||
"verification": "direct_evidence",
|
||||
"detail": "知识库文档明确给出该错误码定义",
|
||||
"evidence_refs": [
|
||||
{
|
||||
"trace_ref": "trace-1",
|
||||
"tool_name": "lookup_knowledge",
|
||||
"topic_domain": "api",
|
||||
"source_invocation_ids": [101, 104],
|
||||
"note": "trace-1 的文档摘要直接给出错误码定义"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"rationale": "所有关键事实均有支撑，且至少一条具有直接证据"
|
||||
}
|
||||
|
||||
输出要求：
|
||||
- `verdict` 只能是 `PASS` / `LOW_CONFID` / `REJECT`
|
||||
- `groundedness_score` 必须是 JSON number
|
||||
- `critical_fact_count` 必须等于 `facts_checked` 中 `is_critical=true` 的数量
|
||||
- `facts_checked` 可以为空数组，但字段不能缺失
|
||||
- 每条 `facts_checked[*]` 都必须包含 `evidence_refs`
|
||||
- 不得输出 schema 之外的字段
|
||||
@@ -0,0 +1,223 @@
|
||||
package com.superbiz.agent.service;
|
||||
|
||||
import com.superbiz.agent.agent.tool.DateTimeTools;
|
||||
import com.superbiz.agent.agent.tool.QueryLogsTools;
|
||||
import com.superbiz.agent.agent.tool.QueryMetricsTools;
|
||||
import com.superbiz.agent.domain.entity.AgentStep;
|
||||
import com.superbiz.agent.domain.entity.DiagnosisSession;
|
||||
import com.superbiz.agent.repository.AgentStepRepository;
|
||||
import com.superbiz.agent.repository.DiagnosisSessionRepository;
|
||||
import com.superbiz.agent.tool.LookupKnowledgeTool;
|
||||
import com.superbiz.agent.tool.RetrievedDocTracker;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.ai.chat.messages.AssistantMessage;
|
||||
import org.springframework.ai.chat.model.ChatModel;
|
||||
import org.springframework.ai.chat.model.ChatResponse;
|
||||
import org.springframework.ai.chat.model.Generation;
|
||||
import org.springframework.ai.chat.prompt.Prompt;
|
||||
import org.springframework.ai.tool.ToolCallback;
|
||||
import org.springframework.test.util.ReflectionTestUtils;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertSame;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.anyString;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
class ChatServiceSequentialAgentTest {
|
||||
|
||||
@Test
|
||||
void executeChatComplexInvokesSequentialWorkflow() throws Exception {
|
||||
ChatService chatService = createChatService();
|
||||
ScriptedChatModel chatModel = new ScriptedChatModel();
|
||||
|
||||
ChatService.ChatResult result = chatService.executeChatComplex(
|
||||
chatModel,
|
||||
new ToolCallback[0],
|
||||
"请分析订单支付超时的原因，并给出修复建议",
|
||||
List.of(),
|
||||
"sequential-test-session"
|
||||
);
|
||||
|
||||
assertEquals("EXECUTOR_FINAL_ANSWER", result.answer());
|
||||
assertEquals("sequential-test-session", result.sessionId());
|
||||
assertEquals(List.of("chat_planner", "chat_executor", "chat_verifier"), chatModel.agentCalls);
|
||||
assertTrue(chatModel.sawVerifierPrompt);
|
||||
}
|
||||
|
||||
@Test
|
||||
void executeChatComplexDoesNotRetryLowConfidenceByDefault() throws Exception {
|
||||
ChatService chatService = createChatService();
|
||||
ScriptedChatModel chatModel = new ScriptedChatModel("""
|
||||
{
|
||||
"verdict": "LOW_CONFID",
|
||||
"groundedness_score": 0.1,
|
||||
"critical_fact_count": 1,
|
||||
"facts_checked": [
|
||||
{
|
||||
"fact": "missing direct evidence",
|
||||
"is_critical": true,
|
||||
"verification": "no_evidence",
|
||||
"detail": "scripted evidence gap",
|
||||
"evidence_refs": []
|
||||
}
|
||||
],
|
||||
"rationale": "scripted low confidence"
|
||||
}
|
||||
""");
|
||||
|
||||
ChatService.ChatResult result = chatService.executeChatComplex(
|
||||
chatModel,
|
||||
new ToolCallback[0],
|
||||
"请分析订单支付超时的原因，并给出修复建议",
|
||||
List.of(),
|
||||
"sequential-low-confidence-session"
|
||||
);
|
||||
|
||||
assertTrue(result.answer().contains("EXECUTOR_FINAL_ANSWER"));
|
||||
assertEquals(List.of("chat_planner", "chat_executor", "chat_verifier"), chatModel.agentCalls);
|
||||
}
|
||||
|
||||
@Test
|
||||
void executeChatComplexRunsPlannerExecutorVerifierInFixedOrder() throws Exception {
|
||||
ChatService chatService = createChatService();
|
||||
ScriptedChatModel chatModel = new ScriptedChatModel();
|
||||
|
||||
ChatService.ChatResult result = chatService.executeChatComplex(
|
||||
chatModel,
|
||||
new ToolCallback[0],
|
||||
"请分析订单支付超时的原因，并给出修复建议",
|
||||
List.of(),
|
||||
"sequential-workflow-session"
|
||||
);
|
||||
|
||||
assertEquals("EXECUTOR_FINAL_ANSWER", result.answer());
|
||||
assertEquals(List.of("chat_planner", "chat_executor", "chat_verifier"), chatModel.agentCalls);
|
||||
assertTrue(chatModel.sawVerifierPrompt);
|
||||
}
|
||||
|
||||
@Test
|
||||
void buildMethodToolsArrayIncludesLogsAndMetricsWhenAvailable() {
|
||||
ChatService chatService = new ChatService();
|
||||
DateTimeTools dateTimeTools = new DateTimeTools();
|
||||
LookupKnowledgeTool lookupKnowledgeTool = new LookupKnowledgeTool();
|
||||
QueryLogsTools queryLogsTools = new QueryLogsTools(mock(ToolInvocationRecorder.class));
|
||||
QueryMetricsTools queryMetricsTools = new QueryMetricsTools(mock(ToolInvocationRecorder.class));
|
||||
|
||||
ReflectionTestUtils.setField(chatService, "dateTimeTools", dateTimeTools);
|
||||
ReflectionTestUtils.setField(chatService, "lookupKnowledgeTool", lookupKnowledgeTool);
|
||||
ReflectionTestUtils.setField(chatService, "queryLogsTools", queryLogsTools);
|
||||
ReflectionTestUtils.setField(chatService, "queryMetricsTools", queryMetricsTools);
|
||||
|
||||
Object[] methodTools = chatService.buildMethodToolsArray();
|
||||
|
||||
assertEquals(4, methodTools.length);
|
||||
assertSame(dateTimeTools, methodTools[0]);
|
||||
assertSame(lookupKnowledgeTool, methodTools[1]);
|
||||
assertSame(queryLogsTools, methodTools[2]);
|
||||
assertSame(queryMetricsTools, methodTools[3]);
|
||||
}
|
||||
|
||||
private ChatService createChatService() {
|
||||
ChatService chatService = new ChatService();
|
||||
|
||||
DiagnosisSessionRepository diagnosisSessionRepository = mock(DiagnosisSessionRepository.class);
|
||||
when(diagnosisSessionRepository.findBySessionId(anyString())).thenReturn(Optional.empty());
|
||||
when(diagnosisSessionRepository.save(any(DiagnosisSession.class))).thenAnswer(invocation -> invocation.getArgument(0));
|
||||
|
||||
AtomicInteger stepId = new AtomicInteger(1);
|
||||
AgentStepRepository agentStepRepository = mock(AgentStepRepository.class);
|
||||
when(agentStepRepository.save(any(AgentStep.class))).thenAnswer(invocation -> {
|
||||
AgentStep step = invocation.getArgument(0);
|
||||
if (step.getId() == null) {
|
||||
step.setId((long) stepId.getAndIncrement());
|
||||
}
|
||||
return step;
|
||||
});
|
||||
when(agentStepRepository.findById(any())).thenReturn(Optional.of(new AgentStep()));
|
||||
when(agentStepRepository.findBySessionIdOrderByStepIndex(anyString())).thenReturn(List.of());
|
||||
|
||||
EvaluationService evaluationService = mock(EvaluationService.class);
|
||||
RetrievedDocTracker retrievedDocTracker = mock(RetrievedDocTracker.class);
|
||||
KnowledgeDomainService knowledgeDomainService = mock(KnowledgeDomainService.class);
|
||||
when(knowledgeDomainService.buildKnowledgeMap()).thenReturn("");
|
||||
ToolTraceSummaryService toolTraceSummaryService = mock(ToolTraceSummaryService.class);
|
||||
when(toolTraceSummaryService.buildVerifierTraceSummary(anyString(), anyString())).thenReturn(List.of());
|
||||
SelfEvaluationMergeService selfEvaluationMergeService = mock(SelfEvaluationMergeService.class);
|
||||
when(selfEvaluationMergeService.mergeVerifierEvaluation(any(), any())).thenReturn("{}");
|
||||
|
||||
ReflectionTestUtils.setField(chatService, "dateTimeTools", new DateTimeTools());
|
||||
ReflectionTestUtils.setField(chatService, "lookupKnowledgeTool", new LookupKnowledgeTool());
|
||||
ReflectionTestUtils.setField(chatService, "queryLogsTools", new QueryLogsTools(mock(ToolInvocationRecorder.class)));
|
||||
ReflectionTestUtils.setField(chatService, "diagnosisSessionRepository", diagnosisSessionRepository);
|
||||
ReflectionTestUtils.setField(chatService, "agentStepRepository", agentStepRepository);
|
||||
ReflectionTestUtils.setField(chatService, "evaluationService", evaluationService);
|
||||
ReflectionTestUtils.setField(chatService, "retrievedDocTracker", retrievedDocTracker);
|
||||
ReflectionTestUtils.setField(chatService, "knowledgeDomainService", knowledgeDomainService);
|
||||
ReflectionTestUtils.setField(chatService, "toolTraceSummaryService", toolTraceSummaryService);
|
||||
ReflectionTestUtils.setField(chatService, "selfEvaluationMergeService", selfEvaluationMergeService);
|
||||
ReflectionTestUtils.setField(chatService, "verifierLowConfidenceThreshold", 0.5d);
|
||||
ReflectionTestUtils.setField(chatService, "chatPlannerPrompt", "PLANNER_TEST_PROMPT");
|
||||
ReflectionTestUtils.setField(chatService, "chatExecutorPrompt", "EXECUTOR_TEST_PROMPT");
|
||||
ReflectionTestUtils.setField(chatService, "chatVerifierPrompt", "VERIFIER_TEST_PROMPT");
|
||||
return chatService;
|
||||
}
|
||||
|
||||
private static final class ScriptedChatModel implements ChatModel {
|
||||
private final java.util.ArrayList<String> agentCalls = new java.util.ArrayList<>();
|
||||
private String promptText = "";
|
||||
private boolean sawVerifierPrompt;
|
||||
private final String verifierOutput;
|
||||
|
||||
private ScriptedChatModel() {
|
||||
this("""
|
||||
{
|
||||
"verdict": "PASS",
|
||||
"groundedness_score": 1.0,
|
||||
"critical_fact_count": 1,
|
||||
"facts_checked": [
|
||||
{
|
||||
"fact": "executor answer generated",
|
||||
"is_critical": true,
|
||||
"verification": "direct_evidence",
|
||||
"detail": "covered by scripted verifier",
|
||||
"evidence_refs": []
|
||||
}
|
||||
],
|
||||
"rationale": "scripted pass"
|
||||
}
|
||||
""");
|
||||
}
|
||||
|
||||
private ScriptedChatModel(String verifierOutput) {
|
||||
this.verifierOutput = verifierOutput;
|
||||
}
|
||||
|
||||
@Override
|
||||
public ChatResponse call(Prompt prompt) {
|
||||
promptText = prompt.getContents();
|
||||
String text;
|
||||
if (promptText.contains("PLANNER_TEST_PROMPT")) {
|
||||
agentCalls.add("chat_planner");
|
||||
text = "PLANNER_PLAN";
|
||||
} else if (promptText.contains("EXECUTOR_TEST_PROMPT")) {
|
||||
agentCalls.add("chat_executor");
|
||||
text = "EXECUTOR_FINAL_ANSWER";
|
||||
} else if (promptText.contains("VERIFIER_TEST_PROMPT")) {
|
||||
agentCalls.add("chat_verifier");
|
||||
sawVerifierPrompt = true;
|
||||
text = verifierOutput;
|
||||
} else {
|
||||
text = "UNEXPECTED_PROMPT";
|
||||
}
|
||||
return new ChatResponse(List.of(new Generation(new AssistantMessage(text))));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
package com.superbiz.agent.service;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.superbiz.agent.domain.entity.AgentStep;
|
||||
import com.superbiz.agent.domain.entity.DiagnosisSession;
|
||||
import com.superbiz.agent.domain.entity.ToolInvocation;
|
||||
import com.superbiz.agent.dto.DiagnosisTraceResponse;
|
||||
import com.superbiz.agent.exception.SessionNotFoundException;
|
||||
import com.superbiz.agent.repository.AgentStepRepository;
|
||||
import com.superbiz.agent.repository.DiagnosisSessionRepository;
|
||||
import com.superbiz.agent.repository.ToolInvocationRepository;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.time.LocalDateTime;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
import static org.mockito.Mockito.*;
|
||||
|
||||
class DiagnosisTraceServiceTest {
|
||||
|
||||
private final DiagnosisSessionRepository diagnosisSessionRepository = mock(DiagnosisSessionRepository.class);
|
||||
private final AgentStepRepository agentStepRepository = mock(AgentStepRepository.class);
|
||||
private final ToolInvocationRepository toolInvocationRepository = mock(ToolInvocationRepository.class);
|
||||
private final DiagnosisTraceService service = new DiagnosisTraceService(
|
||||
diagnosisSessionRepository,
|
||||
agentStepRepository,
|
||||
toolInvocationRepository,
|
||||
new ObjectMapper()
|
||||
);
|
||||
|
||||
@Test
|
||||
void getTraceAggregatesSessionStepsAndTools() {
|
||||
String sessionId = "trace-session-001";
|
||||
LocalDateTime now = LocalDateTime.of(2026, 7, 3, 14, 30);
|
||||
DiagnosisSession session = DiagnosisSession.builder()
|
||||
.id(1L)
|
||||
.sessionId(sessionId)
|
||||
.query("payment timeout")
|
||||
.status("SUCCESS")
|
||||
.agentFlow("COMPLEX")
|
||||
.totalDurationMs(1200)
|
||||
.totalTokenCount(300)
|
||||
.stepCount(2)
|
||||
.toolCallCount(1)
|
||||
.answer("restart payment gateway pool")
|
||||
.selfEvaluation("{\"verifier_evaluation\":{\"verdict\":\"PASS\"}}")
|
||||
.feedback("useful")
|
||||
.createdAt(now)
|
||||
.updatedAt(now)
|
||||
.build();
|
||||
AgentStep step = AgentStep.builder()
|
||||
.id(10L)
|
||||
.sessionId(sessionId)
|
||||
.stepIndex(1)
|
||||
.agentName("chat_executor")
|
||||
.modelInput("input")
|
||||
.modelOutput("output")
|
||||
.thought("executor finished")
|
||||
.hasToolCall(true)
|
||||
.durationMs(500)
|
||||
.tokenCount(100)
|
||||
.createdAt(now)
|
||||
.build();
|
||||
ToolInvocation invocation = ToolInvocation.builder()
|
||||
.id(20L)
|
||||
.sessionId(sessionId)
|
||||
.stepId(10L)
|
||||
.toolName("lookup_knowledge")
|
||||
.inputParams("{\"query\":\"ERR_TIMEOUT\"}")
|
||||
.outputPreview("payment timeout doc")
|
||||
.outputLength(19)
|
||||
.retrievalLayer("L0")
|
||||
.l0MatchCount(1)
|
||||
.l1MatchCount(0)
|
||||
.isTruncated(false)
|
||||
.relevanceLevel("HIGHLY_RELEVANT")
|
||||
.dedupReason("FIRST_HIT")
|
||||
.retrievalDetails("{\"documents\":[\"payment-errors.md\"]}")
|
||||
.durationMs(80)
|
||||
.success(true)
|
||||
.createdAt(now)
|
||||
.build();
|
||||
|
||||
when(diagnosisSessionRepository.findBySessionId(sessionId)).thenReturn(Optional.of(session));
|
||||
when(agentStepRepository.findBySessionIdOrderByStepIndex(sessionId)).thenReturn(List.of(step));
|
||||
when(toolInvocationRepository.findBySessionIdOrderByIdAsc(sessionId)).thenReturn(List.of(invocation));
|
||||
|
||||
DiagnosisTraceResponse response = service.getTrace(sessionId);
|
||||
|
||||
assertEquals(sessionId, response.getSession().getSessionId());
|
||||
assertEquals("payment timeout", response.getSession().getQuery());
|
||||
assertEquals("PASS", ((java.util.Map<?, ?>) response.getSession()
|
||||
.getSelfEvaluation()
|
||||
.get("verifier_evaluation")).get("verdict"));
|
||||
assertEquals(1, response.getSteps().size());
|
||||
assertEquals("chat_executor", response.getSteps().get(0).getAgentName());
|
||||
assertEquals(1, response.getToolInvocations().size());
|
||||
assertEquals("ERR_TIMEOUT", response.getToolInvocations().get(0).getInputParams().get("query"));
|
||||
assertEquals(2, response.getSummary().getPersistedStepCount());
|
||||
assertEquals(1, response.getSummary().getReturnedStepCount());
|
||||
assertEquals(1, response.getSummary().getPersistedToolCallCount());
|
||||
assertEquals(1, response.getSummary().getReturnedToolCallCount());
|
||||
assertTrue(response.getSummary().isHasVerifierEvaluation());
|
||||
assertTrue(response.getSummary().isHasFeedback());
|
||||
}
|
||||
|
||||
@Test
|
||||
void getTraceThrowsWhenSessionMissing() {
|
||||
String sessionId = "missing-session";
|
||||
when(diagnosisSessionRepository.findBySessionId(sessionId)).thenReturn(Optional.empty());
|
||||
|
||||
assertThrows(SessionNotFoundException.class, () -> service.getTrace(sessionId));
|
||||
|
||||
verify(diagnosisSessionRepository).findBySessionId(sessionId);
|
||||
verifyNoInteractions(agentStepRepository, toolInvocationRepository);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
package com.superbiz.agent.service;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.springframework.mock.web.MockMultipartFile;
|
||||
import org.springframework.test.util.ReflectionTestUtils;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
class DocumentManagementServiceTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void saveToLocalStoresRelativePathUnderKnowledgeBase() throws Exception {
|
||||
DocumentManagementService service = new DocumentManagementService();
|
||||
ReflectionTestUtils.setField(service, "knowledgeBasePath", tempDir.toString());
|
||||
MockMultipartFile file = new MockMultipartFile(
|
||||
"file",
|
||||
"runbook.md",
|
||||
"text/markdown",
|
||||
"runbook content".getBytes()
|
||||
);
|
||||
|
||||
String storedPath = ReflectionTestUtils.invokeMethod(
|
||||
service,
|
||||
"saveToLocal",
|
||||
file,
|
||||
"runbook.md",
|
||||
"payment"
|
||||
);
|
||||
|
||||
assertEquals("payment/runbook.md", storedPath);
|
||||
assertTrue(Files.exists(tempDir.resolve("payment").resolve("runbook.md")));
|
||||
}
|
||||
}
|
||||
@@ -4,8 +4,6 @@ import com.superbiz.agent.dto.KnowledgeEntry;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.mockito.Mock;
|
||||
import org.mockito.MockitoAnnotations;
|
||||
import org.springframework.test.util.ReflectionTestUtils;
|
||||
|
||||
import java.nio.file.Files;
|
||||
@@ -21,17 +19,13 @@ class KnowledgeIndexServiceTest {
|
||||
|
||||
private KnowledgeIndexService service;
|
||||
|
||||
@Mock
|
||||
private FrontmatterParser frontmatterParser;
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@BeforeEach
|
||||
void setUp() {
|
||||
MockitoAnnotations.openMocks(this);
|
||||
service = new KnowledgeIndexService();
|
||||
ReflectionTestUtils.setField(service, "frontmatterParser", frontmatterParser);
|
||||
ReflectionTestUtils.setField(service, "knowledgeBasePath", tempDir.toString());
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -144,6 +138,30 @@ class KnowledgeIndexServiceTest {
|
||||
assertTrue(result.contains("Test content"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void testReadDocument_relativePathUnderBasePath() throws Exception {
|
||||
Path categoryDir = tempDir.resolve("payment");
|
||||
Files.createDirectories(categoryDir);
|
||||
Path testFile = categoryDir.resolve("relative.md");
|
||||
Files.writeString(testFile, "Relative content");
|
||||
|
||||
String result = service.readDocument("payment/relative.md", 100);
|
||||
|
||||
assertEquals("Relative content", result);
|
||||
}
|
||||
|
||||
@Test
|
||||
void testReadDocument_legacyPathAlreadyContainsBasePath() throws Exception {
|
||||
Path categoryDir = tempDir.resolve("payment");
|
||||
Files.createDirectories(categoryDir);
|
||||
Path testFile = categoryDir.resolve("legacy.md");
|
||||
Files.writeString(testFile, "Legacy content");
|
||||
|
||||
String result = service.readDocument(tempDir.getFileName() + "/payment/legacy.md", 100);
|
||||
|
||||
assertEquals("Legacy content", result);
|
||||
}
|
||||
|
||||
@Test
|
||||
void testReadDocument_exceedsMaxChars() throws Exception {
|
||||
// 创建超长内容
|
||||
|
||||
Reference in New Issue
Block a user