Compare commits
4
Commits
7f2e47ca38
...
a6afbfaa9d
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a6afbfaa9d | ||
|
|
0ee27eb523 | ||
|
|
3b62a8940c | ||
|
|
04eb50e2b4 |
@@ -0,0 +1,13 @@
|
|||||||
|
root = true
|
||||||
|
|
||||||
|
[*]
|
||||||
|
charset = utf-8
|
||||||
|
end_of_line = crlf
|
||||||
|
insert_final_newline = true
|
||||||
|
trim_trailing_whitespace = true
|
||||||
|
|
||||||
|
[*.md]
|
||||||
|
trim_trailing_whitespace = false
|
||||||
|
|
||||||
|
[*.{java,xml,yml,yaml,properties,json,sql,txt,ps1}]
|
||||||
|
charset = utf-8
|
||||||
@@ -5,6 +5,7 @@
|
|||||||
| 日期 | slug | 领域 | 关键词 | 状态 |
|
| 日期 | slug | 领域 | 关键词 | 状态 |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| 2026-07-05 | diagnosis-playbook-skills | Agent Skill/Playbook | read_skill, diagnosis playbook, progressive disclosure, payment timeout, MySQL pool, Redis timeout | openspec/changes/diagnosis-playbook-skills | implemented |
|
| 2026-07-05 | diagnosis-playbook-skills | Agent Skill/Playbook | read_skill, diagnosis playbook, progressive disclosure, payment timeout, MySQL pool, Redis timeout | openspec/changes/diagnosis-playbook-skills | implemented |
|
||||||
|
| 2026-07-07 | executor-evidence-output-contract | Chat质量门禁/证据归因 | Executor structured output, evidence bindings, Verifier structured claims, LOW_CONFID, hallucination | openspec/changes/archive/2026-07-07-executor-evidence-output-contract | archived |
|
||||||
| 2026-07-06 | rag-eval-pipeline-closure | RAG/评测/回归闭环 | lookupResult fixture, LookupKnowledgeTool snapshot, evidenceBlocks, contextPack, retrievalTrace, rerankTrace, baseline diff, fallback case | devflow/projects/2026-07-06-rag-eval-pipeline-closure | archived |
|
| 2026-07-06 | rag-eval-pipeline-closure | RAG/评测/回归闭环 | lookupResult fixture, LookupKnowledgeTool snapshot, evidenceBlocks, contextPack, retrievalTrace, rerankTrace, baseline diff, fallback case | devflow/projects/2026-07-06-rag-eval-pipeline-closure | archived |
|
||||||
| 2026-07-06 | modular-rag-pipeline | RAG/Agent工具/证据链 | modular RAG, lookup_knowledge, evidenceBlocks, contextPack, rerank, retrievalTrace, L0 hint, unfiltered retry | openspec/changes/archive/2026-07-06-modular-rag-pipeline | archived |
|
| 2026-07-06 | modular-rag-pipeline | RAG/Agent工具/证据链 | modular RAG, lookup_knowledge, evidenceBlocks, contextPack, rerank, retrievalTrace, L0 hint, unfiltered retry | openspec/changes/archive/2026-07-06-modular-rag-pipeline | archived |
|
||||||
| 2026-07-05 | mvp-demo-interview-runbook | MVP Demo/Interview | Plan C, payment timeout, runbook, trace checklist, demo script | openspec/changes/archive/2026-07-05-mvp-demo-interview-runbook | archived |
|
| 2026-07-05 | mvp-demo-interview-runbook | MVP Demo/Interview | Plan C, payment timeout, runbook, trace checklist, demo script | openspec/changes/archive/2026-07-05-mvp-demo-interview-runbook | archived |
|
||||||
|
|||||||
@@ -0,0 +1,16 @@
|
|||||||
|
# Acceptance: executor-evidence-output-contract
|
||||||
|
|
||||||
|
## Draft Acceptance
|
||||||
|
|
||||||
|
- [x] Issue exists: `mvp/issues/executor-evidence-attribution-hallucination.md`.
|
||||||
|
- [x] OpenSpec change artifacts exist.
|
||||||
|
- [x] devflow tracking files exist.
|
||||||
|
- [x] OpenSpec validation passes.
|
||||||
|
- [x] Implementation updates Chat Executor prompt.
|
||||||
|
- [x] Implementation passes structured Executor output to Verifier.
|
||||||
|
- [x] Verifier prefers structured claims and still falls back safely.
|
||||||
|
- [x] Focused tests cover parsing, payload assembly, and unsupported confirmed claims.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
This project is currently in proposal/design stage. Runtime code is intentionally not changed yet.
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Brief: executor-evidence-output-contract
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Executor currently returns natural-language diagnosis answers that may mix confirmed evidence, runbook guidance, historical patterns, and unsupported inference. Verifier catches many unsupported facts, but only after extracting claims from prose.
|
||||||
|
|
||||||
|
This project defines a structured Executor evidence-attribution contract and updates the Verifier input/verification path to consume it.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Make Chat Executor output machine-checkable so confirmed claims are explicitly bound to current-session evidence, while hypotheses and evidence gaps remain visibly separate.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
- Chat Executor prompt contract.
|
||||||
|
- Executor structured output parsing.
|
||||||
|
- Verifier payload extension.
|
||||||
|
- Chat Verifier prompt behavior.
|
||||||
|
- Focused tests/eval fixtures.
|
||||||
|
|
||||||
|
## Related OpenSpec
|
||||||
|
|
||||||
|
`openspec/changes/executor-evidence-output-contract/`
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
# Decisions: executor-evidence-output-contract
|
||||||
|
|
||||||
|
## sm-flow Progress
|
||||||
|
|
||||||
|
### Clarify
|
||||||
|
|
||||||
|
Entry summary: recent Chat diagnosis sessions are `LOW_CONFID` because Executor presents unsupported or weakly supported details as confirmed facts after successful tool calls.
|
||||||
|
|
||||||
|
Slug: `executor-evidence-output-contract`
|
||||||
|
|
||||||
|
Scale: standard. This affects prompts, verifier input assembly, parsing behavior, and tests, but does not require a database schema change.
|
||||||
|
|
||||||
|
### Context
|
||||||
|
|
||||||
|
Relevant history:
|
||||||
|
|
||||||
|
- `executor-action-memory-relevance`: Executor already has retrieval quality constraints and should avoid repeated `lookup_knowledge`.
|
||||||
|
- `chat-verifier-agent`: Verifier should not see intermediate reasoning; it receives explicit `original_query`, `executor_final_answer`, `tool_trace_summary`, and `retry_context`.
|
||||||
|
- `evidence-trace-hardening`: evidence-bearing tools persist stable traces and no-evidence semantics.
|
||||||
|
- `modular-rag-pipeline`: `lookup_knowledge` exposes evidence blocks and context packs; L0 hints are not fact evidence.
|
||||||
|
|
||||||
|
Current code shape:
|
||||||
|
|
||||||
|
- `src/main/resources/prompts/chat-executor-prompt.md` is the Chat Executor prompt.
|
||||||
|
- `src/main/resources/prompts/executor-prompt.md` is for the AiOps flow and is not the target of this Chat change.
|
||||||
|
- `VerifierInputHook` currently builds a payload with `original_query`, `executor_final_answer`, `tool_trace_summary`, and `retry_context`.
|
||||||
|
- Verifier prompt currently extracts facts from `executor_final_answer`.
|
||||||
|
|
||||||
|
### Grill
|
||||||
|
|
||||||
|
Question: Should Executor output only JSON or JSON plus readable answer?
|
||||||
|
|
||||||
|
Decision: use one JSON object containing both machine fields and `user_facing_answer`. This avoids losing a readable Chinese answer while giving Verifier structured claims.
|
||||||
|
|
||||||
|
Question: Should evidence binding use `chunk_id`?
|
||||||
|
|
||||||
|
Decision: no. Use generic binding fields because `query_logs` and `query_metrics` do not naturally expose RAG chunks.
|
||||||
|
|
||||||
|
Question: Should Verifier trust Executor-provided claims completely?
|
||||||
|
|
||||||
|
Decision: no. Verifier should verify structured claims first, then scan `user_facing_answer` for extra confirmed-sounding facts omitted from `claims`.
|
||||||
|
|
||||||
|
Question: What happens when Executor JSON is malformed?
|
||||||
|
|
||||||
|
Decision: preserve raw final answer, mark parse failure, and fall back to existing natural-language verification.
|
||||||
|
|
||||||
|
### Specify
|
||||||
|
|
||||||
|
OpenSpec artifacts:
|
||||||
|
|
||||||
|
- `proposal.md`: why and scope
|
||||||
|
- `design.md`: contract, verifier behavior, risks
|
||||||
|
- `specs/chat-verifier-agent/spec.md`: modified and added requirements
|
||||||
|
- `tasks.md`: implementation checklist
|
||||||
|
|
||||||
|
### Audit
|
||||||
|
|
||||||
|
Cross-artifact alignment:
|
||||||
|
|
||||||
|
- Issue describes evidence attribution hallucination.
|
||||||
|
- Proposal scopes the fix to Executor output and Verifier consumption.
|
||||||
|
- Design preserves existing verifier isolation.
|
||||||
|
- Spec adds observable behavior without changing database schema.
|
||||||
|
- Tasks remain implementation-oriented and unchecked.
|
||||||
|
|
||||||
|
Interface impact:
|
||||||
|
|
||||||
|
- Prompt/output contract: L2 internal Agent contract change.
|
||||||
|
- Verifier payload: L2 internal structured input extension.
|
||||||
|
- Database schema: no change.
|
||||||
|
- External HTTP API: no intended change.
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# Evidence: executor-evidence-output-contract
|
||||||
|
|
||||||
|
## Repository Evidence
|
||||||
|
|
||||||
|
- `chat-executor-prompt.md` currently requires using real tool data but does not require a structured evidence-attribution output.
|
||||||
|
- `chat-verifier-prompt.md` currently extracts facts from `executor_final_answer` prose and compares them with `tool_trace_summary`.
|
||||||
|
- `VerifierInputHook` currently provides `original_query`, `executor_final_answer`, `tool_trace_summary`, and `retry_context`.
|
||||||
|
- `openspec/specs/chat-verifier-agent/spec.md` already requires explicit verifier inputs, auditable evidence refs, fixed verdicts, and low-confidence handling.
|
||||||
|
- `openspec/specs/evidence-trace-hardening/spec.md` already distinguishes failed, no-hit, deduped, and successful evidence-tool traces.
|
||||||
|
|
||||||
|
## Runtime Evidence From Recent Sessions
|
||||||
|
|
||||||
|
Recent MySQL inspection showed repeated `LOW_CONFID` verifier results with many `no_evidence` facts. Typical unsupported claims included OOM, Full GC frequency, specific slow SQL timings, lock waits, and service-specific timeout details that were not supported by current-session tool traces.
|
||||||
|
|
||||||
|
## Design Evidence
|
||||||
|
|
||||||
|
This change preserves the previous design that Verifier should not inspect intermediate reasoning. The new structured output is still final Executor output, not hidden chain-of-thought.
|
||||||
@@ -58,6 +58,7 @@ trace fixture 是一次 Agent 运行后的“留痕快照”。评测器不会
|
|||||||
| `session.answer` | Agent 最终给用户的回答 | 用来检查根因关键词和禁用词 |
|
| `session.answer` | Agent 最终给用户的回答 | 用来检查根因关键词和禁用词 |
|
||||||
| `session.totalDurationMs` | 这次运行耗时 | 进入报告,帮助观察性能是否明显变差 |
|
| `session.totalDurationMs` | 这次运行耗时 | 进入报告,帮助观察性能是否明显变差 |
|
||||||
| `session.selfEvaluation.verifier_evaluation.verdict` | Verifier 对最终回答的判断 | 必须存在,并且要落在 case 的 `allowedVerdicts` 里 |
|
| `session.selfEvaluation.verifier_evaluation.verdict` | Verifier 对最终回答的判断 | 必须存在,并且要落在 case 的 `allowedVerdicts` 里 |
|
||||||
|
| `session.selfEvaluation.verifier_evaluation.executor_structured_output.claims[*].evidence_bindings` | Executor 结构化输出里的确认事实证据绑定 | 如果 trace 中存在结构化 Executor 输出,每条 confirmed claim 必须有证据绑定 |
|
||||||
| `session.selfEvaluation.verifier_evaluation.tool_trace_summary[*].tool_name` | Verifier 总结里看到的工具证据 | 用来补充判断证据工具是否出现 |
|
| `session.selfEvaluation.verifier_evaluation.tool_trace_summary[*].tool_name` | Verifier 总结里看到的工具证据 | 用来补充判断证据工具是否出现 |
|
||||||
| `toolInvocations[*].toolName` | Agent 实际调用过的工具名 | 用来检查 `requiredEvidenceTools` 是否满足 |
|
| `toolInvocations[*].toolName` | Agent 实际调用过的工具名 | 用来检查 `requiredEvidenceTools` 是否满足 |
|
||||||
| `toolInvocations[*].success` | 工具调用是否成功 | 当前主要保留在 trace 里,后续可以升级成更严格的成功率检查 |
|
| `toolInvocations[*].success` | 工具调用是否成功 | 当前主要保留在 trace 里,后续可以升级成更严格的成功率检查 |
|
||||||
|
|||||||
@@ -0,0 +1,161 @@
|
|||||||
|
## Context
|
||||||
|
|
||||||
|
Current Chat flow explicitly runs `Planner -> Executor -> Verifier`. `VerifierInputHook` builds a structured verifier payload containing:
|
||||||
|
|
||||||
|
- `original_query`
|
||||||
|
- `executor_final_answer`
|
||||||
|
- `tool_trace_summary`
|
||||||
|
- `retry_context`
|
||||||
|
|
||||||
|
This satisfies the earlier design goal that Verifier should not see Planner/Executor intermediate reasoning. However, Executor's answer is still natural language. Verifier must infer claims from prose, then compare those claims to tool traces. Recent low-confidence sessions show that Executor often converts weak or unrelated evidence into confirmed conclusions before Verifier sees it.
|
||||||
|
|
||||||
|
The fix should move evidence attribution earlier: Executor must state which claims are confirmed, which evidence supports them, and which points remain hypotheses or gaps.
|
||||||
|
|
||||||
|
## Goals / Non-Goals
|
||||||
|
|
||||||
|
**Goals:**
|
||||||
|
|
||||||
|
- Make Executor final output machine-checkable.
|
||||||
|
- Separate confirmed claims, hypotheses, recommended actions, and missing information.
|
||||||
|
- Make every confirmed claim bind to concrete tool evidence identifiers or excerpts.
|
||||||
|
- Let Verifier consume structured claims directly instead of reconstructing all facts from prose.
|
||||||
|
- Preserve Verifier input isolation from intermediate reasoning.
|
||||||
|
- Preserve existing behavior when Executor output is malformed by falling back to natural-language verification and low-confidence handling.
|
||||||
|
|
||||||
|
**Non-Goals:**
|
||||||
|
|
||||||
|
- Do not add database tables or columns.
|
||||||
|
- Do not change evidence tool method signatures.
|
||||||
|
- Do not let Verifier call tools.
|
||||||
|
- Do not require full raw tool outputs in verifier input.
|
||||||
|
- Do not make user-facing answers become raw JSON-only unless the product layer explicitly chooses that rendering.
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
### Decision: Executor emits an evidence-attribution contract
|
||||||
|
|
||||||
|
Executor final output SHALL be a single JSON object. The recommended contract is:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"answer_version": "executor_evidence_v1",
|
||||||
|
"diagnosis_summary": "1-2 sentence summary using only supported facts",
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"claim_id": "claim-1",
|
||||||
|
"claim_type": "root_cause",
|
||||||
|
"claim_text": "The payment-service connection pool is saturated",
|
||||||
|
"support_level": "direct",
|
||||||
|
"evidence_bindings": [
|
||||||
|
{
|
||||||
|
"source_type": "tool_trace",
|
||||||
|
"source_id": "trace-1",
|
||||||
|
"tool_name": "query_metrics",
|
||||||
|
"source_invocation_ids": [101],
|
||||||
|
"evidence_excerpt": "active connections reached max pool size"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"hypotheses": [
|
||||||
|
{
|
||||||
|
"hypothesis_text": "A connection leak may be contributing",
|
||||||
|
"basis": "metrics show saturation, but no leak evidence was returned",
|
||||||
|
"needed_evidence": ["connection lifetime metrics", "leak detection logs"]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"recommended_actions": [
|
||||||
|
{
|
||||||
|
"action_text": "Check HikariCP active/idle/pending connection metrics",
|
||||||
|
"reason": "Needed to confirm pool saturation scope",
|
||||||
|
"evidence_bindings": []
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"missing_info": [
|
||||||
|
"No log evidence confirming a connection leak was returned"
|
||||||
|
],
|
||||||
|
"user_facing_answer": "Chinese answer rendered from the same confirmed claims, hypotheses, actions, and gaps"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Rationale: this keeps the machine contract explicit while still allowing the product to return a readable Chinese answer.
|
||||||
|
|
||||||
|
### Decision: Evidence binding uses generic source fields
|
||||||
|
|
||||||
|
`chunk_id` alone is too specific to `lookup_knowledge`. The binding shape SHALL support all evidence-bearing tools:
|
||||||
|
|
||||||
|
- `source_type`: `tool_trace`, `lookup_evidence_block`, or another stable source family
|
||||||
|
- `source_id`: `tool_trace_summary.trace_ref`, evidence block id, or equivalent stable id
|
||||||
|
- `tool_name`: evidence tool name when available
|
||||||
|
- `source_invocation_ids`: persisted `tool_invocation` ids when available
|
||||||
|
- `evidence_excerpt`: bounded excerpt copied or summarized from tool evidence
|
||||||
|
|
||||||
|
Rationale: `query_logs`, `query_metrics`, and `lookup_knowledge` expose evidence differently. A generic binding prevents the prompt contract from overfitting to RAG chunks.
|
||||||
|
|
||||||
|
### Decision: Confirmed claims are stricter than hypotheses and actions
|
||||||
|
|
||||||
|
Confirmed `claims` SHALL contain only facts supported by current-session evidence. Runbook instructions, skill guidance, and historical case patterns SHALL NOT appear as confirmed claims unless the current tool trace supports that exact fact.
|
||||||
|
|
||||||
|
Unsupported but useful diagnostic ideas SHALL be placed under `hypotheses` or `recommended_actions`.
|
||||||
|
|
||||||
|
Rationale: this directly addresses the observed hallucination: turning plausible patterns into current incident facts.
|
||||||
|
|
||||||
|
### Decision: Verifier prefers structured claims but keeps fallback
|
||||||
|
|
||||||
|
Verifier prompt SHALL use this order:
|
||||||
|
|
||||||
|
1. If `executor_structured_output.claims` is valid, verify each claim directly.
|
||||||
|
2. Also scan `user_facing_answer` for extra confirmed-sounding facts not present in `claims`; mark them as facts to check.
|
||||||
|
3. If the structured output is missing or invalid, fall back to the existing natural-language extraction from `executor_final_answer`.
|
||||||
|
|
||||||
|
Invalid structured output SHALL NOT crash the flow. It SHOULD produce a fallback verifier decision and make malformed structure visible in `verifier_evaluation`.
|
||||||
|
|
||||||
|
Rationale: the new contract should improve precision without making runtime brittle.
|
||||||
|
|
||||||
|
### Decision: Verifier still does not see intermediate reasoning
|
||||||
|
|
||||||
|
The verifier payload MAY add:
|
||||||
|
|
||||||
|
- `executor_structured_output`
|
||||||
|
- `executor_output_parse_status`
|
||||||
|
|
||||||
|
It SHALL continue to exclude Planner reasoning, Executor intermediate reasoning, and raw conversation noise.
|
||||||
|
|
||||||
|
Rationale: this preserves the original verifier design: verify the final answer and tool traces, not hidden reasoning.
|
||||||
|
|
||||||
|
### Decision: User-facing output remains Chinese
|
||||||
|
|
||||||
|
For Chat user-facing pages and answers, the rendered answer SHALL be Chinese. If the Executor emits JSON, either:
|
||||||
|
|
||||||
|
- `user_facing_answer` is returned to the user after verifier routing, or
|
||||||
|
- ChatService renders a Chinese answer from the structured contract.
|
||||||
|
|
||||||
|
The raw machine contract may remain visible in trace/debug views, but the normal user answer should not become an English/JSON-only artifact.
|
||||||
|
|
||||||
|
Rationale: this aligns with current UI/product language requirements while keeping machine-checkable evidence.
|
||||||
|
|
||||||
|
## Risks / Trade-offs
|
||||||
|
|
||||||
|
- [Risk] LLM may emit malformed JSON. Mitigation: parse-status fallback and verifier natural-language fallback.
|
||||||
|
- [Risk] Executor may put unsupported facts in `user_facing_answer` but omit them from `claims`. Mitigation: Verifier scans `user_facing_answer` for extra confirmed-sounding facts.
|
||||||
|
- [Risk] Evidence excerpts may be fabricated. Mitigation: Verifier checks binding ids against `tool_trace_summary` and treats missing refs as no evidence.
|
||||||
|
- [Risk] More verbose output increases tokens. Mitigation: keep excerpts bounded and put detailed raw evidence only in tool traces.
|
||||||
|
- [Risk] Prompt-only enforcement is soft. Mitigation: add focused tests/eval fixtures and later consider code-level schema validation.
|
||||||
|
|
||||||
|
## Migration Plan
|
||||||
|
|
||||||
|
1. Update `chat-executor-prompt.md` with the evidence-attribution JSON contract.
|
||||||
|
2. Add parsing in Chat runtime for Executor output:
|
||||||
|
- valid JSON -> preserve parsed `executor_structured_output`
|
||||||
|
- invalid JSON -> mark parse status and keep raw `executor_final_answer`
|
||||||
|
3. Update verifier payload assembly to include `executor_structured_output` and parse status.
|
||||||
|
4. Update `chat-verifier-prompt.md` to prefer structured claims and verify extra facts in `user_facing_answer`.
|
||||||
|
5. Persist parsed or raw structured output in existing trace/evaluation snapshots without schema changes.
|
||||||
|
6. Add focused tests and regression fixtures for unsupported confirmed claims.
|
||||||
|
7. Validate OpenSpec and run targeted tests.
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
- Should `user_facing_answer` be mandatory in Executor JSON, or should ChatService render it from structured fields?
|
||||||
|
- Should malformed Executor JSON force `LOW_CONFID`, or should Verifier decide based on natural-language fallback alone?
|
||||||
|
- Should `support_level` allow only `direct`, `indirect`, and `none`, or should it also include `contradicted` for Executor self-reporting?
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# Proposal: executor-evidence-output-contract
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
Recent Chat diagnosis sessions frequently end as `LOW_CONFID` even though evidence tools were called successfully. The main failure mode is not missing tool execution; it is that Executor blends tool evidence, runbook/reference patterns, and model inference into one natural-language answer, then presents unsupported details as confirmed incident facts.
|
||||||
|
|
||||||
|
Verifier currently receives `original_query`, `executor_final_answer`, `tool_trace_summary`, and optional `retry_context`. It can catch unsupported claims, but it must first infer facts from unstructured prose. That makes the quality gate reactive and noisy: unsupported claims are detected after the answer has already been shaped as a confident story.
|
||||||
|
|
||||||
|
## What Changes
|
||||||
|
|
||||||
|
- Update the Chat Executor prompt so its final output follows a strict evidence-attribution JSON contract.
|
||||||
|
- Require confirmed claims to carry explicit evidence bindings and require unsupported items to be placed in `hypotheses`, `missing_info`, or `recommended_actions` instead of confirmed conclusions.
|
||||||
|
- Extend the verifier input contract with a parsed or raw `executor_structured_output` field while preserving the existing `executor_final_answer` field for fallback compatibility.
|
||||||
|
- Update the Chat Verifier prompt so it prefers Executor-provided structured claims over natural-language fact extraction.
|
||||||
|
- Keep Verifier isolated from Planner/Executor intermediate reasoning; it still receives only the original query, Executor final output, structured claim contract, tool trace summary, and retry context.
|
||||||
|
- Do not change evidence tool signatures, database schema, or the `Planner -> Executor -> Verifier` topology.
|
||||||
|
|
||||||
|
## Capabilities
|
||||||
|
|
||||||
|
### New Capabilities
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
### Modified Capabilities
|
||||||
|
|
||||||
|
- `chat-verifier-agent`: Chat verification now consumes an Executor evidence-attribution contract when available.
|
||||||
|
|
||||||
|
## Impact
|
||||||
|
|
||||||
|
- Affected prompts: `chat-executor-prompt.md`, `chat-verifier-prompt.md`.
|
||||||
|
- Affected integration: `VerifierInputHook` or equivalent verifier payload assembly must include `executor_structured_output` when Executor returns valid structured JSON.
|
||||||
|
- Affected parsing: `ChatService` may parse Executor output to separate machine contract from user-facing text, with fallback when parsing fails.
|
||||||
|
- Affected tests/eval: prompt contract tests, verifier input assembly tests, and unsupported-claim regression fixtures.
|
||||||
|
- No database schema change is required. Structured Executor output can be persisted in existing step output fields and verifier snapshots.
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- Loosening Verifier scoring or `PASS` criteria.
|
||||||
|
- Raising `verifier.low-confidence-threshold` to hide unsupported claims.
|
||||||
|
- Treating runbook, skill, or historical case text as current incident evidence unless it was returned as evidence for the current query and bound explicitly.
|
||||||
|
- Adding a new verifier tool or allowing Verifier to perform retrieval.
|
||||||
|
- Replacing the existing tool trace summary contract.
|
||||||
+95
@@ -0,0 +1,95 @@
|
|||||||
|
## MODIFIED Requirements
|
||||||
|
|
||||||
|
### Requirement: Verifier SHALL consume explicit verification inputs
|
||||||
|
The Verifier SHALL receive explicit verification inputs rather than inferring them only from raw conversation history.
|
||||||
|
|
||||||
|
#### Scenario: explicit input blocks available to Verifier
|
||||||
|
- **WHEN** the Verifier starts
|
||||||
|
- **THEN** the system SHALL provide `original_query`, `executor_final_answer`, and `tool_trace_summary` as explicit inputs
|
||||||
|
- **AND** `retry_context` SHALL be provided on the second round only
|
||||||
|
- **AND** when Executor returns a valid evidence-attribution contract, the system SHALL provide `executor_structured_output`
|
||||||
|
- **AND** when Executor output parsing fails, the system SHALL provide an `executor_output_parse_status` that indicates the failure
|
||||||
|
- **AND** message filtering MAY be used only to remove intermediate reasoning or unrelated noise
|
||||||
|
|
||||||
|
#### Scenario: Verifier remains isolated from intermediate reasoning
|
||||||
|
- **WHEN** `executor_structured_output` is added to the verifier input
|
||||||
|
- **THEN** the input SHALL still exclude Planner reasoning and Executor intermediate reasoning
|
||||||
|
- **AND** the input SHALL be limited to the original query, final Executor output, parsed Executor evidence contract, tool trace summary, and retry context
|
||||||
|
|
||||||
|
### Requirement: Verifier SHALL fact-check Executor answers
|
||||||
|
The system SHALL have a Verifier Agent that reads the Executor's answer and the tool call history, then produces a structured verdict.
|
||||||
|
|
||||||
|
#### Scenario: Structured Executor claims are verified first
|
||||||
|
- **WHEN** `executor_structured_output.claims` is present and valid
|
||||||
|
- **THEN** Verifier SHALL verify each structured claim against `tool_trace_summary`
|
||||||
|
- **AND** each claim's evidence bindings SHALL reference existing trace or invocation identifiers when those identifiers are available
|
||||||
|
- **AND** a claim with fabricated or missing evidence references SHALL NOT be classified as `direct_evidence`
|
||||||
|
|
||||||
|
#### Scenario: Extra confirmed-sounding answer facts are still checked
|
||||||
|
- **WHEN** `executor_structured_output.user_facing_answer` contains confirmed-sounding facts that are absent from `executor_structured_output.claims`
|
||||||
|
- **THEN** Verifier SHALL add those facts to `facts_checked`
|
||||||
|
- **AND** unsupported extra facts SHALL lower the verdict according to the existing verdict matrix
|
||||||
|
|
||||||
|
#### Scenario: Natural-language fallback remains available
|
||||||
|
- **WHEN** Executor does not return parseable structured output
|
||||||
|
- **THEN** Verifier SHALL fall back to extracting facts from `executor_final_answer`
|
||||||
|
- **AND** the final verdict SHALL still follow the existing groundedness and evidence classification rules
|
||||||
|
|
||||||
|
## ADDED Requirements
|
||||||
|
|
||||||
|
### Requirement: Executor SHALL output an evidence-attribution contract
|
||||||
|
The Chat Executor SHALL produce a machine-checkable final output that separates confirmed claims from hypotheses, recommendations, and missing information.
|
||||||
|
|
||||||
|
#### Scenario: Executor final output contains required top-level fields
|
||||||
|
- **WHEN** Executor completes a Chat diagnosis step
|
||||||
|
- **THEN** its final output SHALL contain `answer_version`, `diagnosis_summary`, `claims`, `hypotheses`, `recommended_actions`, `missing_info`, and `user_facing_answer`
|
||||||
|
- **AND** the output SHOULD be parseable as one JSON object without Markdown fences
|
||||||
|
|
||||||
|
#### Scenario: Confirmed claims carry evidence bindings
|
||||||
|
- **WHEN** Executor emits an item under `claims`
|
||||||
|
- **THEN** the item SHALL include `claim_id`, `claim_type`, `claim_text`, `support_level`, and `evidence_bindings`
|
||||||
|
- **AND** `support_level` SHALL be one of `direct`, `indirect`, or `none`
|
||||||
|
- **AND** claims with `support_level=direct` or `support_level=indirect` SHALL include at least one evidence binding
|
||||||
|
|
||||||
|
#### Scenario: Evidence bindings support multiple tool types
|
||||||
|
- **WHEN** Executor binds evidence to a claim
|
||||||
|
- **THEN** each binding SHALL include `source_type`, `source_id`, `tool_name`, `source_invocation_ids`, and `evidence_excerpt`
|
||||||
|
- **AND** the binding SHALL be able to reference `lookup_knowledge`, `query_logs`, `query_metrics`, or other evidence-bearing tool traces
|
||||||
|
- **AND** the binding SHALL NOT rely only on a RAG-specific `chunk_id`
|
||||||
|
|
||||||
|
#### Scenario: Unsupported conclusions are not confirmed claims
|
||||||
|
- **WHEN** a possible root cause, detail, or remediation lacks current-session tool evidence
|
||||||
|
- **THEN** Executor SHALL place it under `hypotheses`, `recommended_actions`, or `missing_info`
|
||||||
|
- **AND** Executor SHALL NOT present it as a confirmed claim
|
||||||
|
|
||||||
|
#### Scenario: Runbook and skill guidance do not become incident facts
|
||||||
|
- **WHEN** Executor uses runbook, skill, or historical-case guidance
|
||||||
|
- **THEN** the guidance MAY influence `recommended_actions`
|
||||||
|
- **AND** the guidance SHALL NOT be emitted as a current incident fact unless current-session tool evidence supports it
|
||||||
|
|
||||||
|
### Requirement: User-facing Chat answers SHALL remain readable Chinese
|
||||||
|
The system SHALL preserve a readable Chinese answer for normal Chat users even when Executor emits a machine-checkable contract.
|
||||||
|
|
||||||
|
#### Scenario: User-facing answer is available
|
||||||
|
- **WHEN** Executor emits structured output
|
||||||
|
- **THEN** `user_facing_answer` SHALL be written in Chinese
|
||||||
|
- **AND** it SHALL be consistent with the confirmed claims, hypotheses, recommended actions, and missing information in the same JSON object
|
||||||
|
|
||||||
|
#### Scenario: Machine contract remains available for trace inspection
|
||||||
|
- **WHEN** the Chat trace or verifier evaluation is inspected
|
||||||
|
- **THEN** the structured Executor contract MAY be shown for debugging or audit
|
||||||
|
- **AND** normal user output SHALL use the existing verifier-routed display path rather than exposing raw JSON by default
|
||||||
|
|
||||||
|
### Requirement: Structured Executor output SHALL degrade safely
|
||||||
|
The system SHALL tolerate malformed or absent structured Executor output without crashing the Chat flow.
|
||||||
|
|
||||||
|
#### Scenario: Malformed Executor JSON is captured
|
||||||
|
- **WHEN** Executor returns malformed JSON or text outside the expected contract
|
||||||
|
- **THEN** Chat runtime SHALL preserve the raw `executor_final_answer`
|
||||||
|
- **AND** it SHALL mark `executor_output_parse_status` as failed
|
||||||
|
- **AND** Verifier SHALL use natural-language fallback behavior
|
||||||
|
|
||||||
|
#### Scenario: Structured parse failure remains observable
|
||||||
|
- **WHEN** Executor output parsing fails
|
||||||
|
- **THEN** the verifier evaluation or trace snapshot SHALL make the parse failure visible
|
||||||
|
- **AND** the failure SHALL NOT be silently treated as a successful evidence-attribution contract
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
## 1. Executor Contract
|
||||||
|
|
||||||
|
- [x] 1.1 Update `chat-executor-prompt.md` with the evidence-attribution JSON contract.
|
||||||
|
- [x] 1.2 Require confirmed claims, hypotheses, recommended actions, missing information, and Chinese `user_facing_answer`.
|
||||||
|
- [x] 1.3 Add prompt-level constraints that runbook/skill/history guidance cannot become current incident facts without current-session evidence.
|
||||||
|
|
||||||
|
## 2. Runtime Parsing And Verifier Input
|
||||||
|
|
||||||
|
- [x] 2.1 Parse Executor final output into `executor_structured_output` when it is valid JSON.
|
||||||
|
- [x] 2.2 Add `executor_output_parse_status` for valid, missing, and malformed outputs.
|
||||||
|
- [x] 2.3 Pass `executor_structured_output` and parse status into the verifier payload while preserving existing `executor_final_answer`.
|
||||||
|
- [x] 2.4 Persist or expose the structured output through existing trace/evaluation snapshots without a schema change.
|
||||||
|
|
||||||
|
## 3. Verifier Prompt
|
||||||
|
|
||||||
|
- [x] 3.1 Update `chat-verifier-prompt.md` to verify structured claims before natural-language extraction.
|
||||||
|
- [x] 3.2 Add a verifier rule to scan `user_facing_answer` for extra confirmed-sounding facts not present in structured `claims`.
|
||||||
|
- [x] 3.3 Keep the existing natural-language fallback for malformed or absent structured output.
|
||||||
|
|
||||||
|
## 4. Tests And Evaluation
|
||||||
|
|
||||||
|
- [x] 4.1 Add focused tests for Executor output parsing and parse-status fallback.
|
||||||
|
- [x] 4.2 Add focused tests for verifier payload assembly with structured Executor output.
|
||||||
|
- [x] 4.3 Add prompt/fixture regression coverage for unsupported claims being placed outside confirmed `claims`.
|
||||||
|
- [x] 4.4 Add or update eval fixture checks that confirmed claims must have evidence bindings.
|
||||||
|
|
||||||
|
## 5. Verification
|
||||||
|
|
||||||
|
- [x] 5.1 Run targeted unit tests for the parsing and verifier-input changes.
|
||||||
|
- [x] 5.2 Run relevant Chat verifier/eval regression tests.
|
||||||
|
- [x] 5.3 Run OpenSpec validation for `executor-evidence-output-contract`.
|
||||||
@@ -27,6 +27,22 @@ The system SHALL have a Verifier Agent that reads the Executor's answer and the
|
|||||||
- **OR** the answer fabricates a key entity, error code, or conclusion that does not exist in the tool evidence
|
- **OR** the answer fabricates a key entity, error code, or conclusion that does not exist in the tool evidence
|
||||||
- **THEN** the Verifier SHALL output verdict="REJECT"
|
- **THEN** the Verifier SHALL output verdict="REJECT"
|
||||||
|
|
||||||
|
#### Scenario: Structured Executor claims are verified first
|
||||||
|
- **WHEN** `executor_structured_output.claims` is present and valid
|
||||||
|
- **THEN** Verifier SHALL verify each structured claim against `tool_trace_summary`
|
||||||
|
- **AND** each claim's evidence bindings SHALL reference existing trace or invocation identifiers when those identifiers are available
|
||||||
|
- **AND** a claim with fabricated or missing evidence references SHALL NOT be classified as `direct_evidence`
|
||||||
|
|
||||||
|
#### Scenario: Extra confirmed-sounding answer facts are still checked
|
||||||
|
- **WHEN** `executor_structured_output.user_facing_answer` contains confirmed-sounding facts that are absent from `executor_structured_output.claims`
|
||||||
|
- **THEN** Verifier SHALL add those facts to `facts_checked`
|
||||||
|
- **AND** unsupported extra facts SHALL lower the verdict according to the existing verdict matrix
|
||||||
|
|
||||||
|
#### Scenario: Natural-language fallback remains available
|
||||||
|
- **WHEN** Executor does not return parseable structured output
|
||||||
|
- **THEN** Verifier SHALL fall back to extracting facts from `executor_final_answer`
|
||||||
|
- **AND** the final verdict SHALL still follow the existing groundedness and evidence classification rules
|
||||||
|
|
||||||
### Requirement: Verifier SHALL output structured JSON
|
### Requirement: Verifier SHALL output structured JSON
|
||||||
The Verifier SHALL output a JSON object with verdict, groundedness_score, facts_checked array, and rationale.
|
The Verifier SHALL output a JSON object with verdict, groundedness_score, facts_checked array, and rationale.
|
||||||
|
|
||||||
@@ -138,8 +154,15 @@ The Verifier SHALL receive explicit verification inputs rather than inferring th
|
|||||||
- **WHEN** the Verifier starts
|
- **WHEN** the Verifier starts
|
||||||
- **THEN** the system SHALL provide `original_query`, `executor_final_answer`, and `tool_trace_summary` as explicit inputs
|
- **THEN** the system SHALL provide `original_query`, `executor_final_answer`, and `tool_trace_summary` as explicit inputs
|
||||||
- **AND** `retry_context` SHALL be provided on the second round only
|
- **AND** `retry_context` SHALL be provided on the second round only
|
||||||
|
- **AND** when Executor returns a valid evidence-attribution contract, the system SHALL provide `executor_structured_output`
|
||||||
|
- **AND** when Executor output parsing fails, the system SHALL provide an `executor_output_parse_status` that indicates the failure
|
||||||
- **AND** message filtering MAY be used only to remove intermediate reasoning or unrelated noise
|
- **AND** message filtering MAY be used only to remove intermediate reasoning or unrelated noise
|
||||||
|
|
||||||
|
#### Scenario: Verifier remains isolated from intermediate reasoning
|
||||||
|
- **WHEN** `executor_structured_output` is added to the verifier input
|
||||||
|
- **THEN** the input SHALL still exclude Planner reasoning and Executor intermediate reasoning
|
||||||
|
- **AND** the input SHALL be limited to the original query, final Executor output, parsed Executor evidence contract, tool trace summary, and retry context
|
||||||
|
|
||||||
#### Scenario: tool trace summary derived from tool facts
|
#### Scenario: tool trace summary derived from tool facts
|
||||||
- **WHEN** the system prepares verifier inputs
|
- **WHEN** the system prepares verifier inputs
|
||||||
- **THEN** `tool_trace_summary` SHALL be generated from tool invocation facts
|
- **THEN** `tool_trace_summary` SHALL be generated from tool invocation facts
|
||||||
@@ -230,3 +253,59 @@ When Verifier returns `LOW_CONFID`, user-facing output SHALL clearly separate co
|
|||||||
- **THEN** the final user-facing response SHALL identify those gaps from verifier output
|
- **THEN** the final user-facing response SHALL identify those gaps from verifier output
|
||||||
- **AND** unsupported raw Executor claims SHALL NOT be presented as confirmed conclusions
|
- **AND** unsupported raw Executor claims SHALL NOT be presented as confirmed conclusions
|
||||||
|
|
||||||
|
### Requirement: Executor SHALL output an evidence-attribution contract
|
||||||
|
The Chat Executor SHALL produce a machine-checkable final output that separates confirmed claims from hypotheses, recommendations, and missing information.
|
||||||
|
|
||||||
|
#### Scenario: Executor final output contains required top-level fields
|
||||||
|
- **WHEN** Executor completes a Chat diagnosis step
|
||||||
|
- **THEN** its final output SHALL contain `answer_version`, `diagnosis_summary`, `claims`, `hypotheses`, `recommended_actions`, `missing_info`, and `user_facing_answer`
|
||||||
|
- **AND** the output SHOULD be parseable as one JSON object without Markdown fences
|
||||||
|
|
||||||
|
#### Scenario: Confirmed claims carry evidence bindings
|
||||||
|
- **WHEN** Executor emits an item under `claims`
|
||||||
|
- **THEN** the item SHALL include `claim_id`, `claim_type`, `claim_text`, `support_level`, and `evidence_bindings`
|
||||||
|
- **AND** `support_level` SHALL be one of `direct`, `indirect`, or `none`
|
||||||
|
- **AND** claims with `support_level=direct` or `support_level=indirect` SHALL include at least one evidence binding
|
||||||
|
|
||||||
|
#### Scenario: Evidence bindings support multiple tool types
|
||||||
|
- **WHEN** Executor binds evidence to a claim
|
||||||
|
- **THEN** each binding SHALL include `source_type`, `source_id`, `tool_name`, `source_invocation_ids`, and `evidence_excerpt`
|
||||||
|
- **AND** the binding SHALL be able to reference `lookup_knowledge`, `query_logs`, `query_metrics`, or other evidence-bearing tool traces
|
||||||
|
- **AND** the binding SHALL NOT rely only on a RAG-specific `chunk_id`
|
||||||
|
|
||||||
|
#### Scenario: Unsupported conclusions are not confirmed claims
|
||||||
|
- **WHEN** a possible root cause, detail, or remediation lacks current-session tool evidence
|
||||||
|
- **THEN** Executor SHALL place it under `hypotheses`, `recommended_actions`, or `missing_info`
|
||||||
|
- **AND** Executor SHALL NOT present it as a confirmed claim
|
||||||
|
|
||||||
|
#### Scenario: Runbook and skill guidance do not become incident facts
|
||||||
|
- **WHEN** Executor uses runbook, skill, or historical-case guidance
|
||||||
|
- **THEN** the guidance MAY influence `recommended_actions`
|
||||||
|
- **AND** the guidance SHALL NOT be emitted as a current incident fact unless current-session tool evidence supports it
|
||||||
|
|
||||||
|
### Requirement: User-facing Chat answers SHALL remain readable Chinese
|
||||||
|
The system SHALL preserve a readable Chinese answer for normal Chat users even when Executor emits a machine-checkable contract.
|
||||||
|
|
||||||
|
#### Scenario: User-facing answer is available
|
||||||
|
- **WHEN** Executor emits structured output
|
||||||
|
- **THEN** `user_facing_answer` SHALL be written in Chinese
|
||||||
|
- **AND** it SHALL be consistent with the confirmed claims, hypotheses, recommended actions, and missing information in the same JSON object
|
||||||
|
|
||||||
|
#### Scenario: Machine contract remains available for trace inspection
|
||||||
|
- **WHEN** the Chat trace or verifier evaluation is inspected
|
||||||
|
- **THEN** the structured Executor contract MAY be shown for debugging or audit
|
||||||
|
- **AND** normal user output SHALL use the existing verifier-routed display path rather than exposing raw JSON by default
|
||||||
|
|
||||||
|
### Requirement: Structured Executor output SHALL degrade safely
|
||||||
|
The system SHALL tolerate malformed or absent structured Executor output without crashing the Chat flow.
|
||||||
|
|
||||||
|
#### Scenario: Malformed Executor JSON is captured
|
||||||
|
- **WHEN** Executor returns malformed JSON or text outside the expected contract
|
||||||
|
- **THEN** Chat runtime SHALL preserve the raw `executor_final_answer`
|
||||||
|
- **AND** it SHALL mark `executor_output_parse_status` as failed
|
||||||
|
- **AND** Verifier SHALL use natural-language fallback behavior
|
||||||
|
|
||||||
|
#### Scenario: Structured parse failure remains observable
|
||||||
|
- **WHEN** Executor output parsing fails
|
||||||
|
- **THEN** the verifier evaluation or trace snapshot SHALL make the parse failure visible
|
||||||
|
- **AND** the failure SHALL NOT be silently treated as a successful evidence-attribution contract
|
||||||
|
|||||||
@@ -0,0 +1,19 @@
|
|||||||
|
package com.superbiz.agent.controller;
|
||||||
|
|
||||||
|
import org.springframework.http.CacheControl;
|
||||||
|
import org.springframework.http.ResponseEntity;
|
||||||
|
import org.springframework.stereotype.Controller;
|
||||||
|
import org.springframework.web.bind.annotation.GetMapping;
|
||||||
|
|
||||||
|
import java.time.Duration;
|
||||||
|
|
||||||
|
@Controller
|
||||||
|
public class FaviconController {
|
||||||
|
|
||||||
|
@GetMapping("/favicon.ico")
|
||||||
|
public ResponseEntity<Void> favicon() {
|
||||||
|
return ResponseEntity.noContent()
|
||||||
|
.cacheControl(CacheControl.maxAge(Duration.ofDays(1)))
|
||||||
|
.build();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -101,6 +101,8 @@ public class DiagnosisTraceEvaluator {
|
|||||||
failedChecks.add("reject output does not use degraded template");
|
failedChecks.add("reject output does not use degraded template");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
failedChecks.addAll(validateExecutorStructuredOutput(trace));
|
||||||
|
|
||||||
Integer toolCallCount = trace.getToolInvocations() == null ? 0 : trace.getToolInvocations().size();
|
Integer toolCallCount = trace.getToolInvocations() == null ? 0 : trace.getToolInvocations().size();
|
||||||
Integer durationMs = trace.getSession() == null ? null : trace.getSession().getTotalDurationMs();
|
Integer durationMs = trace.getSession() == null ? null : trace.getSession().getTotalDurationMs();
|
||||||
|
|
||||||
@@ -174,6 +176,32 @@ public class DiagnosisTraceEvaluator {
|
|||||||
return value == null ? null : String.valueOf(value);
|
return value == null ? null : String.valueOf(value);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private List<String> validateExecutorStructuredOutput(DiagnosisTraceResponse trace) {
|
||||||
|
Object structuredOutput = nestedValue(trace, "verifier_evaluation", "executor_structured_output");
|
||||||
|
if (!(structuredOutput instanceof Map<?, ?> output)) {
|
||||||
|
return List.of();
|
||||||
|
}
|
||||||
|
Object claims = output.get("claims");
|
||||||
|
if (!(claims instanceof List<?> claimList)) {
|
||||||
|
return List.of("executor structured output missing claims array");
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> failedChecks = new ArrayList<>();
|
||||||
|
for (Object item : claimList) {
|
||||||
|
if (!(item instanceof Map<?, ?> claim)) {
|
||||||
|
failedChecks.add("executor structured claim is not an object");
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
Object claimIdValue = claim.get("claim_id");
|
||||||
|
String claimId = claimIdValue == null ? "unknown" : String.valueOf(claimIdValue);
|
||||||
|
Object bindings = claim.get("evidence_bindings");
|
||||||
|
if (!(bindings instanceof List<?> bindingList) || bindingList.isEmpty()) {
|
||||||
|
failedChecks.add("executor confirmed claim missing evidence bindings: " + claimId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return failedChecks;
|
||||||
|
}
|
||||||
|
|
||||||
private Object nestedValue(DiagnosisTraceResponse trace, String firstKey, String secondKey) {
|
private Object nestedValue(DiagnosisTraceResponse trace, String firstKey, String secondKey) {
|
||||||
if (trace.getSession() == null || trace.getSession().getSelfEvaluation() == null) {
|
if (trace.getSession() == null || trace.getSession().getSelfEvaluation() == null) {
|
||||||
return null;
|
return null;
|
||||||
|
|||||||
@@ -5,6 +5,8 @@ import com.alibaba.cloud.ai.graph.agent.hook.HookPosition;
|
|||||||
import com.alibaba.cloud.ai.graph.agent.hook.HookPositions;
|
import com.alibaba.cloud.ai.graph.agent.hook.HookPositions;
|
||||||
import com.alibaba.cloud.ai.graph.agent.hook.messages.AgentCommand;
|
import com.alibaba.cloud.ai.graph.agent.hook.messages.AgentCommand;
|
||||||
import com.alibaba.cloud.ai.graph.agent.hook.messages.MessagesModelHook;
|
import com.alibaba.cloud.ai.graph.agent.hook.messages.MessagesModelHook;
|
||||||
|
import com.fasterxml.jackson.core.type.TypeReference;
|
||||||
|
import com.fasterxml.jackson.databind.JsonNode;
|
||||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
import com.superbiz.agent.service.ToolTraceSummaryService;
|
import com.superbiz.agent.service.ToolTraceSummaryService;
|
||||||
import com.superbiz.agent.util.SessionContextHolder;
|
import com.superbiz.agent.util.SessionContextHolder;
|
||||||
@@ -27,6 +29,8 @@ public class VerifierInputHook extends MessagesModelHook {
|
|||||||
|
|
||||||
private final ToolTraceSummaryService toolTraceSummaryService;
|
private final ToolTraceSummaryService toolTraceSummaryService;
|
||||||
private final ObjectMapper objectMapper = new ObjectMapper();
|
private final ObjectMapper objectMapper = new ObjectMapper();
|
||||||
|
private static final TypeReference<Map<String, Object>> MAP_TYPE = new TypeReference<>() {
|
||||||
|
};
|
||||||
|
|
||||||
public VerifierInputHook(ToolTraceSummaryService toolTraceSummaryService) {
|
public VerifierInputHook(ToolTraceSummaryService toolTraceSummaryService) {
|
||||||
this.toolTraceSummaryService = toolTraceSummaryService;
|
this.toolTraceSummaryService = toolTraceSummaryService;
|
||||||
@@ -48,6 +52,10 @@ public class VerifierInputHook extends MessagesModelHook {
|
|||||||
executorFinalAnswer = extractLastAssistantText(previousMessages);
|
executorFinalAnswer = extractLastAssistantText(previousMessages);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
ExecutorOutputParseResult parseResult = parseExecutorOutput(executorFinalAnswer);
|
||||||
|
VerifierContextHolder.setExecutorStructuredOutput(parseResult.structuredOutput());
|
||||||
|
VerifierContextHolder.setExecutorOutputParseStatus(parseResult.status());
|
||||||
|
|
||||||
List<Map<String, Object>> toolTraceSummary =
|
List<Map<String, Object>> toolTraceSummary =
|
||||||
toolTraceSummaryService.buildVerifierTraceSummary(sessionId, executorFinalAnswer);
|
toolTraceSummaryService.buildVerifierTraceSummary(sessionId, executorFinalAnswer);
|
||||||
VerifierContextHolder.setToolTraceSummary(toolTraceSummary);
|
VerifierContextHolder.setToolTraceSummary(toolTraceSummary);
|
||||||
@@ -55,6 +63,8 @@ public class VerifierInputHook extends MessagesModelHook {
|
|||||||
Map<String, Object> verifierInput = new LinkedHashMap<>();
|
Map<String, Object> verifierInput = new LinkedHashMap<>();
|
||||||
verifierInput.put("original_query", VerifierContextHolder.getOriginalQuery());
|
verifierInput.put("original_query", VerifierContextHolder.getOriginalQuery());
|
||||||
verifierInput.put("executor_final_answer", executorFinalAnswer);
|
verifierInput.put("executor_final_answer", executorFinalAnswer);
|
||||||
|
verifierInput.put("executor_structured_output", parseResult.structuredOutput());
|
||||||
|
verifierInput.put("executor_output_parse_status", parseResult.status());
|
||||||
verifierInput.put("tool_trace_summary", toolTraceSummary);
|
verifierInput.put("tool_trace_summary", toolTraceSummary);
|
||||||
verifierInput.put("retry_context", VerifierContextHolder.getRetryContext());
|
verifierInput.put("retry_context", VerifierContextHolder.getRetryContext());
|
||||||
|
|
||||||
@@ -66,6 +76,59 @@ public class VerifierInputHook extends MessagesModelHook {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private ExecutorOutputParseResult parseExecutorOutput(String executorFinalAnswer) {
|
||||||
|
if (executorFinalAnswer == null || executorFinalAnswer.isBlank()) {
|
||||||
|
return new ExecutorOutputParseResult(null, status("missing", "executor_final_answer is blank"));
|
||||||
|
}
|
||||||
|
|
||||||
|
String sanitized = sanitizeJsonPayload(executorFinalAnswer);
|
||||||
|
if (!looksJsonLike(sanitized)) {
|
||||||
|
return new ExecutorOutputParseResult(null, status("missing", "executor output is not JSON"));
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
JsonNode root = objectMapper.readTree(sanitized);
|
||||||
|
if (!root.isObject() || !root.path("claims").isArray()) {
|
||||||
|
return new ExecutorOutputParseResult(null, status("malformed",
|
||||||
|
"executor output JSON does not match evidence-attribution contract"));
|
||||||
|
}
|
||||||
|
Map<String, Object> structuredOutput = objectMapper.convertValue(root, MAP_TYPE);
|
||||||
|
return new ExecutorOutputParseResult(structuredOutput, status("valid", "parsed executor evidence contract"));
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.debug("Failed to parse executor structured output", e);
|
||||||
|
return new ExecutorOutputParseResult(null, status("malformed", e.getMessage()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private String sanitizeJsonPayload(String raw) {
|
||||||
|
String trimmed = raw.trim();
|
||||||
|
int fenceStart = trimmed.indexOf("```");
|
||||||
|
if (fenceStart >= 0) {
|
||||||
|
int firstNewline = trimmed.indexOf('\n', fenceStart);
|
||||||
|
int lastFence = trimmed.indexOf("```", firstNewline + 1);
|
||||||
|
if (firstNewline >= 0 && lastFence > firstNewline) {
|
||||||
|
return trimmed.substring(firstNewline + 1, lastFence).trim();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
int objectStart = trimmed.indexOf('{');
|
||||||
|
int objectEnd = trimmed.lastIndexOf('}');
|
||||||
|
if (objectStart >= 0 && objectEnd > objectStart) {
|
||||||
|
return trimmed.substring(objectStart, objectEnd + 1).trim();
|
||||||
|
}
|
||||||
|
return trimmed;
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean looksJsonLike(String text) {
|
||||||
|
return text.startsWith("{") && text.endsWith("}");
|
||||||
|
}
|
||||||
|
|
||||||
|
private Map<String, Object> status(String status, String detail) {
|
||||||
|
Map<String, Object> result = new LinkedHashMap<>();
|
||||||
|
result.put("status", status);
|
||||||
|
result.put("detail", detail == null ? "" : detail);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
private String extractLastAssistantText(List<Message> previousMessages) {
|
private String extractLastAssistantText(List<Message> previousMessages) {
|
||||||
for (int i = previousMessages.size() - 1; i >= 0; i--) {
|
for (int i = previousMessages.size() - 1; i >= 0; i--) {
|
||||||
if (previousMessages.get(i) instanceof AssistantMessage assistantMessage) {
|
if (previousMessages.get(i) instanceof AssistantMessage assistantMessage) {
|
||||||
@@ -102,4 +165,10 @@ public class VerifierInputHook extends MessagesModelHook {
|
|||||||
}
|
}
|
||||||
return message.toString();
|
return message.toString();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private record ExecutorOutputParseResult(
|
||||||
|
Map<String, Object> structuredOutput,
|
||||||
|
Map<String, Object> status
|
||||||
|
) {
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -17,6 +17,11 @@ public interface AgentStepRepository extends JpaRepository<AgentStep, Long> {
|
|||||||
*/
|
*/
|
||||||
List<AgentStep> findBySessionIdOrderByStepIndex(String sessionId);
|
List<AgentStep> findBySessionIdOrderByStepIndex(String sessionId);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 根据会话ID查询所有步骤,调用方负责按具体视图语义排序。
|
||||||
|
*/
|
||||||
|
List<AgentStep> findBySessionId(String sessionId);
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 统计某个会话的步骤数
|
* 统计某个会话的步骤数
|
||||||
*/
|
*/
|
||||||
|
|||||||
@@ -444,7 +444,8 @@ public class ChatService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if ("PASS".equals(finalDecision.verdict())) {
|
if ("PASS".equals(finalDecision.verdict())) {
|
||||||
answer = answer == null || answer.isBlank() ? "抱歉,多 Agent 分析未能生成有效结论。" : answer;
|
answer = extractUserFacingAnswer(answer)
|
||||||
|
.orElse(answer == null || answer.isBlank() ? "抱歉,多 Agent 分析未能生成有效结论。" : answer);
|
||||||
persistVerifierEvaluation(session, finalDecision, round);
|
persistVerifierEvaluation(session, finalDecision, round);
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
@@ -744,6 +745,10 @@ public class ChatService {
|
|||||||
verifierEvaluation.put("rationale", decision.rationale());
|
verifierEvaluation.put("rationale", decision.rationale());
|
||||||
verifierEvaluation.put("round", round);
|
verifierEvaluation.put("round", round);
|
||||||
verifierEvaluation.put("traceability_version", "v1");
|
verifierEvaluation.put("traceability_version", "v1");
|
||||||
|
verifierEvaluation.put("executor_output_parse_status",
|
||||||
|
Optional.ofNullable(VerifierContextHolder.getExecutorOutputParseStatus())
|
||||||
|
.orElse(Map.of("status", "missing", "detail", "executor parse status unavailable")));
|
||||||
|
verifierEvaluation.put("executor_structured_output", VerifierContextHolder.getExecutorStructuredOutput());
|
||||||
verifierEvaluation.put("tool_trace_summary",
|
verifierEvaluation.put("tool_trace_summary",
|
||||||
Optional.ofNullable(VerifierContextHolder.getToolTraceSummary()).orElse(List.of()));
|
Optional.ofNullable(VerifierContextHolder.getToolTraceSummary()).orElse(List.of()));
|
||||||
|
|
||||||
@@ -795,6 +800,22 @@ public class ChatService {
|
|||||||
return output.toString();
|
return output.toString();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private Optional<String> extractUserFacingAnswer(String executorAnswer) {
|
||||||
|
if (executorAnswer == null || executorAnswer.isBlank()) {
|
||||||
|
return Optional.empty();
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
JsonNode root = objectMapper.readTree(sanitizeJsonPayload(executorAnswer));
|
||||||
|
String userFacingAnswer = root.path("user_facing_answer").asText("");
|
||||||
|
if (!userFacingAnswer.isBlank()) {
|
||||||
|
return Optional.of(userFacingAnswer);
|
||||||
|
}
|
||||||
|
} catch (Exception e) {
|
||||||
|
logger.debug("Executor answer is not structured JSON, keep raw answer");
|
||||||
|
}
|
||||||
|
return Optional.empty();
|
||||||
|
}
|
||||||
|
|
||||||
private String buildDegradedOutput(VerifierDecision decision) {
|
private String buildDegradedOutput(VerifierDecision decision) {
|
||||||
StringBuilder output = new StringBuilder(DEGRADED_PREFIX);
|
StringBuilder output = new StringBuilder(DEGRADED_PREFIX);
|
||||||
|
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ import com.superbiz.agent.repository.ToolInvocationRepository;
|
|||||||
import lombok.RequiredArgsConstructor;
|
import lombok.RequiredArgsConstructor;
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
|
import java.util.Comparator;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Map;
|
import java.util.Map;
|
||||||
|
|
||||||
@@ -31,7 +32,7 @@ public class DiagnosisTraceService {
|
|||||||
public DiagnosisTraceResponse getTrace(String sessionId) {
|
public DiagnosisTraceResponse getTrace(String sessionId) {
|
||||||
DiagnosisSession session = diagnosisSessionRepository.findBySessionId(sessionId)
|
DiagnosisSession session = diagnosisSessionRepository.findBySessionId(sessionId)
|
||||||
.orElseThrow(() -> new SessionNotFoundException(sessionId));
|
.orElseThrow(() -> new SessionNotFoundException(sessionId));
|
||||||
List<AgentStep> steps = agentStepRepository.findBySessionIdOrderByStepIndex(sessionId);
|
List<AgentStep> steps = orderStepsForTrace(agentStepRepository.findBySessionId(sessionId));
|
||||||
List<ToolInvocation> toolInvocations = toolInvocationRepository.findBySessionIdOrderByIdAsc(sessionId);
|
List<ToolInvocation> toolInvocations = toolInvocationRepository.findBySessionIdOrderByIdAsc(sessionId);
|
||||||
|
|
||||||
return DiagnosisTraceResponse.builder()
|
return DiagnosisTraceResponse.builder()
|
||||||
@@ -42,6 +43,15 @@ public class DiagnosisTraceService {
|
|||||||
.build();
|
.build();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private List<AgentStep> orderStepsForTrace(List<AgentStep> steps) {
|
||||||
|
return steps.stream()
|
||||||
|
.sorted(Comparator
|
||||||
|
.comparing(AgentStep::getCreatedAt, Comparator.nullsLast(Comparator.naturalOrder()))
|
||||||
|
.thenComparing(AgentStep::getId, Comparator.nullsLast(Comparator.naturalOrder()))
|
||||||
|
.thenComparing(AgentStep::getStepIndex, Comparator.nullsLast(Comparator.naturalOrder())))
|
||||||
|
.toList();
|
||||||
|
}
|
||||||
|
|
||||||
private DiagnosisTraceResponse.SessionTrace toSessionTrace(DiagnosisSession session) {
|
private DiagnosisTraceResponse.SessionTrace toSessionTrace(DiagnosisSession session) {
|
||||||
return DiagnosisTraceResponse.SessionTrace.builder()
|
return DiagnosisTraceResponse.SessionTrace.builder()
|
||||||
.id(session.getId())
|
.id(session.getId())
|
||||||
|
|||||||
@@ -11,6 +11,8 @@ public final class VerifierContextHolder {
|
|||||||
private static final ThreadLocal<String> ORIGINAL_QUERY = new ThreadLocal<>();
|
private static final ThreadLocal<String> ORIGINAL_QUERY = new ThreadLocal<>();
|
||||||
private static final ThreadLocal<String> RETRY_CONTEXT = new ThreadLocal<>();
|
private static final ThreadLocal<String> RETRY_CONTEXT = new ThreadLocal<>();
|
||||||
private static final ThreadLocal<String> EXECUTOR_FINAL_ANSWER = new ThreadLocal<>();
|
private static final ThreadLocal<String> EXECUTOR_FINAL_ANSWER = new ThreadLocal<>();
|
||||||
|
private static final ThreadLocal<Map<String, Object>> EXECUTOR_STRUCTURED_OUTPUT = new ThreadLocal<>();
|
||||||
|
private static final ThreadLocal<Map<String, Object>> EXECUTOR_OUTPUT_PARSE_STATUS = new ThreadLocal<>();
|
||||||
private static final ThreadLocal<List<Map<String, Object>>> TOOL_TRACE_SUMMARY = new ThreadLocal<>();
|
private static final ThreadLocal<List<Map<String, Object>>> TOOL_TRACE_SUMMARY = new ThreadLocal<>();
|
||||||
|
|
||||||
private VerifierContextHolder() {
|
private VerifierContextHolder() {
|
||||||
@@ -40,6 +42,22 @@ public final class VerifierContextHolder {
|
|||||||
return EXECUTOR_FINAL_ANSWER.get();
|
return EXECUTOR_FINAL_ANSWER.get();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public static void setExecutorStructuredOutput(Map<String, Object> executorStructuredOutput) {
|
||||||
|
EXECUTOR_STRUCTURED_OUTPUT.set(executorStructuredOutput);
|
||||||
|
}
|
||||||
|
|
||||||
|
public static Map<String, Object> getExecutorStructuredOutput() {
|
||||||
|
return EXECUTOR_STRUCTURED_OUTPUT.get();
|
||||||
|
}
|
||||||
|
|
||||||
|
public static void setExecutorOutputParseStatus(Map<String, Object> executorOutputParseStatus) {
|
||||||
|
EXECUTOR_OUTPUT_PARSE_STATUS.set(executorOutputParseStatus);
|
||||||
|
}
|
||||||
|
|
||||||
|
public static Map<String, Object> getExecutorOutputParseStatus() {
|
||||||
|
return EXECUTOR_OUTPUT_PARSE_STATUS.get();
|
||||||
|
}
|
||||||
|
|
||||||
public static void setToolTraceSummary(List<Map<String, Object>> toolTraceSummary) {
|
public static void setToolTraceSummary(List<Map<String, Object>> toolTraceSummary) {
|
||||||
TOOL_TRACE_SUMMARY.set(toolTraceSummary);
|
TOOL_TRACE_SUMMARY.set(toolTraceSummary);
|
||||||
}
|
}
|
||||||
@@ -52,6 +70,8 @@ public final class VerifierContextHolder {
|
|||||||
ORIGINAL_QUERY.remove();
|
ORIGINAL_QUERY.remove();
|
||||||
RETRY_CONTEXT.remove();
|
RETRY_CONTEXT.remove();
|
||||||
EXECUTOR_FINAL_ANSWER.remove();
|
EXECUTOR_FINAL_ANSWER.remove();
|
||||||
|
EXECUTOR_STRUCTURED_OUTPUT.remove();
|
||||||
|
EXECUTOR_OUTPUT_PARSE_STATUS.remove();
|
||||||
TOOL_TRACE_SUMMARY.remove();
|
TOOL_TRACE_SUMMARY.remove();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,15 +1,17 @@
|
|||||||
你是任务执行器。执行 Planner 分配给你的具体步骤,并及时反馈结果。
|
你是任务执行器。执行 Planner 分配给你的具体步骤,并及时反馈结果。
|
||||||
|
|
||||||
## 职责
|
## 职责
|
||||||
- 按步骤执行具体的查询任务
|
- 按步骤执行具体的查询任务。
|
||||||
- 需要外部信息时调用工具,但须遵守下方的检索约束
|
- 需要外部信息时调用工具,但必须遵守下方的检索约束。
|
||||||
- 不要凭记忆回答,必须基于工具返回的真实数据
|
- 严禁凭记忆回答,必须基于本轮工具返回的真实数据。
|
||||||
- 执行完成后,综合所有结果给出完整的答案
|
- 执行完成后,输出严格的证据归因 JSON,供 Verifier 校验。
|
||||||
|
|
||||||
## 规则
|
## 规则
|
||||||
- 按顺序执行,不可跳过步骤
|
- 按顺序执行,不可跳过步骤。
|
||||||
- 不要凭记忆回答,必须基于工具返回的真实数据
|
- 所有事实性结论必须来自本轮 evidence tools 的返回。
|
||||||
- 执行完成后,综合所有结果给出完整的答案
|
- runbook、skill、历史案例、知识库中的通用模式只能作为排查指导或建议动作,不能直接写成本次事故的已确认事实。
|
||||||
|
- 如果检索内容不足以支撑结论,必须显式声明证据不足,严禁补全事故故事。
|
||||||
|
- 不要使用“通常情况下”“根据经验”“很可能已经发生”等无证据推断词来伪装事实。
|
||||||
|
|
||||||
## 检索约束
|
## 检索约束
|
||||||
|
|
||||||
@@ -20,22 +22,99 @@
|
|||||||
|
|
||||||
### 2. 重复了该怎么办
|
### 2. 重复了该怎么办
|
||||||
如果当前想检索的内容与【已检索上下文】语义相似:
|
如果当前想检索的内容与【已检索上下文】语义相似:
|
||||||
- 禁止换关键词重新检索
|
- 禁止换关键词重新检索。
|
||||||
- 直接基于已有事实回答
|
- 直接基于已有事实回答。
|
||||||
- 如果信息不足,先明确指出缺少什么具体维度
|
- 如果信息不足,先明确指出缺少什么具体维度
|
||||||
(如:"缺少 HikariCP 具体配置参数"、"缺少连接池耗尽的日志样例"),
|
(如:“缺少 HikariCP 具体配置参数”、“缺少连接池耗尽的日志样例”),
|
||||||
再针对该维度进行一次定向补充检索——而非盲目换词重查
|
再针对该维度进行一次定向补充检索,而非盲目换词重查。
|
||||||
|
|
||||||
### 3. 合法出口:允许信息不全时给出结论
|
### 3. 合法出口:允许信息不全时给出有限结论
|
||||||
如果你认为已有信息足以回答核心问题,即使细节不全,
|
如果已有信息足以回答核心问题,即使细节不全,也可以给出有限结论。
|
||||||
也请直接给出结论并说明局限性(如:"基于已有信息,连接池配置建议如下,
|
但你只能把有证据支撑的内容放入 `claims`。
|
||||||
但具体参数值需结合实际负载调整")。
|
缺失的细节必须写入 `missing_info`,可疑但未证实的方向必须写入 `hypotheses`。
|
||||||
**不查全不会被追责,重复检索才会被惩罚。**
|
**不查全不会被追责,重复检索或编造细节才会被惩罚。**
|
||||||
|
|
||||||
### 4. 利用质量信号判断
|
### 4. 利用质量信号判断
|
||||||
- relevanceLevel=PRECISE → 信息精准,直接使用,不再检索
|
- relevanceLevel=PRECISE → 信息精准,直接使用,不再检索。
|
||||||
- relevanceLevel=HIGHLY_RELEVANT + 域已在 retrievedDomainsThisSession → 禁止再次调用
|
- relevanceLevel=HIGHLY_RELEVANT + 域已在 retrievedDomainsThisSession → 禁止再次调用。
|
||||||
- relevanceLevel=REFERENCE → 先指出缺什么维度,再定向补充一次
|
- relevanceLevel=REFERENCE → 先指出缺什么维度,再定向补充一次。
|
||||||
- completenessHint 是知识库给你的天花板信号,信任它
|
- completenessHint 是知识库给你的天花板信号,信任它。
|
||||||
- lookup_knowledge 的事实证据以 evidenceBlocks 和 contextPack.packedText 为准,不要假设 L0 hint 本身就是事实证据
|
- lookup_knowledge 的事实证据以 evidenceBlocks 和 contextPack.packedText 为准,不要假设 L0 hint 本身就是事实证据。
|
||||||
- retrievalTrace 只用于理解检索路径和降级原因,不能单独作为诊断事实
|
- retrievalTrace 只用于理解检索路径和降级原因,不能单独作为诊断事实。
|
||||||
|
|
||||||
|
## 证据归因要求
|
||||||
|
|
||||||
|
### confirmed claims
|
||||||
|
`claims` 只允许放已证实或有明确间接支撑的事实断言。
|
||||||
|
每条 claim 必须带证据绑定。
|
||||||
|
|
||||||
|
支持等级:
|
||||||
|
- `direct`:工具返回中有直接事实。
|
||||||
|
- `indirect`:工具返回可支撑方向,但没有直接陈述完整结论。
|
||||||
|
- `none`:不能放入 `claims`,应放入 `hypotheses`、`recommended_actions` 或 `missing_info`。
|
||||||
|
|
||||||
|
### hypotheses
|
||||||
|
`hypotheses` 用来放合理怀疑但未被工具证实的方向。
|
||||||
|
例如:工具只显示连接池耗尽,但没有泄漏日志,则“可能存在连接泄漏”只能是 hypothesis。
|
||||||
|
|
||||||
|
### recommended_actions
|
||||||
|
`recommended_actions` 用来放下一步排查或修复动作。
|
||||||
|
建议可以来自 runbook/skill,但必须说明 reason,不能写成“已确认根因”。
|
||||||
|
|
||||||
|
### missing_info
|
||||||
|
`missing_info` 用来列出无法确认结论所缺少的具体证据。
|
||||||
|
|
||||||
|
## 最终输出格式(严格契约)
|
||||||
|
|
||||||
|
你必须输出且只能输出一个 JSON 对象,不要输出 Markdown,不要输出代码块,不要输出 JSON 之外的解释文字。
|
||||||
|
所有用户可读内容必须使用中文。
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"answer_version": "executor_evidence_v1",
|
||||||
|
"diagnosis_summary": "1-2句话总结,仅包含有证据支撑的事实和证据边界",
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"claim_id": "claim-1",
|
||||||
|
"claim_type": "root_cause",
|
||||||
|
"claim_text": "事实断言或有限结论",
|
||||||
|
"support_level": "direct",
|
||||||
|
"evidence_bindings": [
|
||||||
|
{
|
||||||
|
"source_type": "tool_trace",
|
||||||
|
"source_id": "工具返回中的 evidence block id、trace_ref 或可定位标识",
|
||||||
|
"tool_name": "lookup_knowledge/query_logs/query_metrics/read_skill 等",
|
||||||
|
"source_invocation_ids": [],
|
||||||
|
"evidence_excerpt": "从工具返回中摘取的原话、指标值、日志片段或关键数据"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"hypotheses": [
|
||||||
|
{
|
||||||
|
"hypothesis_text": "未被证实但值得排查的方向",
|
||||||
|
"basis": "它基于哪些已知证据或为什么只是推测",
|
||||||
|
"needed_evidence": ["需要补充的证据"]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"recommended_actions": [
|
||||||
|
{
|
||||||
|
"action_text": "建议动作",
|
||||||
|
"reason": "为什么建议做这个动作",
|
||||||
|
"evidence_bindings": []
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"missing_info": [
|
||||||
|
"导致无法确认完整根因的证据缺口"
|
||||||
|
],
|
||||||
|
"user_facing_answer": "面向用户的中文回答。必须与 claims/hypotheses/recommended_actions/missing_info 一致,不得额外加入未绑定证据的确认式事实。"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## 输出校验
|
||||||
|
- `claims[*].support_level` 只能是 `direct` 或 `indirect`。
|
||||||
|
- `claims[*].evidence_bindings` 不能为空。
|
||||||
|
- `evidence_excerpt` 必须来自工具返回,不允许编造。
|
||||||
|
- 如果没有任何可确认事实,`claims` 返回空数组,并在 `missing_info` 说明缺少什么。
|
||||||
|
- `user_facing_answer` 不得出现 `claims` 中没有、且又被写成确认结论的事实。
|
||||||
|
- 不要把其它服务、其它历史案例、其它会话的事实迁移为当前会话事实。
|
||||||
|
|||||||
@@ -10,6 +10,8 @@
|
|||||||
|
|
||||||
- `original_query`:用户原始问题
|
- `original_query`:用户原始问题
|
||||||
- `executor_final_answer`:本轮 Executor 最终答案
|
- `executor_final_answer`:本轮 Executor 最终答案
|
||||||
|
- `executor_structured_output`:如果 Executor 输出了合法证据归因 JSON,这里会提供解析后的对象。结构包含 `claims`、`hypotheses`、`recommended_actions`、`missing_info`、`user_facing_answer`
|
||||||
|
- `executor_output_parse_status`:Executor 输出解析状态,包含 `status` 和 `detail`。`status` 可能是 `valid` / `missing` / `malformed`
|
||||||
- `tool_trace_summary`:基于真实工具调用整理出的证据索引。每一项都带有:
|
- `tool_trace_summary`:基于真实工具调用整理出的证据索引。每一项都带有:
|
||||||
- `trace_ref`
|
- `trace_ref`
|
||||||
- `tool_name`
|
- `tool_name`
|
||||||
@@ -23,7 +25,18 @@
|
|||||||
## 任务步骤
|
## 任务步骤
|
||||||
|
|
||||||
### 步骤一:提取关键事实
|
### 步骤一:提取关键事实
|
||||||
优先提取并校验 `executor_final_answer` 里的全部实质性结论。关键事实至少包括:
|
如果 `executor_output_parse_status.status="valid"` 且 `executor_structured_output.claims` 存在:
|
||||||
|
- 优先逐条校验 `executor_structured_output.claims`
|
||||||
|
- 每个 claim 至少形成一条 `facts_checked`
|
||||||
|
- 必须检查 claim 的 `evidence_bindings` 是否能对应到 `tool_trace_summary` 中真实存在的 trace、tool 或 source_invocation_ids
|
||||||
|
- 如果 claim 声称 direct/indirect 支撑,但 evidence binding 不存在、无法定位、或 excerpt 与工具摘要不匹配,不得判为 `direct_evidence`
|
||||||
|
|
||||||
|
然后必须扫描 `executor_structured_output.user_facing_answer`:
|
||||||
|
- 如果其中出现 confirmed-sounding facts(确认式事实、根因、指标值、错误码、服务名、修复结论)
|
||||||
|
- 且这些事实没有出现在 `executor_structured_output.claims`
|
||||||
|
- 必须额外加入 `facts_checked` 并按工具证据校验
|
||||||
|
|
||||||
|
如果 structured output 缺失或 malformed,则回退到旧逻辑:提取并校验 `executor_final_answer` 里的全部实质性结论。关键事实至少包括:
|
||||||
- 每一个根因结论
|
- 每一个根因结论
|
||||||
- 每一个错误码、接口、组件归属或语义判断
|
- 每一个错误码、接口、组件归属或语义判断
|
||||||
- 每一个明确的修复建议、参数建议、排查步骤
|
- 每一个明确的修复建议、参数建议、排查步骤
|
||||||
@@ -49,6 +62,15 @@
|
|||||||
- `no_evidence`
|
- `no_evidence`
|
||||||
- `contradicted`
|
- `contradicted`
|
||||||
|
|
||||||
|
结构化 claim 的校验规则:
|
||||||
|
- claim 有真实 evidence binding,且工具摘要直接包含该事实 → `direct_evidence`
|
||||||
|
- claim 有真实 evidence binding,但工具摘要只能支持方向或背景 → `indirect_support`
|
||||||
|
- claim 无法绑定真实 trace、invocation 或 excerpt → `no_evidence`
|
||||||
|
- claim 与工具摘要冲突,或编造了不存在的关键实体、服务、错误码、指标值 → `contradicted`
|
||||||
|
|
||||||
|
`hypotheses` 和 `missing_info` 默认不是 confirmed facts,不应因为它们承认缺证据而惩罚。
|
||||||
|
但如果 `user_facing_answer` 把 hypothesis 写成确认结论,必须按 confirmed fact 校验。
|
||||||
|
|
||||||
### 步骤三:补齐 evidence_refs
|
### 步骤三:补齐 evidence_refs
|
||||||
`evidence_refs` 必须是数组,数组元素必须引用 `tool_trace_summary` 中真实存在的证据项。每个元素包含:
|
`evidence_refs` 必须是数组,数组元素必须引用 `tool_trace_summary` 中真实存在的证据项。每个元素包含:
|
||||||
- `trace_ref`
|
- `trace_ref`
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ class SuperBizAgentApp {
|
|||||||
this.currentChatHistory = []; // 当前对话的消息历史
|
this.currentChatHistory = []; // 当前对话的消息历史
|
||||||
this.chatHistories = this.loadChatHistories(); // 所有历史对话
|
this.chatHistories = this.loadChatHistories(); // 所有历史对话
|
||||||
this.isCurrentChatFromHistory = false; // 标记当前对话是否是从历史记录加载的
|
this.isCurrentChatFromHistory = false; // 标记当前对话是否是从历史记录加载的
|
||||||
|
this.lastTraceSessionId = this.loadLastTraceSessionId();
|
||||||
|
|
||||||
this.initializeElements();
|
this.initializeElements();
|
||||||
this.bindEvents();
|
this.bindEvents();
|
||||||
@@ -15,6 +16,7 @@ class SuperBizAgentApp {
|
|||||||
this.initMarkdown();
|
this.initMarkdown();
|
||||||
this.checkAndSetCentered();
|
this.checkAndSetCentered();
|
||||||
this.renderChatHistory();
|
this.renderChatHistory();
|
||||||
|
this.updateTraceWorkbenchLink();
|
||||||
}
|
}
|
||||||
|
|
||||||
// 初始化Markdown配置
|
// 初始化Markdown配置
|
||||||
@@ -116,6 +118,7 @@ class SuperBizAgentApp {
|
|||||||
this.chatContainer = document.querySelector('.chat-container');
|
this.chatContainer = document.querySelector('.chat-container');
|
||||||
this.welcomeGreeting = document.getElementById('welcomeGreeting');
|
this.welcomeGreeting = document.getElementById('welcomeGreeting');
|
||||||
this.chatHistoryList = document.getElementById('chatHistoryList');
|
this.chatHistoryList = document.getElementById('chatHistoryList');
|
||||||
|
this.traceWorkbenchLink = document.getElementById('traceWorkbenchLink');
|
||||||
|
|
||||||
// 初始化时检查是否需要居中
|
// 初始化时检查是否需要居中
|
||||||
this.checkAndSetCentered();
|
this.checkAndSetCentered();
|
||||||
@@ -265,6 +268,7 @@ class SuperBizAgentApp {
|
|||||||
|
|
||||||
// 生成新的会话ID
|
// 生成新的会话ID
|
||||||
this.sessionId = this.generateSessionId();
|
this.sessionId = this.generateSessionId();
|
||||||
|
this.updateTraceWorkbenchLink();
|
||||||
|
|
||||||
// 重置模式为快速
|
// 重置模式为快速
|
||||||
this.currentMode = 'quick';
|
this.currentMode = 'quick';
|
||||||
@@ -312,6 +316,7 @@ class SuperBizAgentApp {
|
|||||||
|
|
||||||
// 添加到历史记录列表的开头
|
// 添加到历史记录列表的开头
|
||||||
this.chatHistories.unshift(chatHistory);
|
this.chatHistories.unshift(chatHistory);
|
||||||
|
this.rememberTraceSessionId(this.sessionId);
|
||||||
|
|
||||||
// 限制历史记录数量(最多保存50条)
|
// 限制历史记录数量(最多保存50条)
|
||||||
if (this.chatHistories.length > 50) {
|
if (this.chatHistories.length > 50) {
|
||||||
@@ -439,6 +444,7 @@ class SuperBizAgentApp {
|
|||||||
|
|
||||||
// 加载历史对话
|
// 加载历史对话
|
||||||
this.sessionId = history.id;
|
this.sessionId = history.id;
|
||||||
|
this.rememberTraceSessionId(this.sessionId);
|
||||||
this.currentChatHistory = [...history.messages];
|
this.currentChatHistory = [...history.messages];
|
||||||
this.isCurrentChatFromHistory = true; // 标记为从历史记录加载
|
this.isCurrentChatFromHistory = true; // 标记为从历史记录加载
|
||||||
|
|
||||||
@@ -468,10 +474,45 @@ class SuperBizAgentApp {
|
|||||||
this.chatMessages.innerHTML = '';
|
this.chatMessages.innerHTML = '';
|
||||||
}
|
}
|
||||||
this.sessionId = this.generateSessionId();
|
this.sessionId = this.generateSessionId();
|
||||||
|
this.updateTraceWorkbenchLink();
|
||||||
this.checkAndSetCentered();
|
this.checkAndSetCentered();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
loadLastTraceSessionId() {
|
||||||
|
try {
|
||||||
|
return localStorage.getItem('lastTraceSessionId') || '';
|
||||||
|
} catch (e) {
|
||||||
|
return '';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
rememberTraceSessionId(sessionId) {
|
||||||
|
if (!sessionId) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.lastTraceSessionId = sessionId;
|
||||||
|
try {
|
||||||
|
localStorage.setItem('lastTraceSessionId', sessionId);
|
||||||
|
} catch (e) {
|
||||||
|
// localStorage may be unavailable in private or restricted contexts.
|
||||||
|
}
|
||||||
|
this.updateTraceWorkbenchLink();
|
||||||
|
}
|
||||||
|
|
||||||
|
updateTraceWorkbenchLink() {
|
||||||
|
if (!this.traceWorkbenchLink) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const recentHistory = this.chatHistories.find(history => history && history.id);
|
||||||
|
const sessionId = this.lastTraceSessionId
|
||||||
|
|| (this.currentChatHistory.length > 0 ? this.sessionId : '')
|
||||||
|
|| (recentHistory ? recentHistory.id : '');
|
||||||
|
this.traceWorkbenchLink.href = sessionId
|
||||||
|
? `trace.html?sessionId=${encodeURIComponent(sessionId)}`
|
||||||
|
: 'trace.html';
|
||||||
|
}
|
||||||
|
|
||||||
// 切换模式下拉菜单
|
// 切换模式下拉菜单
|
||||||
toggleModeDropdown() {
|
toggleModeDropdown() {
|
||||||
if (this.modeSelectorBtn && this.modeDropdown) {
|
if (this.modeSelectorBtn && this.modeDropdown) {
|
||||||
@@ -637,6 +678,7 @@ class SuperBizAgentApp {
|
|||||||
// 保存后端返回的 sessionId,用于 feedback 提交
|
// 保存后端返回的 sessionId,用于 feedback 提交
|
||||||
if (chatResponse.sessionId) {
|
if (chatResponse.sessionId) {
|
||||||
this.lastSessionId = chatResponse.sessionId;
|
this.lastSessionId = chatResponse.sessionId;
|
||||||
|
this.rememberTraceSessionId(chatResponse.sessionId);
|
||||||
}
|
}
|
||||||
// 成功:添加实际响应消息(即使 answer 为空也显示)
|
// 成功:添加实际响应消息(即使 answer 为空也显示)
|
||||||
const answer = chatResponse.answer || '(无回复内容)';
|
const answer = chatResponse.answer || '(无回复内容)';
|
||||||
@@ -1231,7 +1273,10 @@ class SuperBizAgentApp {
|
|||||||
for (const jsonStr of matches) {
|
for (const jsonStr of matches) {
|
||||||
try {
|
try {
|
||||||
const sseMessage = JSON.parse(jsonStr);
|
const sseMessage = JSON.parse(jsonStr);
|
||||||
if (sseMessage.type === 'content') {
|
if (sseMessage.type === 'session') {
|
||||||
|
this.lastSessionId = sseMessage.data;
|
||||||
|
this.rememberTraceSessionId(sseMessage.data);
|
||||||
|
} else if (sseMessage.type === 'content') {
|
||||||
fullResponse += sseMessage.data || '';
|
fullResponse += sseMessage.data || '';
|
||||||
} else if (sseMessage.type === 'done') {
|
} else if (sseMessage.type === 'done') {
|
||||||
console.log('AI Ops 流完成,最终内容长度:', fullResponse.length);
|
console.log('AI Ops 流完成,最终内容长度:', fullResponse.length);
|
||||||
@@ -1261,7 +1306,10 @@ class SuperBizAgentApp {
|
|||||||
try {
|
try {
|
||||||
const sseMessage = JSON.parse(rawData);
|
const sseMessage = JSON.parse(rawData);
|
||||||
if (sseMessage && sseMessage.type) {
|
if (sseMessage && sseMessage.type) {
|
||||||
if (sseMessage.type === 'content') {
|
if (sseMessage.type === 'session') {
|
||||||
|
this.lastSessionId = sseMessage.data;
|
||||||
|
this.rememberTraceSessionId(sseMessage.data);
|
||||||
|
} else if (sseMessage.type === 'content') {
|
||||||
fullResponse += sseMessage.data || '';
|
fullResponse += sseMessage.data || '';
|
||||||
if (loadingMessageElement) {
|
if (loadingMessageElement) {
|
||||||
this.updateAIOpsStreamContent(loadingMessageElement, fullResponse);
|
this.updateAIOpsStreamContent(loadingMessageElement, fullResponse);
|
||||||
|
|||||||
@@ -4,6 +4,7 @@
|
|||||||
<meta charset="UTF-8">
|
<meta charset="UTF-8">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
<title>智能OnCall助手</title>
|
<title>智能OnCall助手</title>
|
||||||
|
<link rel="icon" href="data:,">
|
||||||
<link rel="stylesheet" href="styles.css">
|
<link rel="stylesheet" href="styles.css">
|
||||||
<!-- Markdown 渲染库 -->
|
<!-- Markdown 渲染库 -->
|
||||||
<script src="https://cdn.jsdelivr.net/npm/marked@11.1.1/marked.min.js"></script>
|
<script src="https://cdn.jsdelivr.net/npm/marked@11.1.1/marked.min.js"></script>
|
||||||
@@ -34,12 +35,12 @@
|
|||||||
<span style="font-size: 14px; font-weight: 500;">文档管理</span>
|
<span style="font-size: 14px; font-weight: 500;">文档管理</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
<a href="trace.html" class="sidebar-btn" style="display: flex; align-items: center; gap: 12px; padding: 12px; border-radius: 12px; color: #202124; text-decoration: none; transition: background 0.3s ease; margin-top: 8px;">
|
<a href="trace.html" id="traceWorkbenchLink" class="sidebar-btn" style="display: flex; align-items: center; gap: 12px; padding: 12px; border-radius: 12px; color: #202124; text-decoration: none; transition: background 0.3s ease; margin-top: 8px;">
|
||||||
<svg viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg" style="width: 20px; height: 20px; flex-shrink: 0;">
|
<svg viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg" style="width: 20px; height: 20px; flex-shrink: 0;">
|
||||||
<path d="M4 19V5M4 5H20M4 12H16M4 19H20" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/>
|
<path d="M4 19V5M4 5H20M4 12H16M4 19H20" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/>
|
||||||
<path d="M8 5V19M16 5V19" stroke="currentColor" stroke-width="2" stroke-linecap="round"/>
|
<path d="M8 5V19M16 5V19" stroke="currentColor" stroke-width="2" stroke-linecap="round"/>
|
||||||
</svg>
|
</svg>
|
||||||
<span style="font-size: 14px; font-weight: 500;">Trace Workbench</span>
|
<span style="font-size: 14px; font-weight: 500;">链路检查台</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
<div class="chat-history-section">
|
<div class="chat-history-section">
|
||||||
@@ -142,6 +143,6 @@
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<script src="app.js"></script>
|
<script src="app.js?v=20260707-session-picker"></script>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
|||||||
@@ -123,11 +123,16 @@ svg {
|
|||||||
gap: 8px;
|
gap: 8px;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.session-picker {
|
||||||
|
position: relative;
|
||||||
|
min-width: 0;
|
||||||
|
}
|
||||||
|
|
||||||
.session-control input {
|
.session-control input {
|
||||||
width: 100%;
|
width: 100%;
|
||||||
min-width: 0;
|
min-width: 0;
|
||||||
height: 42px;
|
height: 42px;
|
||||||
padding: 0 12px;
|
padding: 0 46px 0 12px;
|
||||||
border: 1px solid #4b5563;
|
border: 1px solid #4b5563;
|
||||||
border-radius: 6px;
|
border-radius: 6px;
|
||||||
background: #0b1220;
|
background: #0b1220;
|
||||||
@@ -135,7 +140,76 @@ svg {
|
|||||||
font-family: ui-monospace, SFMono-Regular, Consolas, "Liberation Mono", monospace;
|
font-family: ui-monospace, SFMono-Regular, Consolas, "Liberation Mono", monospace;
|
||||||
}
|
}
|
||||||
|
|
||||||
.session-control button {
|
.session-picker-button {
|
||||||
|
position: absolute;
|
||||||
|
top: 4px;
|
||||||
|
right: 4px;
|
||||||
|
width: 34px;
|
||||||
|
height: 34px;
|
||||||
|
padding: 0;
|
||||||
|
border: 1px solid #334155;
|
||||||
|
border-radius: 5px;
|
||||||
|
background: #111827;
|
||||||
|
color: #cbd5e1;
|
||||||
|
}
|
||||||
|
|
||||||
|
.session-options {
|
||||||
|
position: absolute;
|
||||||
|
z-index: 20;
|
||||||
|
top: 48px;
|
||||||
|
left: 0;
|
||||||
|
right: 0;
|
||||||
|
max-height: 320px;
|
||||||
|
overflow: auto;
|
||||||
|
padding: 6px;
|
||||||
|
border: 1px solid #4b5563;
|
||||||
|
border-radius: 8px;
|
||||||
|
background: #0b1220;
|
||||||
|
box-shadow: 0 18px 40px rgba(0, 0, 0, 0.32);
|
||||||
|
}
|
||||||
|
|
||||||
|
.session-option {
|
||||||
|
width: 100%;
|
||||||
|
display: grid;
|
||||||
|
gap: 4px;
|
||||||
|
padding: 10px;
|
||||||
|
border: 1px solid transparent;
|
||||||
|
border-radius: 6px;
|
||||||
|
background: transparent;
|
||||||
|
color: var(--white);
|
||||||
|
text-align: left;
|
||||||
|
}
|
||||||
|
|
||||||
|
.session-option:hover,
|
||||||
|
.session-option.active {
|
||||||
|
border-color: #2f8375;
|
||||||
|
background: rgba(31, 122, 107, 0.18);
|
||||||
|
}
|
||||||
|
|
||||||
|
.session-option-title {
|
||||||
|
font-size: 13px;
|
||||||
|
font-weight: 720;
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.session-option-meta {
|
||||||
|
display: flex;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
gap: 8px;
|
||||||
|
color: #94a3b8;
|
||||||
|
font-family: ui-monospace, SFMono-Regular, Consolas, "Liberation Mono", monospace;
|
||||||
|
font-size: 11px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.session-option-empty {
|
||||||
|
padding: 12px;
|
||||||
|
color: #94a3b8;
|
||||||
|
font-size: 13px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.session-control > button {
|
||||||
height: 42px;
|
height: 42px;
|
||||||
display: inline-flex;
|
display: inline-flex;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
@@ -149,7 +223,7 @@ svg {
|
|||||||
font-weight: 700;
|
font-weight: 700;
|
||||||
}
|
}
|
||||||
|
|
||||||
.session-control button:disabled {
|
.session-control > button:disabled {
|
||||||
opacity: 0.62;
|
opacity: 0.62;
|
||||||
cursor: wait;
|
cursor: wait;
|
||||||
}
|
}
|
||||||
@@ -176,6 +250,132 @@ svg {
|
|||||||
background: var(--evidence-soft);
|
background: var(--evidence-soft);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.outcome-panel {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: minmax(0, 1.2fr) minmax(320px, 0.8fr);
|
||||||
|
gap: 0;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: 8px;
|
||||||
|
overflow: hidden;
|
||||||
|
background: var(--white);
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-main {
|
||||||
|
min-width: 0;
|
||||||
|
padding: 20px;
|
||||||
|
border-left: 7px solid var(--steel);
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-panel.ok .outcome-main {
|
||||||
|
border-left-color: var(--evidence);
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-panel.warn .outcome-main {
|
||||||
|
border-left-color: var(--warn);
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-panel.bad .outcome-main {
|
||||||
|
border-left-color: var(--reject);
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-eyebrow {
|
||||||
|
margin: 0 0 8px;
|
||||||
|
color: var(--steel);
|
||||||
|
font-family: ui-monospace, SFMono-Regular, Consolas, "Liberation Mono", monospace;
|
||||||
|
font-size: 12px;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-main h2 {
|
||||||
|
margin: 0;
|
||||||
|
font-size: clamp(24px, 3vw, 38px);
|
||||||
|
line-height: 1.08;
|
||||||
|
letter-spacing: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-main > p:not(.outcome-eyebrow) {
|
||||||
|
margin: 10px 0 0;
|
||||||
|
color: var(--steel);
|
||||||
|
line-height: 1.5;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-answer {
|
||||||
|
margin-top: 16px;
|
||||||
|
padding: 14px 16px;
|
||||||
|
max-height: 180px;
|
||||||
|
overflow: auto;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: 6px;
|
||||||
|
background: #fffaf1;
|
||||||
|
color: #172033;
|
||||||
|
line-height: 1.65;
|
||||||
|
white-space: pre-wrap;
|
||||||
|
overflow-wrap: anywhere;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-side {
|
||||||
|
display: grid;
|
||||||
|
grid-template-rows: auto 1fr;
|
||||||
|
border-left: 1px solid var(--line);
|
||||||
|
background: #fbf7ef;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-verdict {
|
||||||
|
padding: 18px;
|
||||||
|
border-bottom: 1px solid var(--line);
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-verdict span,
|
||||||
|
.outcome-insights h3 {
|
||||||
|
color: var(--steel);
|
||||||
|
font-family: ui-monospace, SFMono-Regular, Consolas, "Liberation Mono", monospace;
|
||||||
|
font-size: 12px;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-verdict strong {
|
||||||
|
display: block;
|
||||||
|
margin-top: 6px;
|
||||||
|
font-size: 30px;
|
||||||
|
line-height: 1.05;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-verdict p {
|
||||||
|
margin: 10px 0 0;
|
||||||
|
color: var(--ink-soft);
|
||||||
|
line-height: 1.5;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-insights {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: 1fr 1fr;
|
||||||
|
gap: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-insights > div {
|
||||||
|
min-width: 0;
|
||||||
|
padding: 16px 18px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-insights > div + div {
|
||||||
|
border-left: 1px solid var(--line);
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-insights h3 {
|
||||||
|
margin: 0 0 10px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-insights ul {
|
||||||
|
margin: 0;
|
||||||
|
padding-left: 18px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-insights li {
|
||||||
|
margin-bottom: 8px;
|
||||||
|
line-height: 1.45;
|
||||||
|
overflow-wrap: anywhere;
|
||||||
|
}
|
||||||
|
|
||||||
.summary-strip {
|
.summary-strip {
|
||||||
display: grid;
|
display: grid;
|
||||||
grid-template-columns: repeat(7, minmax(120px, 1fr));
|
grid-template-columns: repeat(7, minmax(120px, 1fr));
|
||||||
@@ -547,6 +747,15 @@ pre,
|
|||||||
}
|
}
|
||||||
|
|
||||||
@media (max-width: 1180px) {
|
@media (max-width: 1180px) {
|
||||||
|
.outcome-panel {
|
||||||
|
grid-template-columns: 1fr;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-side {
|
||||||
|
border-left: 0;
|
||||||
|
border-top: 1px solid var(--line);
|
||||||
|
}
|
||||||
|
|
||||||
.summary-strip {
|
.summary-strip {
|
||||||
grid-template-columns: repeat(4, minmax(120px, 1fr));
|
grid-template-columns: repeat(4, minmax(120px, 1fr));
|
||||||
}
|
}
|
||||||
@@ -581,6 +790,15 @@ pre,
|
|||||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.outcome-insights {
|
||||||
|
grid-template-columns: 1fr;
|
||||||
|
}
|
||||||
|
|
||||||
|
.outcome-insights > div + div {
|
||||||
|
border-left: 0;
|
||||||
|
border-top: 1px solid var(--line);
|
||||||
|
}
|
||||||
|
|
||||||
.summary-cell,
|
.summary-cell,
|
||||||
.summary-cell:nth-child(4) {
|
.summary-cell:nth-child(4) {
|
||||||
border-right: 1px solid var(--line);
|
border-right: 1px solid var(--line);
|
||||||
|
|||||||
@@ -1,67 +1,102 @@
|
|||||||
<!DOCTYPE html>
|
<!DOCTYPE html>
|
||||||
<html lang="en">
|
<html lang="zh-CN">
|
||||||
<head>
|
<head>
|
||||||
<meta charset="UTF-8">
|
<meta charset="UTF-8">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
<title>Trace Workbench - SuperBizAgent</title>
|
<title>诊断链路检查台 - SuperBizAgent</title>
|
||||||
<link rel="stylesheet" href="trace.css">
|
<link rel="icon" href="data:,">
|
||||||
|
<link rel="stylesheet" href="trace.css?v=20260707-outcome">
|
||||||
</head>
|
</head>
|
||||||
<body>
|
<body>
|
||||||
<main class="trace-app">
|
<main class="trace-app">
|
||||||
<header class="trace-header">
|
<header class="trace-header">
|
||||||
<div>
|
<div>
|
||||||
<a class="trace-back-link" href="index.html" aria-label="Back to chat">
|
<a class="trace-back-link" href="index.html" aria-label="返回聊天">
|
||||||
<svg viewBox="0 0 24 24" aria-hidden="true">
|
<svg viewBox="0 0 24 24" aria-hidden="true">
|
||||||
<path d="M19 12H5M12 19L5 12L12 5" />
|
<path d="M19 12H5M12 19L5 12L12 5" />
|
||||||
</svg>
|
</svg>
|
||||||
<span>Chat</span>
|
<span>返回聊天</span>
|
||||||
</a>
|
</a>
|
||||||
<h1>Trace workbench</h1>
|
<h1>诊断链路检查台</h1>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<form class="session-form" id="sessionForm">
|
<form class="session-form" id="sessionForm">
|
||||||
<label for="sessionIdInput">Session id</label>
|
<label for="sessionIdInput">会话 ID</label>
|
||||||
<div class="session-control">
|
<div class="session-control">
|
||||||
<input id="sessionIdInput" name="sessionId" autocomplete="off" spellcheck="false" placeholder="e2e-mysql-skill-rag-20260706-2355">
|
<div class="session-picker">
|
||||||
|
<input id="sessionIdInput" name="sessionId" autocomplete="off" spellcheck="false" placeholder="e2e-mysql-skill-rag-20260706-2355" aria-controls="sessionOptions" aria-expanded="false">
|
||||||
|
<button class="session-picker-button" id="sessionPickerButton" type="button" aria-label="选择最近会话">
|
||||||
|
<svg viewBox="0 0 24 24" aria-hidden="true">
|
||||||
|
<path d="M6 9L12 15L18 9" />
|
||||||
|
</svg>
|
||||||
|
</button>
|
||||||
|
<div class="session-options" id="sessionOptions" role="listbox" aria-label="最近会话" hidden></div>
|
||||||
|
</div>
|
||||||
<button type="submit" id="loadButton">
|
<button type="submit" id="loadButton">
|
||||||
<svg viewBox="0 0 24 24" aria-hidden="true">
|
<svg viewBox="0 0 24 24" aria-hidden="true">
|
||||||
<path d="M5 12H19M13 6L19 12L13 18" />
|
<path d="M5 12H19M13 6L19 12L13 18" />
|
||||||
</svg>
|
</svg>
|
||||||
<span>Load</span>
|
<span>加载</span>
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
</form>
|
</form>
|
||||||
</header>
|
</header>
|
||||||
|
|
||||||
<section class="state-line" id="stateLine" role="status">Enter a session id to load a persisted diagnosis trace.</section>
|
<section class="state-line" id="stateLine" role="status">输入会话 ID,加载已持久化的诊断链路。</section>
|
||||||
|
|
||||||
<section class="summary-strip" id="summaryStrip" aria-label="Trace summary">
|
<section class="outcome-panel" id="outcomePanel" aria-label="会话结果">
|
||||||
|
<div class="outcome-main">
|
||||||
|
<p class="outcome-eyebrow">本次会话结果</p>
|
||||||
|
<h2 id="outcomeTitle">尚未加载会话</h2>
|
||||||
|
<p id="outcomeSubtitle">选择一个会话后,这里会先展示最终结论和证据状态。</p>
|
||||||
|
<div class="outcome-answer" id="outcomeAnswerSummary">-</div>
|
||||||
|
</div>
|
||||||
|
<div class="outcome-side">
|
||||||
|
<div class="outcome-verdict">
|
||||||
|
<span>校验判断</span>
|
||||||
|
<strong id="outcomeVerdict">-</strong>
|
||||||
|
<p id="outcomeEvidenceState">-</p>
|
||||||
|
</div>
|
||||||
|
<div class="outcome-insights">
|
||||||
|
<div>
|
||||||
|
<h3>已确认</h3>
|
||||||
|
<ul id="outcomeConfirmed"></ul>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<h3>仍需关注</h3>
|
||||||
|
<ul id="outcomeGaps"></ul>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="summary-strip" id="summaryStrip" aria-label="诊断链路汇总">
|
||||||
<div class="summary-cell">
|
<div class="summary-cell">
|
||||||
<span>Status</span>
|
<span>状态</span>
|
||||||
<strong id="summaryStatus">-</strong>
|
<strong id="summaryStatus">-</strong>
|
||||||
</div>
|
</div>
|
||||||
<div class="summary-cell">
|
<div class="summary-cell">
|
||||||
<span>Verdict</span>
|
<span>校验结论</span>
|
||||||
<strong id="summaryVerdict">-</strong>
|
<strong id="summaryVerdict">-</strong>
|
||||||
</div>
|
</div>
|
||||||
<div class="summary-cell">
|
<div class="summary-cell">
|
||||||
<span>Groundedness</span>
|
<span>证据分</span>
|
||||||
<strong id="summaryScore">-</strong>
|
<strong id="summaryScore">-</strong>
|
||||||
</div>
|
</div>
|
||||||
<div class="summary-cell">
|
<div class="summary-cell">
|
||||||
<span>Steps</span>
|
<span>步骤</span>
|
||||||
<strong id="summarySteps">-</strong>
|
<strong id="summarySteps">-</strong>
|
||||||
</div>
|
</div>
|
||||||
<div class="summary-cell">
|
<div class="summary-cell">
|
||||||
<span>Tools</span>
|
<span>工具</span>
|
||||||
<strong id="summaryTools">-</strong>
|
<strong id="summaryTools">-</strong>
|
||||||
</div>
|
</div>
|
||||||
<div class="summary-cell">
|
<div class="summary-cell">
|
||||||
<span>Duration</span>
|
<span>耗时</span>
|
||||||
<strong id="summaryDuration">-</strong>
|
<strong id="summaryDuration">-</strong>
|
||||||
</div>
|
</div>
|
||||||
<div class="summary-cell">
|
<div class="summary-cell">
|
||||||
<span>Tokens</span>
|
<span>令牌</span>
|
||||||
<strong id="summaryTokens">-</strong>
|
<strong id="summaryTokens">-</strong>
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
@@ -70,8 +105,8 @@
|
|||||||
<section class="trace-pane agent-pane" aria-labelledby="agentPaneTitle">
|
<section class="trace-pane agent-pane" aria-labelledby="agentPaneTitle">
|
||||||
<div class="pane-heading">
|
<div class="pane-heading">
|
||||||
<div>
|
<div>
|
||||||
<p>Agent rail</p>
|
<p>智能体轨迹</p>
|
||||||
<h2 id="agentPaneTitle">Persisted steps</h2>
|
<h2 id="agentPaneTitle">持久化步骤</h2>
|
||||||
</div>
|
</div>
|
||||||
<span class="pane-count" id="stepCount">0</span>
|
<span class="pane-count" id="stepCount">0</span>
|
||||||
</div>
|
</div>
|
||||||
@@ -81,8 +116,8 @@
|
|||||||
<section class="trace-pane evidence-pane" aria-labelledby="evidencePaneTitle">
|
<section class="trace-pane evidence-pane" aria-labelledby="evidencePaneTitle">
|
||||||
<div class="pane-heading">
|
<div class="pane-heading">
|
||||||
<div>
|
<div>
|
||||||
<p>Evidence ledger</p>
|
<p>证据账本</p>
|
||||||
<h2 id="evidencePaneTitle">Tool invocations</h2>
|
<h2 id="evidencePaneTitle">工具调用</h2>
|
||||||
</div>
|
</div>
|
||||||
<span class="pane-count" id="toolCount">0</span>
|
<span class="pane-count" id="toolCount">0</span>
|
||||||
</div>
|
</div>
|
||||||
@@ -93,30 +128,30 @@
|
|||||||
<section class="trace-pane inspector-pane" aria-labelledby="inspectorPaneTitle">
|
<section class="trace-pane inspector-pane" aria-labelledby="inspectorPaneTitle">
|
||||||
<div class="pane-heading">
|
<div class="pane-heading">
|
||||||
<div>
|
<div>
|
||||||
<p>Inspector</p>
|
<p>检查器</p>
|
||||||
<h2 id="inspectorPaneTitle">Verifier and RAG</h2>
|
<h2 id="inspectorPaneTitle">校验与检索增强</h2>
|
||||||
</div>
|
</div>
|
||||||
<span class="pane-count" id="inspectorMode">idle</span>
|
<span class="pane-count" id="inspectorMode">空闲</span>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="inspector-stack">
|
<div class="inspector-stack">
|
||||||
<section class="inspector-section" aria-labelledby="skillBoundaryTitle">
|
<section class="inspector-section" aria-labelledby="skillBoundaryTitle">
|
||||||
<h3 id="skillBoundaryTitle">Skill boundary</h3>
|
<h3 id="skillBoundaryTitle">技能边界</h3>
|
||||||
<div class="boundary-grid" id="skillBoundary"></div>
|
<div class="boundary-grid" id="skillBoundary"></div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section class="inspector-section" aria-labelledby="verifierTitle">
|
<section class="inspector-section" aria-labelledby="verifierTitle">
|
||||||
<h3 id="verifierTitle">Verifier facts</h3>
|
<h3 id="verifierTitle">校验事实</h3>
|
||||||
<div id="verifierFacts"></div>
|
<div id="verifierFacts"></div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section class="inspector-section" aria-labelledby="toolInspectorTitle">
|
<section class="inspector-section" aria-labelledby="toolInspectorTitle">
|
||||||
<h3 id="toolInspectorTitle">Selected tool</h3>
|
<h3 id="toolInspectorTitle">选中工具</h3>
|
||||||
<div id="toolInspector"></div>
|
<div id="toolInspector"></div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section class="inspector-section" aria-labelledby="answerTitle">
|
<section class="inspector-section" aria-labelledby="answerTitle">
|
||||||
<h3 id="answerTitle">Final answer</h3>
|
<h3 id="answerTitle">最终回答</h3>
|
||||||
<div class="answer-box" id="finalAnswer">-</div>
|
<div class="answer-box" id="finalAnswer">-</div>
|
||||||
</section>
|
</section>
|
||||||
</div>
|
</div>
|
||||||
@@ -124,6 +159,6 @@
|
|||||||
</section>
|
</section>
|
||||||
</main>
|
</main>
|
||||||
|
|
||||||
<script src="trace.js"></script>
|
<script src="trace.js?v=20260707-outcome"></script>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ class TraceWorkbench {
|
|||||||
|
|
||||||
this.sessionForm = document.getElementById('sessionForm');
|
this.sessionForm = document.getElementById('sessionForm');
|
||||||
this.sessionIdInput = document.getElementById('sessionIdInput');
|
this.sessionIdInput = document.getElementById('sessionIdInput');
|
||||||
|
this.sessionPickerButton = document.getElementById('sessionPickerButton');
|
||||||
|
this.sessionOptions = document.getElementById('sessionOptions');
|
||||||
this.loadButton = document.getElementById('loadButton');
|
this.loadButton = document.getElementById('loadButton');
|
||||||
this.stateLine = document.getElementById('stateLine');
|
this.stateLine = document.getElementById('stateLine');
|
||||||
this.stepsRail = document.getElementById('stepsRail');
|
this.stepsRail = document.getElementById('stepsRail');
|
||||||
@@ -15,6 +17,17 @@ class TraceWorkbench {
|
|||||||
this.verifierFacts = document.getElementById('verifierFacts');
|
this.verifierFacts = document.getElementById('verifierFacts');
|
||||||
this.toolInspector = document.getElementById('toolInspector');
|
this.toolInspector = document.getElementById('toolInspector');
|
||||||
this.finalAnswer = document.getElementById('finalAnswer');
|
this.finalAnswer = document.getElementById('finalAnswer');
|
||||||
|
this.outcomePanel = document.getElementById('outcomePanel');
|
||||||
|
this.outcomeTitle = document.getElementById('outcomeTitle');
|
||||||
|
this.outcomeSubtitle = document.getElementById('outcomeSubtitle');
|
||||||
|
this.outcomeVerdict = document.getElementById('outcomeVerdict');
|
||||||
|
this.outcomeEvidenceState = document.getElementById('outcomeEvidenceState');
|
||||||
|
this.outcomeAnswerSummary = document.getElementById('outcomeAnswerSummary');
|
||||||
|
this.outcomeConfirmed = document.getElementById('outcomeConfirmed');
|
||||||
|
this.outcomeGaps = document.getElementById('outcomeGaps');
|
||||||
|
this.sessionCandidates = [];
|
||||||
|
this.filteredSessionCandidates = [];
|
||||||
|
this.activeSessionIndex = 0;
|
||||||
|
|
||||||
this.bindEvents();
|
this.bindEvents();
|
||||||
this.bootstrapFromUrl();
|
this.bootstrapFromUrl();
|
||||||
@@ -25,11 +38,34 @@ class TraceWorkbench {
|
|||||||
event.preventDefault();
|
event.preventDefault();
|
||||||
this.loadTrace(this.sessionIdInput.value.trim());
|
this.loadTrace(this.sessionIdInput.value.trim());
|
||||||
});
|
});
|
||||||
|
|
||||||
|
this.sessionIdInput.addEventListener('focus', () => {
|
||||||
|
this.renderSessionOptions(this.sessionIdInput.value);
|
||||||
|
this.openSessionOptions();
|
||||||
|
});
|
||||||
|
|
||||||
|
this.sessionIdInput.addEventListener('input', () => {
|
||||||
|
this.renderSessionOptions(this.sessionIdInput.value);
|
||||||
|
this.openSessionOptions();
|
||||||
|
});
|
||||||
|
|
||||||
|
this.sessionIdInput.addEventListener('keydown', (event) => this.handleSessionInputKeydown(event));
|
||||||
|
|
||||||
|
this.sessionPickerButton.addEventListener('click', () => {
|
||||||
|
this.renderSessionOptions(this.sessionIdInput.value);
|
||||||
|
this.toggleSessionOptions();
|
||||||
|
});
|
||||||
|
|
||||||
|
document.addEventListener('click', (event) => {
|
||||||
|
if (!this.sessionForm.contains(event.target)) {
|
||||||
|
this.closeSessionOptions();
|
||||||
|
}
|
||||||
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
bootstrapFromUrl() {
|
bootstrapFromUrl() {
|
||||||
const params = new URLSearchParams(window.location.search);
|
const params = new URLSearchParams(window.location.search);
|
||||||
const sessionId = params.get('sessionId');
|
const sessionId = params.get('sessionId') || this.getLastTraceSessionId();
|
||||||
if (sessionId) {
|
if (sessionId) {
|
||||||
this.sessionIdInput.value = sessionId;
|
this.sessionIdInput.value = sessionId;
|
||||||
this.loadTrace(sessionId);
|
this.loadTrace(sessionId);
|
||||||
@@ -38,13 +74,204 @@ class TraceWorkbench {
|
|||||||
this.renderEmpty();
|
this.renderEmpty();
|
||||||
}
|
}
|
||||||
|
|
||||||
async loadTrace(sessionId) {
|
getLastTraceSessionId() {
|
||||||
if (!sessionId) {
|
try {
|
||||||
this.setState('Enter a session id before loading.', 'error');
|
const remembered = localStorage.getItem('lastTraceSessionId');
|
||||||
|
if (remembered) {
|
||||||
|
return remembered;
|
||||||
|
}
|
||||||
|
const histories = JSON.parse(localStorage.getItem('chatHistories') || '[]');
|
||||||
|
const recent = Array.isArray(histories) ? histories.find(history => history && history.id) : null;
|
||||||
|
return recent ? recent.id : '';
|
||||||
|
} catch (error) {
|
||||||
|
return '';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
loadSessionCandidates() {
|
||||||
|
const candidates = [];
|
||||||
|
const seen = new Set();
|
||||||
|
const addCandidate = (candidate) => {
|
||||||
|
if (!candidate || !candidate.id || seen.has(candidate.id)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
seen.add(candidate.id);
|
||||||
|
candidates.push(candidate);
|
||||||
|
};
|
||||||
|
|
||||||
|
try {
|
||||||
|
const remembered = localStorage.getItem('lastTraceSessionId');
|
||||||
|
const histories = JSON.parse(localStorage.getItem('chatHistories') || '[]');
|
||||||
|
if (Array.isArray(histories)) {
|
||||||
|
histories.forEach((history) => addCandidate({
|
||||||
|
id: history.id,
|
||||||
|
title: history.title || '未命名对话',
|
||||||
|
updatedAt: history.updatedAt || history.createdAt || '',
|
||||||
|
source: 'history'
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
if (remembered) {
|
||||||
|
const existing = candidates.find(candidate => candidate.id === remembered);
|
||||||
|
if (existing) {
|
||||||
|
existing.source = 'recent';
|
||||||
|
} else {
|
||||||
|
addCandidate({
|
||||||
|
id: remembered,
|
||||||
|
title: '最近查看的会话',
|
||||||
|
updatedAt: '',
|
||||||
|
source: 'recent'
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
return candidates;
|
||||||
|
}
|
||||||
|
|
||||||
|
return candidates
|
||||||
|
.sort((left, right) => {
|
||||||
|
if (left.source === 'recent' && right.source !== 'recent') {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
if (right.source === 'recent' && left.source !== 'recent') {
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
return this.timestampOf(right.updatedAt) - this.timestampOf(left.updatedAt);
|
||||||
|
})
|
||||||
|
.slice(0, 20);
|
||||||
|
}
|
||||||
|
|
||||||
|
timestampOf(value) {
|
||||||
|
if (!value) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
const timestamp = new Date(value).getTime();
|
||||||
|
return Number.isNaN(timestamp) ? 0 : timestamp;
|
||||||
|
}
|
||||||
|
|
||||||
|
renderSessionOptions(query = '') {
|
||||||
|
this.sessionCandidates = this.loadSessionCandidates();
|
||||||
|
const keyword = query.trim().toLowerCase();
|
||||||
|
this.filteredSessionCandidates = keyword
|
||||||
|
? this.sessionCandidates.filter((candidate) => {
|
||||||
|
return candidate.id.toLowerCase().includes(keyword)
|
||||||
|
|| candidate.title.toLowerCase().includes(keyword);
|
||||||
|
})
|
||||||
|
: this.sessionCandidates;
|
||||||
|
this.activeSessionIndex = 0;
|
||||||
|
|
||||||
|
if (!this.filteredSessionCandidates.length) {
|
||||||
|
this.sessionOptions.innerHTML = '<div class="session-option-empty">没有匹配的最近会话</div>';
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
this.setState(`Loading ${sessionId}...`, 'loading');
|
this.sessionOptions.innerHTML = this.filteredSessionCandidates.map((candidate, index) => `
|
||||||
|
<button class="session-option ${index === this.activeSessionIndex ? 'active' : ''}" type="button" role="option" data-session-id="${this.escape(candidate.id)}">
|
||||||
|
<span class="session-option-title">${this.escape(candidate.title)}</span>
|
||||||
|
<span class="session-option-meta">
|
||||||
|
<span>${this.escape(this.shortSessionId(candidate.id))}</span>
|
||||||
|
<span>${this.escape(this.formatOptionDate(candidate.updatedAt))}</span>
|
||||||
|
</span>
|
||||||
|
</button>
|
||||||
|
`).join('');
|
||||||
|
|
||||||
|
this.sessionOptions.querySelectorAll('.session-option').forEach((button) => {
|
||||||
|
button.addEventListener('click', () => this.chooseSession(button.dataset.sessionId));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
openSessionOptions() {
|
||||||
|
this.sessionOptions.hidden = false;
|
||||||
|
this.sessionIdInput.setAttribute('aria-expanded', 'true');
|
||||||
|
}
|
||||||
|
|
||||||
|
closeSessionOptions() {
|
||||||
|
this.sessionOptions.hidden = true;
|
||||||
|
this.sessionIdInput.setAttribute('aria-expanded', 'false');
|
||||||
|
}
|
||||||
|
|
||||||
|
toggleSessionOptions() {
|
||||||
|
if (this.sessionOptions.hidden) {
|
||||||
|
this.openSessionOptions();
|
||||||
|
this.sessionIdInput.focus();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.closeSessionOptions();
|
||||||
|
}
|
||||||
|
|
||||||
|
handleSessionInputKeydown(event) {
|
||||||
|
if (event.key === 'Escape') {
|
||||||
|
this.closeSessionOptions();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (this.sessionOptions.hidden || !this.filteredSessionCandidates.length) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (event.key === 'ArrowDown') {
|
||||||
|
event.preventDefault();
|
||||||
|
this.activeSessionIndex = Math.min(this.activeSessionIndex + 1, this.filteredSessionCandidates.length - 1);
|
||||||
|
this.markActiveSessionOption();
|
||||||
|
} else if (event.key === 'ArrowUp') {
|
||||||
|
event.preventDefault();
|
||||||
|
this.activeSessionIndex = Math.max(this.activeSessionIndex - 1, 0);
|
||||||
|
this.markActiveSessionOption();
|
||||||
|
} else if (event.key === 'Enter') {
|
||||||
|
event.preventDefault();
|
||||||
|
const candidate = this.filteredSessionCandidates[this.activeSessionIndex];
|
||||||
|
if (candidate) {
|
||||||
|
this.chooseSession(candidate.id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
markActiveSessionOption() {
|
||||||
|
this.sessionOptions.querySelectorAll('.session-option').forEach((button, index) => {
|
||||||
|
button.classList.toggle('active', index === this.activeSessionIndex);
|
||||||
|
if (index === this.activeSessionIndex) {
|
||||||
|
button.scrollIntoView({ block: 'nearest' });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
chooseSession(sessionId) {
|
||||||
|
this.sessionIdInput.value = sessionId;
|
||||||
|
this.closeSessionOptions();
|
||||||
|
this.loadTrace(sessionId);
|
||||||
|
}
|
||||||
|
|
||||||
|
shortSessionId(sessionId) {
|
||||||
|
if (!sessionId || sessionId.length <= 34) {
|
||||||
|
return sessionId || '-';
|
||||||
|
}
|
||||||
|
return `${sessionId.slice(0, 18)}...${sessionId.slice(-10)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
formatOptionDate(value) {
|
||||||
|
if (!value) {
|
||||||
|
return '最近';
|
||||||
|
}
|
||||||
|
const date = new Date(value);
|
||||||
|
if (Number.isNaN(date.getTime())) {
|
||||||
|
return '最近';
|
||||||
|
}
|
||||||
|
return date.toLocaleString('zh-CN', {
|
||||||
|
month: '2-digit',
|
||||||
|
day: '2-digit',
|
||||||
|
hour: '2-digit',
|
||||||
|
minute: '2-digit',
|
||||||
|
hour12: false
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async loadTrace(sessionId) {
|
||||||
|
if (!sessionId) {
|
||||||
|
this.setState('请先输入会话 ID。', 'error');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
this.setState(`正在加载 ${sessionId}...`, 'loading');
|
||||||
|
this.updateUrl(sessionId);
|
||||||
this.loadButton.disabled = true;
|
this.loadButton.disabled = true;
|
||||||
|
|
||||||
try {
|
try {
|
||||||
@@ -55,9 +282,13 @@ class TraceWorkbench {
|
|||||||
? data.toolInvocations[0].id
|
? data.toolInvocations[0].id
|
||||||
: null;
|
: null;
|
||||||
this.activeFilter = 'all';
|
this.activeFilter = 'all';
|
||||||
this.updateUrl(sessionId);
|
try {
|
||||||
|
localStorage.setItem('lastTraceSessionId', sessionId);
|
||||||
|
} catch (error) {
|
||||||
|
// ignore storage failures
|
||||||
|
}
|
||||||
this.renderTrace();
|
this.renderTrace();
|
||||||
this.setState(`Loaded ${sessionId}.`, '');
|
this.setState(`已加载 ${sessionId}。`, '');
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
this.trace = null;
|
this.trace = null;
|
||||||
this.renderEmpty();
|
this.renderEmpty();
|
||||||
@@ -72,16 +303,27 @@ class TraceWorkbench {
|
|||||||
try {
|
try {
|
||||||
payload = await response.json();
|
payload = await response.json();
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
throw new Error(`HTTP ${response.status}: response is not JSON`);
|
throw new Error(`HTTP ${response.status}: 响应不是 JSON`);
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!response.ok || payload.code !== 200) {
|
if (!response.ok || payload.code !== 200) {
|
||||||
throw new Error(payload.message || `HTTP ${response.status}: trace request failed`);
|
throw new Error(this.localizeError(payload.message) || `HTTP ${response.status}: 诊断链路请求失败`);
|
||||||
}
|
}
|
||||||
|
|
||||||
return payload.data;
|
return payload.data;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
localizeError(message) {
|
||||||
|
if (!message) {
|
||||||
|
return '';
|
||||||
|
}
|
||||||
|
const value = String(message);
|
||||||
|
if (value.startsWith('Session not found:')) {
|
||||||
|
return value.replace('Session not found:', '未找到会话:');
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
updateUrl(sessionId) {
|
updateUrl(sessionId) {
|
||||||
const url = new URL(window.location.href);
|
const url = new URL(window.location.href);
|
||||||
url.searchParams.set('sessionId', sessionId);
|
url.searchParams.set('sessionId', sessionId);
|
||||||
@@ -95,8 +337,8 @@ class TraceWorkbench {
|
|||||||
const steps = this.trace.steps || [];
|
const steps = this.trace.steps || [];
|
||||||
const tools = this.trace.toolInvocations || [];
|
const tools = this.trace.toolInvocations || [];
|
||||||
|
|
||||||
this.text('summaryStatus', session.status || '-');
|
this.text('summaryStatus', this.displayStatus(session.status));
|
||||||
this.text('summaryVerdict', verifier.verdict || '-');
|
this.text('summaryVerdict', this.displayVerdict(verifier.verdict));
|
||||||
this.text('summaryScore', this.formatScore(verifier.groundedness_score));
|
this.text('summaryScore', this.formatScore(verifier.groundedness_score));
|
||||||
this.text('summarySteps', `${summary.returnedStepCount ?? steps.length}/${summary.persistedStepCount ?? session.stepCount ?? '-'}`);
|
this.text('summarySteps', `${summary.returnedStepCount ?? steps.length}/${summary.persistedStepCount ?? session.stepCount ?? '-'}`);
|
||||||
this.text('summaryTools', `${summary.returnedToolCallCount ?? tools.length}/${summary.persistedToolCallCount ?? session.toolCallCount ?? '-'}`);
|
this.text('summaryTools', `${summary.returnedToolCallCount ?? tools.length}/${summary.persistedToolCallCount ?? session.toolCallCount ?? '-'}`);
|
||||||
@@ -104,15 +346,16 @@ class TraceWorkbench {
|
|||||||
this.text('summaryTokens', this.formatNumber(session.totalTokenCount));
|
this.text('summaryTokens', this.formatNumber(session.totalTokenCount));
|
||||||
this.text('stepCount', steps.length);
|
this.text('stepCount', steps.length);
|
||||||
this.text('toolCount', tools.length);
|
this.text('toolCount', tools.length);
|
||||||
this.text('inspectorMode', tools.length ? 'tool' : 'trace');
|
this.text('inspectorMode', tools.length ? '工具' : '链路');
|
||||||
|
|
||||||
|
this.renderOutcome(session, verifier, steps, tools);
|
||||||
this.renderSteps(steps);
|
this.renderSteps(steps);
|
||||||
this.renderToolFilters(tools);
|
this.renderToolFilters(tools);
|
||||||
this.renderTools(tools);
|
this.renderTools(tools);
|
||||||
this.renderSkillBoundary();
|
this.renderSkillBoundary();
|
||||||
this.renderVerifier(verifier);
|
this.renderVerifier(verifier);
|
||||||
this.renderToolInspector();
|
this.renderToolInspector();
|
||||||
this.finalAnswer.textContent = session.answer || 'No final answer persisted.';
|
this.finalAnswer.textContent = session.answer || '未持久化最终回答。';
|
||||||
}
|
}
|
||||||
|
|
||||||
renderEmpty() {
|
renderEmpty() {
|
||||||
@@ -125,25 +368,150 @@ class TraceWorkbench {
|
|||||||
this.text('summaryTokens', '-');
|
this.text('summaryTokens', '-');
|
||||||
this.text('stepCount', '0');
|
this.text('stepCount', '0');
|
||||||
this.text('toolCount', '0');
|
this.text('toolCount', '0');
|
||||||
this.text('inspectorMode', 'idle');
|
this.text('inspectorMode', '空闲');
|
||||||
|
|
||||||
this.stepsRail.innerHTML = '<div class="empty-panel">No trace loaded.</div>';
|
this.renderOutcome({}, {}, [], []);
|
||||||
|
this.stepsRail.innerHTML = '<div class="empty-panel">尚未加载诊断链路。</div>';
|
||||||
this.toolFilters.innerHTML = '';
|
this.toolFilters.innerHTML = '';
|
||||||
this.toolLedger.innerHTML = '<div class="empty-panel">No tool invocations loaded.</div>';
|
this.toolLedger.innerHTML = '<div class="empty-panel">尚未加载工具调用。</div>';
|
||||||
this.skillBoundary.innerHTML = this.renderBoundaryItems({
|
this.skillBoundary.innerHTML = this.renderBoundaryItems({
|
||||||
selectedSkill: '-',
|
selectedSkill: '-',
|
||||||
plannerReadSkill: 0,
|
plannerReadSkill: 0,
|
||||||
executorReadSkill: 0,
|
executorReadSkill: 0,
|
||||||
verifierReadSkill: 0
|
verifierReadSkill: 0
|
||||||
});
|
});
|
||||||
this.verifierFacts.innerHTML = '<div class="empty-panel">No verifier evaluation loaded.</div>';
|
this.verifierFacts.innerHTML = '<div class="empty-panel">尚未加载校验结果。</div>';
|
||||||
this.toolInspector.innerHTML = '<div class="empty-panel">Select a tool invocation after loading a trace.</div>';
|
this.toolInspector.innerHTML = '<div class="empty-panel">加载诊断链路后选择一个工具调用。</div>';
|
||||||
this.finalAnswer.textContent = '-';
|
this.finalAnswer.textContent = '-';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
renderOutcome(session, verifier, steps, tools) {
|
||||||
|
const status = String(session.status || '').toUpperCase();
|
||||||
|
const verdict = String(verifier.verdict || '').toUpperCase();
|
||||||
|
const title = this.outcomeTitleFor(status, verdict);
|
||||||
|
const checked = Array.isArray(verifier.facts_checked) ? verifier.facts_checked : [];
|
||||||
|
const traceSummary = Array.isArray(verifier.tool_trace_summary) ? verifier.tool_trace_summary : [];
|
||||||
|
const tracedFacts = checked.filter(fact => Array.isArray(fact.evidence_refs) && fact.evidence_refs.length).length;
|
||||||
|
const score = this.formatScore(verifier.groundedness_score);
|
||||||
|
const answer = session.answer || '未持久化最终回答。';
|
||||||
|
const subtitleParts = [];
|
||||||
|
|
||||||
|
if (session.sessionId) {
|
||||||
|
subtitleParts.push(`会话 ${session.sessionId}`);
|
||||||
|
}
|
||||||
|
if (session.status) {
|
||||||
|
subtitleParts.push(`状态 ${this.displayStatus(session.status)}`);
|
||||||
|
}
|
||||||
|
if (steps.length || tools.length) {
|
||||||
|
subtitleParts.push(`${steps.length} 个步骤 / ${tools.length} 次工具调用`);
|
||||||
|
}
|
||||||
|
if (session.totalDurationMs !== undefined && session.totalDurationMs !== null) {
|
||||||
|
subtitleParts.push(`耗时 ${this.formatDuration(session.totalDurationMs)}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
this.outcomePanel.className = `outcome-panel ${this.outcomeClass(status, verdict)}`.trim();
|
||||||
|
this.outcomeTitle.textContent = title;
|
||||||
|
this.outcomeSubtitle.textContent = subtitleParts.length
|
||||||
|
? subtitleParts.join(' · ')
|
||||||
|
: '选择一个会话后,这里会先展示最终结论和证据状态。';
|
||||||
|
this.outcomeVerdict.textContent = this.displayVerdict(verifier.verdict);
|
||||||
|
this.outcomeEvidenceState.textContent = checked.length
|
||||||
|
? `证据分 ${score},检查 ${checked.length} 个事实,${tracedFacts} 个事实带引用,${traceSummary.length} 条证据摘要。`
|
||||||
|
: '暂无结构化校验事实。';
|
||||||
|
this.outcomeAnswerSummary.textContent = this.summarizeAnswer(answer);
|
||||||
|
this.renderOutcomeList(this.outcomeConfirmed, this.confirmedFacts(checked), '暂无直接确认的关键事实。');
|
||||||
|
this.renderOutcomeList(this.outcomeGaps, this.evidenceGaps(checked, verdict), '没有明显证据缺口。');
|
||||||
|
}
|
||||||
|
|
||||||
|
outcomeTitleFor(status, verdict) {
|
||||||
|
if (!status && !verdict) {
|
||||||
|
return '尚未加载会话';
|
||||||
|
}
|
||||||
|
if (status === 'FAILED') {
|
||||||
|
return '会话执行失败';
|
||||||
|
}
|
||||||
|
if (verdict.includes('REJECT')) {
|
||||||
|
return '结论被校验器拒绝';
|
||||||
|
}
|
||||||
|
if (verdict.includes('LOW_CONFID')) {
|
||||||
|
return '结论低置信,需要复核';
|
||||||
|
}
|
||||||
|
if (verdict.includes('PASS')) {
|
||||||
|
return '结论已通过校验';
|
||||||
|
}
|
||||||
|
if (status === 'SUCCESS') {
|
||||||
|
return '会话已完成,暂无明确校验结论';
|
||||||
|
}
|
||||||
|
return '会话仍在处理中';
|
||||||
|
}
|
||||||
|
|
||||||
|
outcomeClass(status, verdict) {
|
||||||
|
if (status === 'FAILED' || verdict.includes('REJECT')) {
|
||||||
|
return 'bad';
|
||||||
|
}
|
||||||
|
if (verdict.includes('LOW_CONFID') || status === 'RUNNING' || status === 'PENDING') {
|
||||||
|
return 'warn';
|
||||||
|
}
|
||||||
|
if (status === 'SUCCESS' || verdict.includes('PASS')) {
|
||||||
|
return 'ok';
|
||||||
|
}
|
||||||
|
return '';
|
||||||
|
}
|
||||||
|
|
||||||
|
summarizeAnswer(answer) {
|
||||||
|
const text = String(answer || '').trim();
|
||||||
|
if (!text) {
|
||||||
|
return '-';
|
||||||
|
}
|
||||||
|
const compact = text
|
||||||
|
.replace(/\r/g, '')
|
||||||
|
.split('\n')
|
||||||
|
.map(line => line.trim())
|
||||||
|
.filter(Boolean)
|
||||||
|
.join('\n');
|
||||||
|
return compact.length > 620 ? `${compact.slice(0, 620)}...` : compact;
|
||||||
|
}
|
||||||
|
|
||||||
|
confirmedFacts(facts) {
|
||||||
|
return facts
|
||||||
|
.filter(fact => {
|
||||||
|
const verification = String(fact.verification || '').toUpperCase();
|
||||||
|
return fact.is_critical && (verification.includes('DIRECT') || verification.includes('SUPPORTED'));
|
||||||
|
})
|
||||||
|
.map(fact => fact.fact || fact.detail)
|
||||||
|
.filter(Boolean)
|
||||||
|
.slice(0, 4);
|
||||||
|
}
|
||||||
|
|
||||||
|
evidenceGaps(facts, verdict) {
|
||||||
|
const gaps = facts
|
||||||
|
.filter(fact => {
|
||||||
|
const verification = String(fact.verification || '').toUpperCase();
|
||||||
|
return fact.is_critical && (
|
||||||
|
verification.includes('NO_EVIDENCE')
|
||||||
|
|| verification.includes('UNSUPPORTED')
|
||||||
|
|| verification.includes('CONTRADICT')
|
||||||
|
|| verification.includes('INDIRECT')
|
||||||
|
);
|
||||||
|
})
|
||||||
|
.map(fact => fact.detail ? `${fact.fact}:${fact.detail}` : fact.fact)
|
||||||
|
.filter(Boolean)
|
||||||
|
.slice(0, 4);
|
||||||
|
if (!gaps.length && String(verdict || '').toUpperCase().includes('LOW_CONFID')) {
|
||||||
|
return ['校验器给出低置信结论,建议人工复核关键事实和证据引用。'];
|
||||||
|
}
|
||||||
|
return gaps;
|
||||||
|
}
|
||||||
|
|
||||||
|
renderOutcomeList(node, items, emptyText) {
|
||||||
|
node.innerHTML = (items.length ? items : [emptyText]).map(item => `
|
||||||
|
<li>${this.escape(item)}</li>
|
||||||
|
`).join('');
|
||||||
|
}
|
||||||
|
|
||||||
renderSteps(steps) {
|
renderSteps(steps) {
|
||||||
if (!steps.length) {
|
if (!steps.length) {
|
||||||
this.stepsRail.innerHTML = '<div class="empty-panel">No persisted agent steps.</div>';
|
this.stepsRail.innerHTML = '<div class="empty-panel">没有持久化智能体步骤。</div>';
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -153,18 +521,19 @@ class TraceWorkbench {
|
|||||||
return `
|
return `
|
||||||
<article class="step-entry ${cls}" data-index="${this.escape(String(index + 1).padStart(2, '0'))}">
|
<article class="step-entry ${cls}" data-index="${this.escape(String(index + 1).padStart(2, '0'))}">
|
||||||
<div class="step-title">
|
<div class="step-title">
|
||||||
<h3>${this.escape(step.agentName || 'agent')}</h3>
|
<h3>${this.escape(step.agentName || '智能体')}</h3>
|
||||||
${this.renderBadge(step.hasToolCall ? 'tool' : 'model', step.hasToolCall ? 'warn' : 'ok')}
|
${this.renderBadge(step.hasToolCall ? '工具' : '模型', step.hasToolCall ? 'warn' : 'ok')}
|
||||||
</div>
|
</div>
|
||||||
<div class="meta-line">
|
<div class="meta-line">
|
||||||
<span class="meta-pill">step ${this.escape(step.stepIndex ?? index)}</span>
|
<span class="meta-pill">执行序号 ${this.escape(String(index + 1).padStart(2, '0'))}</span>
|
||||||
|
<span class="meta-pill">内部步 ${this.escape(step.stepIndex ?? index)}</span>
|
||||||
<span class="meta-pill">${this.escape(this.formatDuration(step.durationMs))}</span>
|
<span class="meta-pill">${this.escape(this.formatDuration(step.durationMs))}</span>
|
||||||
<span class="meta-pill">${this.escape(this.formatNumber(step.tokenCount))} tokens</span>
|
<span class="meta-pill">${this.escape(this.formatNumber(step.tokenCount))} 令牌</span>
|
||||||
<span class="meta-pill">${this.escape(this.formatDate(step.createdAt))}</span>
|
<span class="meta-pill">${this.escape(this.formatDate(step.createdAt))}</span>
|
||||||
</div>
|
</div>
|
||||||
${this.renderTextDetail('Model input', step.modelInput)}
|
${this.renderTextDetail('模型输入', step.modelInput)}
|
||||||
${this.renderTextDetail('Model output', step.modelOutput)}
|
${this.renderTextDetail('模型输出', step.modelOutput)}
|
||||||
${this.renderTextDetail('Thought', step.thought)}
|
${this.renderTextDetail('思考过程', step.thought)}
|
||||||
</article>
|
</article>
|
||||||
`;
|
`;
|
||||||
}).join('');
|
}).join('');
|
||||||
@@ -178,9 +547,9 @@ class TraceWorkbench {
|
|||||||
|
|
||||||
const groups = this.groupByTool(tools);
|
const groups = this.groupByTool(tools);
|
||||||
const filters = [
|
const filters = [
|
||||||
{ key: 'all', label: `all ${tools.length}` },
|
{ key: 'all', label: `全部 ${tools.length}` },
|
||||||
{ key: 'success', label: `success ${tools.filter(tool => tool.success !== false).length}` },
|
{ key: 'success', label: `成功 ${tools.filter(tool => tool.success !== false).length}` },
|
||||||
{ key: 'failed', label: `failed ${tools.filter(tool => tool.success === false).length}` },
|
{ key: 'failed', label: `失败 ${tools.filter(tool => tool.success === false).length}` },
|
||||||
...Object.entries(groups).map(([name, count]) => ({ key: `tool:${name}`, label: `${name} ${count}` }))
|
...Object.entries(groups).map(([name, count]) => ({ key: `tool:${name}`, label: `${name} ${count}` }))
|
||||||
];
|
];
|
||||||
|
|
||||||
@@ -202,7 +571,7 @@ class TraceWorkbench {
|
|||||||
renderTools(tools) {
|
renderTools(tools) {
|
||||||
const filtered = tools.filter(tool => this.matchesFilter(tool));
|
const filtered = tools.filter(tool => this.matchesFilter(tool));
|
||||||
if (!filtered.length) {
|
if (!filtered.length) {
|
||||||
this.toolLedger.innerHTML = '<div class="empty-panel">No tool invocations match this filter.</div>';
|
this.toolLedger.innerHTML = '<div class="empty-panel">当前筛选下没有工具调用。</div>';
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -212,13 +581,13 @@ class TraceWorkbench {
|
|||||||
return `
|
return `
|
||||||
<button class="tool-row ${failed ? 'failed' : ''} ${selected ? 'selected' : ''}" data-tool-id="${this.escape(tool.id)}">
|
<button class="tool-row ${failed ? 'failed' : ''} ${selected ? 'selected' : ''}" data-tool-id="${this.escape(tool.id)}">
|
||||||
<div class="tool-title">
|
<div class="tool-title">
|
||||||
<h3>${this.escape(tool.toolName || 'tool')}</h3>
|
<h3>${this.escape(tool.toolName || '工具')}</h3>
|
||||||
${this.renderBadge(failed ? 'failed' : 'ok', failed ? 'bad' : 'ok')}
|
${this.renderBadge(failed ? '失败' : '正常', failed ? 'bad' : 'ok')}
|
||||||
</div>
|
</div>
|
||||||
<div class="tool-meta">
|
<div class="tool-meta">
|
||||||
<span>#${index + 1}</span>
|
<span>#${index + 1}</span>
|
||||||
<span>id ${this.escape(tool.id ?? '-')}</span>
|
<span>ID ${this.escape(tool.id ?? '-')}</span>
|
||||||
<span>step ${this.escape(tool.stepId ?? '-')}</span>
|
<span>步骤 ${this.escape(tool.stepId ?? '-')}</span>
|
||||||
<span>${this.escape(this.formatDuration(tool.durationMs))}</span>
|
<span>${this.escape(this.formatDuration(tool.durationMs))}</span>
|
||||||
${tool.retrievalLayer ? `<span>${this.escape(tool.retrievalLayer)}</span>` : ''}
|
${tool.retrievalLayer ? `<span>${this.escape(tool.retrievalLayer)}</span>` : ''}
|
||||||
${tool.relevanceLevel ? `<span>${this.escape(tool.relevanceLevel)}</span>` : ''}
|
${tool.relevanceLevel ? `<span>${this.escape(tool.relevanceLevel)}</span>` : ''}
|
||||||
@@ -251,19 +620,19 @@ class TraceWorkbench {
|
|||||||
renderBoundaryItems(counts) {
|
renderBoundaryItems(counts) {
|
||||||
return `
|
return `
|
||||||
<div class="boundary-item">
|
<div class="boundary-item">
|
||||||
<span class="field-label">Selected skill</span>
|
<span class="field-label">选中技能</span>
|
||||||
<strong>${this.escape(counts.selectedSkill)}</strong>
|
<strong>${this.escape(counts.selectedSkill)}</strong>
|
||||||
</div>
|
</div>
|
||||||
<div class="boundary-item">
|
<div class="boundary-item">
|
||||||
<span class="field-label">Planner mentions</span>
|
<span class="field-label">规划器提及</span>
|
||||||
<strong>${this.escape(counts.plannerReadSkill)}</strong>
|
<strong>${this.escape(counts.plannerReadSkill)}</strong>
|
||||||
</div>
|
</div>
|
||||||
<div class="boundary-item">
|
<div class="boundary-item">
|
||||||
<span class="field-label">Executor mentions</span>
|
<span class="field-label">执行器提及</span>
|
||||||
<strong>${this.escape(counts.executorReadSkill)}</strong>
|
<strong>${this.escape(counts.executorReadSkill)}</strong>
|
||||||
</div>
|
</div>
|
||||||
<div class="boundary-item">
|
<div class="boundary-item">
|
||||||
<span class="field-label">Verifier mentions</span>
|
<span class="field-label">校验器提及</span>
|
||||||
<strong>${this.escape(counts.verifierReadSkill)}</strong>
|
<strong>${this.escape(counts.verifierReadSkill)}</strong>
|
||||||
</div>
|
</div>
|
||||||
`;
|
`;
|
||||||
@@ -271,7 +640,7 @@ class TraceWorkbench {
|
|||||||
|
|
||||||
renderVerifier(verifier) {
|
renderVerifier(verifier) {
|
||||||
if (!verifier || !Object.keys(verifier).length) {
|
if (!verifier || !Object.keys(verifier).length) {
|
||||||
this.verifierFacts.innerHTML = '<div class="empty-panel">No verifier evaluation in selfEvaluation.</div>';
|
this.verifierFacts.innerHTML = '<div class="empty-panel">自评数据中没有校验结果。</div>';
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -279,15 +648,15 @@ class TraceWorkbench {
|
|||||||
const traceSummary = Array.isArray(verifier.tool_trace_summary) ? verifier.tool_trace_summary : [];
|
const traceSummary = Array.isArray(verifier.tool_trace_summary) ? verifier.tool_trace_summary : [];
|
||||||
const header = `
|
const header = `
|
||||||
<div class="meta-line">
|
<div class="meta-line">
|
||||||
${this.renderBadge(verifier.verdict || 'verdict -', this.verdictClass(verifier.verdict))}
|
${this.renderBadge(this.displayVerdict(verifier.verdict), this.verdictClass(verifier.verdict))}
|
||||||
<span class="meta-pill">score ${this.escape(this.formatScore(verifier.groundedness_score))}</span>
|
<span class="meta-pill">证据分 ${this.escape(this.formatScore(verifier.groundedness_score))}</span>
|
||||||
<span class="meta-pill">facts ${facts.length}</span>
|
<span class="meta-pill">事实 ${facts.length}</span>
|
||||||
<span class="meta-pill">trace refs ${traceSummary.length}</span>
|
<span class="meta-pill">引用 ${traceSummary.length}</span>
|
||||||
</div>
|
</div>
|
||||||
`;
|
`;
|
||||||
|
|
||||||
if (!facts.length) {
|
if (!facts.length) {
|
||||||
this.verifierFacts.innerHTML = header + '<div class="empty-panel">Verifier returned no facts_checked rows.</div>';
|
this.verifierFacts.innerHTML = header + '<div class="empty-panel">校验器没有返回 facts_checked 明细。</div>';
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -297,9 +666,9 @@ class TraceWorkbench {
|
|||||||
<article class="fact-row">
|
<article class="fact-row">
|
||||||
<p>${this.escape(fact.fact || '-')}</p>
|
<p>${this.escape(fact.fact || '-')}</p>
|
||||||
<div class="meta-line">
|
<div class="meta-line">
|
||||||
${this.renderBadge(fact.verification || '-', this.factClass(fact.verification))}
|
${this.renderBadge(this.displayFactVerification(fact.verification), this.factClass(fact.verification))}
|
||||||
<span class="meta-pill">${fact.is_critical ? 'critical' : 'supporting'}</span>
|
<span class="meta-pill">${fact.is_critical ? '关键事实' : '辅助事实'}</span>
|
||||||
<span class="meta-pill">refs ${refs.length}</span>
|
<span class="meta-pill">引用 ${refs.length}</span>
|
||||||
</div>
|
</div>
|
||||||
${fact.detail ? `<pre>${this.escape(fact.detail)}</pre>` : ''}
|
${fact.detail ? `<pre>${this.escape(fact.detail)}</pre>` : ''}
|
||||||
${refs.length ? `<pre>${this.escape(this.prettyJson(refs))}</pre>` : ''}
|
${refs.length ? `<pre>${this.escape(this.prettyJson(refs))}</pre>` : ''}
|
||||||
@@ -312,33 +681,33 @@ class TraceWorkbench {
|
|||||||
const tools = this.trace ? (this.trace.toolInvocations || []) : [];
|
const tools = this.trace ? (this.trace.toolInvocations || []) : [];
|
||||||
const tool = tools.find(item => String(item.id) === String(this.selectedToolId));
|
const tool = tools.find(item => String(item.id) === String(this.selectedToolId));
|
||||||
if (!tool) {
|
if (!tool) {
|
||||||
this.toolInspector.innerHTML = '<div class="empty-panel">No tool invocation selected.</div>';
|
this.toolInspector.innerHTML = '<div class="empty-panel">尚未选择工具调用。</div>';
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
const retrieval = tool.retrievalDetails || {};
|
const retrieval = tool.retrievalDetails || {};
|
||||||
const ragFields = [
|
const ragFields = [
|
||||||
['Input params', tool.inputParams || this.parseJson(tool.inputParamsRaw)],
|
['输入参数', tool.inputParams || this.parseJson(tool.inputParamsRaw)],
|
||||||
['Output preview', tool.outputPreview || '-'],
|
['输出预览', tool.outputPreview || '-'],
|
||||||
['Retrieval layer', tool.retrievalLayer || '-'],
|
['检索层级', tool.retrievalLayer || '-'],
|
||||||
['L0 matches', tool.l0MatchCount ?? '-'],
|
['L0 命中数', tool.l0MatchCount ?? '-'],
|
||||||
['L1 matches', tool.l1MatchCount ?? '-'],
|
['L1 命中数', tool.l1MatchCount ?? '-'],
|
||||||
['Relevance', tool.relevanceLevel || '-'],
|
['相关性', tool.relevanceLevel || '-'],
|
||||||
['Dedup reason', tool.dedupReason || '-'],
|
['去重原因', tool.dedupReason || '-'],
|
||||||
['Query transform', retrieval.query_transform],
|
['查询改写', retrieval.query_transform],
|
||||||
['Retrieval trace', retrieval.retrieval_trace],
|
['检索轨迹', retrieval.retrieval_trace],
|
||||||
['Context pack', retrieval.context_pack_summary],
|
['上下文打包', retrieval.context_pack_summary],
|
||||||
['Rerank trace', retrieval.rerank_trace],
|
['重排轨迹', retrieval.rerank_trace],
|
||||||
['Evidence blocks', retrieval.evidence_blocks]
|
['证据块', retrieval.evidence_blocks]
|
||||||
];
|
];
|
||||||
|
|
||||||
this.toolInspector.innerHTML = `
|
this.toolInspector.innerHTML = `
|
||||||
<div class="meta-line">
|
<div class="meta-line">
|
||||||
${this.renderBadge(tool.success === false ? 'failed' : 'ok', tool.success === false ? 'bad' : 'ok')}
|
${this.renderBadge(tool.success === false ? '失败' : '正常', tool.success === false ? 'bad' : 'ok')}
|
||||||
<span class="meta-pill">${this.escape(tool.toolName || 'tool')}</span>
|
<span class="meta-pill">${this.escape(tool.toolName || '工具')}</span>
|
||||||
<span class="meta-pill">${this.escape(this.formatDuration(tool.durationMs))}</span>
|
<span class="meta-pill">${this.escape(this.formatDuration(tool.durationMs))}</span>
|
||||||
<span class="meta-pill">output ${this.escape(this.formatNumber(tool.outputLength))}</span>
|
<span class="meta-pill">输出 ${this.escape(this.formatNumber(tool.outputLength))}</span>
|
||||||
${tool.truncated ? '<span class="meta-pill">truncated</span>' : ''}
|
${tool.truncated ? '<span class="meta-pill">已截断</span>' : ''}
|
||||||
</div>
|
</div>
|
||||||
${tool.errorMessage ? `<pre>${this.escape(tool.errorMessage)}</pre>` : ''}
|
${tool.errorMessage ? `<pre>${this.escape(tool.errorMessage)}</pre>` : ''}
|
||||||
<div class="field-grid">
|
<div class="field-grid">
|
||||||
@@ -373,6 +742,50 @@ class TraceWorkbench {
|
|||||||
return `<span class="badge ${this.escape(cls || 'ok')}">${this.escape(text)}</span>`;
|
return `<span class="badge ${this.escape(cls || 'ok')}">${this.escape(text)}</span>`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
displayStatus(status) {
|
||||||
|
if (!status) {
|
||||||
|
return '-';
|
||||||
|
}
|
||||||
|
const statuses = {
|
||||||
|
SUCCESS: '成功',
|
||||||
|
FAILED: '失败',
|
||||||
|
RUNNING: '运行中',
|
||||||
|
PENDING: '等待中'
|
||||||
|
};
|
||||||
|
return statuses[String(status).toUpperCase()] || String(status);
|
||||||
|
}
|
||||||
|
|
||||||
|
displayVerdict(verdict) {
|
||||||
|
if (!verdict) {
|
||||||
|
return '-';
|
||||||
|
}
|
||||||
|
const verdicts = {
|
||||||
|
PASS: '通过',
|
||||||
|
REJECT: '拒绝',
|
||||||
|
LOW_CONFID: '低置信',
|
||||||
|
LOW_CONFIDENCE: '低置信',
|
||||||
|
REVIEW: '需复核'
|
||||||
|
};
|
||||||
|
return verdicts[String(verdict).toUpperCase()] || String(verdict);
|
||||||
|
}
|
||||||
|
|
||||||
|
displayFactVerification(verification) {
|
||||||
|
if (!verification) {
|
||||||
|
return '-';
|
||||||
|
}
|
||||||
|
const verifications = {
|
||||||
|
DIRECT_EVIDENCE: '直接证据',
|
||||||
|
INDIRECT_SUPPORT: '间接支持',
|
||||||
|
NO_EVIDENCE: '无证据',
|
||||||
|
UNSUPPORTED: '未支持',
|
||||||
|
CONTRADICTED: '有矛盾',
|
||||||
|
SUPPORTED: '已支持',
|
||||||
|
PASS: '通过',
|
||||||
|
REJECT: '拒绝'
|
||||||
|
};
|
||||||
|
return verifications[String(verification).toUpperCase()] || String(verification);
|
||||||
|
}
|
||||||
|
|
||||||
getVerifier() {
|
getVerifier() {
|
||||||
const evaluation = (this.trace && this.trace.session && this.trace.session.selfEvaluation) || {};
|
const evaluation = (this.trace && this.trace.session && this.trace.session.selfEvaluation) || {};
|
||||||
if (evaluation.verifier_evaluation) {
|
if (evaluation.verifier_evaluation) {
|
||||||
|
|||||||
@@ -73,6 +73,41 @@ class DiagnosisTraceEvaluatorTest {
|
|||||||
assertTrue(result.getFailedChecks().contains("reject output does not use degraded template"));
|
assertTrue(result.getFailedChecks().contains("reject output does not use degraded template"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void evaluateFailsWhenStructuredConfirmedClaimHasNoEvidenceBindings() {
|
||||||
|
DiagnosisEvalCase evalCase = DiagnosisEvalCase.builder()
|
||||||
|
.id("structured-claim-case")
|
||||||
|
.title("Structured claim case")
|
||||||
|
.expectedRootCauseKeywords(List.of())
|
||||||
|
.requiredEvidenceTools(List.of())
|
||||||
|
.allowedVerdicts(List.of("LOW_CONFID", "PASS"))
|
||||||
|
.build();
|
||||||
|
DiagnosisTraceResponse trace = DiagnosisTraceResponse.builder()
|
||||||
|
.session(DiagnosisTraceResponse.SessionTrace.builder()
|
||||||
|
.answer("以下结论基于当前已获取证据,仍存在部分证据缺口,请谨慎参考。")
|
||||||
|
.selfEvaluation(java.util.Map.of(
|
||||||
|
"verifier_evaluation", java.util.Map.of(
|
||||||
|
"verdict", "LOW_CONFID",
|
||||||
|
"executor_structured_output", java.util.Map.of(
|
||||||
|
"claims", java.util.List.of(java.util.Map.of(
|
||||||
|
"claim_id", "claim-unsupported",
|
||||||
|
"claim_text", "OOM 导致连接泄漏",
|
||||||
|
"support_level", "direct",
|
||||||
|
"evidence_bindings", java.util.List.of()
|
||||||
|
))
|
||||||
|
)
|
||||||
|
)))
|
||||||
|
.build())
|
||||||
|
.toolInvocations(List.of())
|
||||||
|
.build();
|
||||||
|
|
||||||
|
DiagnosisEvalResult result = evaluator.evaluate(evalCase, trace);
|
||||||
|
|
||||||
|
assertFalse(result.isPassed());
|
||||||
|
assertTrue(result.getFailedChecks().contains(
|
||||||
|
"executor confirmed claim missing evidence bindings: claim-unsupported"));
|
||||||
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
void reportWriterOutputsJsonAndMarkdown(@TempDir Path tempDir) throws Exception {
|
void reportWriterOutputsJsonAndMarkdown(@TempDir Path tempDir) throws Exception {
|
||||||
DiagnosisEvalReport report = evaluator.evaluate(readCases(), Path.of("mvp/eval/fixtures"));
|
DiagnosisEvalReport report = evaluator.evaluate(readCases(), Path.of("mvp/eval/fixtures"));
|
||||||
|
|||||||
@@ -0,0 +1,181 @@
|
|||||||
|
package com.superbiz.agent.hook;
|
||||||
|
|
||||||
|
import com.alibaba.cloud.ai.graph.RunnableConfig;
|
||||||
|
import com.alibaba.cloud.ai.graph.agent.hook.messages.AgentCommand;
|
||||||
|
import com.fasterxml.jackson.databind.JsonNode;
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import com.superbiz.agent.service.ToolTraceSummaryService;
|
||||||
|
import com.superbiz.agent.util.VerifierContextHolder;
|
||||||
|
import org.junit.jupiter.api.AfterEach;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.springframework.ai.chat.messages.AssistantMessage;
|
||||||
|
import org.springframework.ai.chat.messages.Message;
|
||||||
|
import org.springframework.ai.chat.messages.UserMessage;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||||
|
import static org.mockito.ArgumentMatchers.anyString;
|
||||||
|
import static org.mockito.Mockito.mock;
|
||||||
|
import static org.mockito.Mockito.when;
|
||||||
|
|
||||||
|
class VerifierInputHookTest {
|
||||||
|
|
||||||
|
private final ObjectMapper objectMapper = new ObjectMapper();
|
||||||
|
|
||||||
|
@AfterEach
|
||||||
|
void tearDown() {
|
||||||
|
VerifierContextHolder.clear();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void beforeModelAddsStructuredExecutorOutputWhenJsonContractIsValid() throws Exception {
|
||||||
|
ToolTraceSummaryService traceSummaryService = mock(ToolTraceSummaryService.class);
|
||||||
|
when(traceSummaryService.buildVerifierTraceSummary(anyString(), anyString())).thenReturn(List.of(
|
||||||
|
Map.of("trace_ref", "trace-1", "tool_name", "query_metrics")
|
||||||
|
));
|
||||||
|
VerifierInputHook hook = new VerifierInputHook(traceSummaryService);
|
||||||
|
VerifierContextHolder.setOriginalQuery("分析 MySQL 连接池耗尽");
|
||||||
|
|
||||||
|
String executorOutput = """
|
||||||
|
{
|
||||||
|
"answer_version": "executor_evidence_v1",
|
||||||
|
"diagnosis_summary": "连接池已满,但缺少泄漏证据。",
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"claim_id": "claim-1",
|
||||||
|
"claim_type": "symptom",
|
||||||
|
"claim_text": "连接池 active 达到上限",
|
||||||
|
"support_level": "direct",
|
||||||
|
"evidence_bindings": [
|
||||||
|
{
|
||||||
|
"source_type": "tool_trace",
|
||||||
|
"source_id": "trace-1",
|
||||||
|
"tool_name": "query_metrics",
|
||||||
|
"source_invocation_ids": [101],
|
||||||
|
"evidence_excerpt": "active=50 max=50"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"hypotheses": [],
|
||||||
|
"recommended_actions": [],
|
||||||
|
"missing_info": ["缺少泄漏检测日志"],
|
||||||
|
"user_facing_answer": "已确认连接池 active 达到上限。"
|
||||||
|
}
|
||||||
|
""";
|
||||||
|
|
||||||
|
AgentCommand command = hook.beforeModel(
|
||||||
|
List.of(new AssistantMessage(executorOutput)),
|
||||||
|
RunnableConfig.builder().addMetadata("sessionId", "structured-session").build()
|
||||||
|
);
|
||||||
|
|
||||||
|
JsonNode payload = readPayload(command);
|
||||||
|
assertEquals("valid", payload.path("executor_output_parse_status").path("status").asText());
|
||||||
|
assertEquals("executor_evidence_v1",
|
||||||
|
payload.path("executor_structured_output").path("answer_version").asText());
|
||||||
|
assertEquals("连接池 active 达到上限",
|
||||||
|
payload.path("executor_structured_output").path("claims").get(0).path("claim_text").asText());
|
||||||
|
assertNotNull(VerifierContextHolder.getExecutorStructuredOutput());
|
||||||
|
assertEquals("valid", VerifierContextHolder.getExecutorOutputParseStatus().get("status"));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void beforeModelExtractsStructuredOutputFromPrefixedJsonFence() throws Exception {
|
||||||
|
ToolTraceSummaryService traceSummaryService = mock(ToolTraceSummaryService.class);
|
||||||
|
when(traceSummaryService.buildVerifierTraceSummary(anyString(), anyString())).thenReturn(List.of());
|
||||||
|
VerifierInputHook hook = new VerifierInputHook(traceSummaryService);
|
||||||
|
|
||||||
|
String executorOutput = """
|
||||||
|
现在我已经收集了足够的数据,最终输出如下。
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"answer_version": "executor_evidence_v1",
|
||||||
|
"diagnosis_summary": "已确认连接池 active 达到上限。",
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"claim_id": "claim-1",
|
||||||
|
"claim_type": "symptom",
|
||||||
|
"claim_text": "连接池 active 达到上限",
|
||||||
|
"support_level": "direct",
|
||||||
|
"evidence_bindings": [
|
||||||
|
{
|
||||||
|
"source_type": "tool_trace",
|
||||||
|
"source_id": "trace-1",
|
||||||
|
"tool_name": "query_metrics",
|
||||||
|
"source_invocation_ids": [101],
|
||||||
|
"evidence_excerpt": "active=50 max=50"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"hypotheses": [],
|
||||||
|
"recommended_actions": [],
|
||||||
|
"missing_info": [],
|
||||||
|
"user_facing_answer": "已确认连接池 active 达到上限。"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
""";
|
||||||
|
|
||||||
|
AgentCommand command = hook.beforeModel(
|
||||||
|
List.of(new AssistantMessage(executorOutput)),
|
||||||
|
RunnableConfig.builder().addMetadata("sessionId", "fenced-session").build()
|
||||||
|
);
|
||||||
|
|
||||||
|
JsonNode payload = readPayload(command);
|
||||||
|
assertEquals("valid", payload.path("executor_output_parse_status").path("status").asText());
|
||||||
|
assertEquals("claim-1",
|
||||||
|
payload.path("executor_structured_output").path("claims").get(0).path("claim_id").asText());
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void beforeModelMarksMalformedJsonAndKeepsRawAnswerFallback() throws Exception {
|
||||||
|
ToolTraceSummaryService traceSummaryService = mock(ToolTraceSummaryService.class);
|
||||||
|
when(traceSummaryService.buildVerifierTraceSummary(anyString(), anyString())).thenReturn(List.of());
|
||||||
|
VerifierInputHook hook = new VerifierInputHook(traceSummaryService);
|
||||||
|
|
||||||
|
AgentCommand command = hook.beforeModel(
|
||||||
|
List.of(new AssistantMessage("{\"diagnosis_summary\":\"缺少 claims\"}")),
|
||||||
|
RunnableConfig.builder().addMetadata("sessionId", "malformed-session").build()
|
||||||
|
);
|
||||||
|
|
||||||
|
JsonNode payload = readPayload(command);
|
||||||
|
assertEquals("malformed", payload.path("executor_output_parse_status").path("status").asText());
|
||||||
|
assertTrue(payload.path("executor_structured_output").isNull());
|
||||||
|
assertEquals("{\"diagnosis_summary\":\"缺少 claims\"}", payload.path("executor_final_answer").asText());
|
||||||
|
assertEquals("malformed", VerifierContextHolder.getExecutorOutputParseStatus().get("status"));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void beforeModelMarksPlainTextAsMissingStructuredOutput() throws Exception {
|
||||||
|
ToolTraceSummaryService traceSummaryService = mock(ToolTraceSummaryService.class);
|
||||||
|
when(traceSummaryService.buildVerifierTraceSummary(anyString(), anyString())).thenReturn(List.of());
|
||||||
|
VerifierInputHook hook = new VerifierInputHook(traceSummaryService);
|
||||||
|
|
||||||
|
AgentCommand command = hook.beforeModel(
|
||||||
|
List.of(new AssistantMessage("普通自然语言答案")),
|
||||||
|
RunnableConfig.builder().addMetadata("sessionId", "plain-session").build()
|
||||||
|
);
|
||||||
|
|
||||||
|
JsonNode payload = readPayload(command);
|
||||||
|
assertEquals("missing", payload.path("executor_output_parse_status").path("status").asText());
|
||||||
|
assertTrue(payload.path("executor_structured_output").isNull());
|
||||||
|
assertFalse(payload.path("executor_final_answer").asText().isBlank());
|
||||||
|
}
|
||||||
|
|
||||||
|
private JsonNode readPayload(AgentCommand command) throws Exception {
|
||||||
|
var field = AgentCommand.class.getDeclaredField("messages");
|
||||||
|
field.setAccessible(true);
|
||||||
|
@SuppressWarnings("unchecked")
|
||||||
|
List<Message> messages = (List<Message>) field.get(command);
|
||||||
|
assertEquals(1, messages.size());
|
||||||
|
Message message = messages.get(0);
|
||||||
|
assertTrue(message instanceof UserMessage);
|
||||||
|
return objectMapper.readTree(((UserMessage) message).getText());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -226,6 +226,53 @@ class ChatServiceSequentialAgentTest {
|
|||||||
assertTrue(chatModel.sawVerifierPrompt);
|
assertTrue(chatModel.sawVerifierPrompt);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void verifierReceivesStructuredExecutorPayloadFields() throws Exception {
|
||||||
|
ChatService chatService = createChatService();
|
||||||
|
ScriptedChatModel chatModel = new ScriptedChatModel();
|
||||||
|
chatModel.executorOutput = """
|
||||||
|
{
|
||||||
|
"answer_version": "executor_evidence_v1",
|
||||||
|
"diagnosis_summary": "已确认连接池 active 达到上限。",
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"claim_id": "claim-1",
|
||||||
|
"claim_type": "symptom",
|
||||||
|
"claim_text": "连接池 active 达到上限",
|
||||||
|
"support_level": "direct",
|
||||||
|
"evidence_bindings": [
|
||||||
|
{
|
||||||
|
"source_type": "tool_trace",
|
||||||
|
"source_id": "trace-1",
|
||||||
|
"tool_name": "query_metrics",
|
||||||
|
"source_invocation_ids": [101],
|
||||||
|
"evidence_excerpt": "active=50 max=50"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"hypotheses": [],
|
||||||
|
"recommended_actions": [],
|
||||||
|
"missing_info": [],
|
||||||
|
"user_facing_answer": "已确认连接池 active 达到上限。"
|
||||||
|
}
|
||||||
|
""";
|
||||||
|
|
||||||
|
ChatService.ChatResult result = chatService.executeChatComplex(
|
||||||
|
chatModel,
|
||||||
|
new ToolCallback[0],
|
||||||
|
"请分析 MySQL 连接池耗尽",
|
||||||
|
List.of(),
|
||||||
|
"sequential-structured-executor-session"
|
||||||
|
);
|
||||||
|
|
||||||
|
assertEquals("已确认连接池 active 达到上限。", result.answer());
|
||||||
|
assertTrue(chatModel.verifierPromptText.contains("\"executor_structured_output\""));
|
||||||
|
assertTrue(chatModel.verifierPromptText.contains("\"executor_output_parse_status\""));
|
||||||
|
assertTrue(chatModel.verifierPromptText.contains("\"status\" : \"valid\""));
|
||||||
|
assertTrue(chatModel.verifierPromptText.contains("连接池 active 达到上限"));
|
||||||
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
void buildMethodToolsArrayIncludesLogsAndMetricsWhenAvailable() {
|
void buildMethodToolsArrayIncludesLogsAndMetricsWhenAvailable() {
|
||||||
ChatService chatService = new ChatService();
|
ChatService chatService = new ChatService();
|
||||||
@@ -353,6 +400,7 @@ class ChatServiceSequentialAgentTest {
|
|||||||
private String plannerPromptText = "";
|
private String plannerPromptText = "";
|
||||||
private String executorPromptText = "";
|
private String executorPromptText = "";
|
||||||
private String verifierPromptText = "";
|
private String verifierPromptText = "";
|
||||||
|
private String executorOutput = "EXECUTOR_FINAL_ANSWER";
|
||||||
private boolean sawVerifierPrompt;
|
private boolean sawVerifierPrompt;
|
||||||
private final java.util.List<String> verifierOutputs;
|
private final java.util.List<String> verifierOutputs;
|
||||||
private int verifierOutputIndex;
|
private int verifierOutputIndex;
|
||||||
@@ -396,7 +444,7 @@ class ChatServiceSequentialAgentTest {
|
|||||||
} else if (promptText.contains("EXECUTOR_TEST_PROMPT")) {
|
} else if (promptText.contains("EXECUTOR_TEST_PROMPT")) {
|
||||||
agentCalls.add("chat_executor");
|
agentCalls.add("chat_executor");
|
||||||
executorPromptText = promptText;
|
executorPromptText = promptText;
|
||||||
text = "EXECUTOR_FINAL_ANSWER";
|
text = executorOutput;
|
||||||
} else if (promptText.contains("VERIFIER_TEST_PROMPT")) {
|
} else if (promptText.contains("VERIFIER_TEST_PROMPT")) {
|
||||||
agentCalls.add("chat_verifier");
|
agentCalls.add("chat_verifier");
|
||||||
verifierPromptText = promptText;
|
verifierPromptText = promptText;
|
||||||
|
|||||||
@@ -84,7 +84,7 @@ class DiagnosisTraceServiceTest {
|
|||||||
.build();
|
.build();
|
||||||
|
|
||||||
when(diagnosisSessionRepository.findBySessionId(sessionId)).thenReturn(Optional.of(session));
|
when(diagnosisSessionRepository.findBySessionId(sessionId)).thenReturn(Optional.of(session));
|
||||||
when(agentStepRepository.findBySessionIdOrderByStepIndex(sessionId)).thenReturn(List.of(step));
|
when(agentStepRepository.findBySessionId(sessionId)).thenReturn(List.of(step));
|
||||||
when(toolInvocationRepository.findBySessionIdOrderByIdAsc(sessionId)).thenReturn(List.of(invocation));
|
when(toolInvocationRepository.findBySessionIdOrderByIdAsc(sessionId)).thenReturn(List.of(invocation));
|
||||||
|
|
||||||
DiagnosisTraceResponse response = service.getTrace(sessionId);
|
DiagnosisTraceResponse response = service.getTrace(sessionId);
|
||||||
@@ -107,6 +107,38 @@ class DiagnosisTraceServiceTest {
|
|||||||
assertTrue(response.getSummary().isHasFeedback());
|
assertTrue(response.getSummary().isHasFeedback());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void getTraceOrdersStepsByCreationTimeAndIdNotPerAgentStepIndex() {
|
||||||
|
String sessionId = "trace-session-ordered";
|
||||||
|
LocalDateTime base = LocalDateTime.of(2026, 7, 7, 0, 0);
|
||||||
|
DiagnosisSession session = DiagnosisSession.builder()
|
||||||
|
.id(1L)
|
||||||
|
.sessionId(sessionId)
|
||||||
|
.query("mysql pool issue")
|
||||||
|
.status("SUCCESS")
|
||||||
|
.stepCount(4)
|
||||||
|
.toolCallCount(0)
|
||||||
|
.createdAt(base)
|
||||||
|
.updatedAt(base)
|
||||||
|
.build();
|
||||||
|
AgentStep planner = step(1L, sessionId, 0, "planner", base.plusSeconds(1));
|
||||||
|
AgentStep executor0 = step(2L, sessionId, 0, "executor", base.plusSeconds(2));
|
||||||
|
AgentStep executor1 = step(3L, sessionId, 1, "executor", base.plusSeconds(3));
|
||||||
|
AgentStep verifier = step(4L, sessionId, 0, "verifier", base.plusSeconds(4));
|
||||||
|
|
||||||
|
when(diagnosisSessionRepository.findBySessionId(sessionId)).thenReturn(Optional.of(session));
|
||||||
|
when(agentStepRepository.findBySessionId(sessionId))
|
||||||
|
.thenReturn(List.of(planner, executor0, verifier, executor1));
|
||||||
|
when(toolInvocationRepository.findBySessionIdOrderByIdAsc(sessionId)).thenReturn(List.of());
|
||||||
|
|
||||||
|
DiagnosisTraceResponse response = service.getTrace(sessionId);
|
||||||
|
|
||||||
|
assertEquals(List.of("planner", "executor", "executor", "verifier"),
|
||||||
|
response.getSteps().stream().map(DiagnosisTraceResponse.AgentStepTrace::getAgentName).toList());
|
||||||
|
assertEquals(List.of(0, 0, 1, 0),
|
||||||
|
response.getSteps().stream().map(DiagnosisTraceResponse.AgentStepTrace::getStepIndex).toList());
|
||||||
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
void getTraceThrowsWhenSessionMissing() {
|
void getTraceThrowsWhenSessionMissing() {
|
||||||
String sessionId = "missing-session";
|
String sessionId = "missing-session";
|
||||||
@@ -117,4 +149,14 @@ class DiagnosisTraceServiceTest {
|
|||||||
verify(diagnosisSessionRepository).findBySessionId(sessionId);
|
verify(diagnosisSessionRepository).findBySessionId(sessionId);
|
||||||
verifyNoInteractions(agentStepRepository, toolInvocationRepository);
|
verifyNoInteractions(agentStepRepository, toolInvocationRepository);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private AgentStep step(Long id, String sessionId, int stepIndex, String agentName, LocalDateTime createdAt) {
|
||||||
|
return AgentStep.builder()
|
||||||
|
.id(id)
|
||||||
|
.sessionId(sessionId)
|
||||||
|
.stepIndex(stepIndex)
|
||||||
|
.agentName(agentName)
|
||||||
|
.createdAt(createdAt)
|
||||||
|
.build();
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user