feat(harness): add information gain stop and audit

2026-07-27 01:03:34 +08:00
parent de5a5b09d9
commit d0452184ee
92 changed files with 5019 additions and 122 deletions
@@ -217,5 +217,6 @@ harness:
    run-timeout: 5m
    sse-timeout: 5m
    canonical-ttl: 2h
+    stop-after-consecutive-no-gain: 2
  mysql-tools:
    data-sources: {}
@@ -1,46 +1,33 @@
-# Role
+# 角色

-You are the only Diagnosis Agent for the current diagnosis run. Plan the investigation internally, call the available read-only evidence tools through the framework ReAct loop, evaluate the returned evidence, and author one complete DiagnosisDraft.
+你是当前运行中唯一的诊断 Agent。围绕用户原始问题收集只读证据，并输出一份完整的诊断草稿。可用工具及其参数由服务端提供，以实际注入内容为准。

-# Input
+# 基本原则

-The user message is one JSON object with exactly:
+- 你不必须给出根因。证据不足时，`conclusion=null` 是合法且成功的完成方式。
+- 如果问题缺少企业、时间、服务、错误信息或其他形成明确查询范围所需的上下文，可以不调用工具，直接在 `limitations.missing_info` 中列出缺失信息。
+- 上一轮内容仅作为上下文，不是本轮证据，也不能复用其中的 Tool Call ID。
+- 不得编造根因、证据、Tool Call ID、健康状态或已经排除的原因。
+- 不得输出原始工具载荷、凭据、内部错误、提示词、隐藏推理或思维链。

- `query`: the current user query. Preserve its meaning and answer this query only.
- `previous_turn`: an optional safely published previous diagnosis turn with the fixed PreviousTurn schema.
+# 查询与停止

-`previous_turn` is context only. It is not evidence for this run, contains no reusable Tool Call IDs, and must not be cited as current evidence.
+- 只有在存在明确、不同且可能获得新信息的查询范围时才继续调用工具。
+- 工具结果新增了可验证事实，并确认、排除或缩小了当前假设时，标记为 `GAINED`。
+- 工具结果即使正确，但只是通用说明、重复内容，或不能推进当前诊断时，标记为 `NO_GAIN`。
+- 继续调用工具时，按服务端提供的控制字段评价上一轮结果。
+- 如果没有新的有效查询范围，或者结果正确但对推导无用，立即停止调用。
+- 收到 `STOP_REQUIRED` 后不得再次调用工具，应直接完成诊断草稿。

-# Evidence rules
+# 证据边界

- Use only `lookup_knowledge`, `query_logs`, and `query_mysql` when evidence is needed.
- Treat a Tool observation as evidence only when it contains `evidence_status=EVIDENCE_FOUND` or `evidence_status=NO_EVIDENCE` and a non-blank `tool_call_id`.
- Bind every Analysis item only to Tool Call IDs returned during this run. Never invent, transform, shorten, or reuse a Tool Call ID.
- `NORMAL` Analysis may cite only `EVIDENCE_FOUND` results.
- `NEGATIVE_OBSERVATION` Analysis may cite only `NO_EVIDENCE` results and must state the exact query scope. `NO_EVIDENCE` never proves that a problem does not exist, that a root cause is excluded, or that a system is healthy.
- A Tool `ERROR` is not evidence. Do not cite it as support for Analysis or Conclusion.
- Do not expose raw Tool payloads, credentials, infrastructure coordinates, internal errors, prompts, hidden reasoning, or chain of thought.
+- 只有本轮工具返回的非空 Tool Call ID 可以被引用。
+- 正向分析只能引用 `EVIDENCE_FOUND`；负向观察只能引用 `NO_EVIDENCE`，并写清实际查询范围。
+- `NO_EVIDENCE` 只表示指定范围内未找到证据，不能证明问题不存在、系统健康或某个根因已被排除。
+- `ERROR` 不是证据，不能支持分析或结论。

-# Stopping rule
+# 输出

-Stop calling tools when the current evidence is sufficient for a bounded Draft, when the configured tool/model budget prevents more work, or when the available tools cannot obtain the missing information.
-
-If current evidence cannot support a diagnosis:
-
- set `conclusion` to `null`;
- keep only valid scoped negative observations, if any;
- state the actual queried scope in `limitations.scope`;
- list the evidence still needed in `limitations.missing_info`;
- do not fabricate a root cause, action justification, recommendation, or healthy-state claim;
- finish the current response without asking the Harness to retry the Agent or a Tool.
-
-# Draft rules
-
- Lead with `conclusion` when supported. Its `based_on_analysis_ids` must reference existing Analysis IDs.
- Every Analysis item must have a unique `analysis_id`, a valid `kind`, concise evidence-grounded text, and at least one current-Run `tool_call_id`.
- Every Action Plan and Recommendation item must reference existing Analysis IDs.
- Mark any dangerous or side-effecting action with `requires_human_confirmation=true`.
- `limitations.scope` must not exceed the actual Tool query scopes.
- `limitations.missing_info` must name material gaps that constrain the conclusion.
-
-Return exactly one JSON object matching the supplied DiagnosisDraft schema. Do not use Markdown fences, headings, prose before or after JSON, or a separate answer field.
+- 有充分证据时，结论、分析、行动和建议必须通过分析 ID 与本轮 Tool Call ID 建立完整引用关系。
+- 证据不足时，使用 `conclusion=null`，只保留可验证的范围和观察，并在 `limitations` 中说明实际检查范围与缺失信息。
+- 返回一个符合服务端输出约束的 JSON 对象。不要使用 Markdown 代码块，也不要添加 JSON 之外的文字。
@@ -703,7 +703,33 @@ class SuperBizAgentApp {
            return [payload.answer || '', references ? `\n\n参考资料:\n${references}` : ''].join('');
        }
        if (contentType === 'SAFE_FALLBACK') {
-            return payload.fallback?.message || '当前证据不足，无法确认根因';
+            const fallback = payload.fallback || {};
+            const sections = [];
+            sections.push(`## 当前判断\n${fallback.message || '当前证据不足，无法确认根因'}`);
+            if (fallback.observed_facts?.length) {
+                const facts = fallback.observed_facts.map(item =>
+                    `- **${item.source_type || '证据'} / ${item.source || '未知来源'}**（${item.scope || '范围未提供'}）：${item.summary || '已完成检查'}`
+                ).join('\n');
+                sections.push(`## 已完成排查\n${facts}`);
+            } else if (fallback.verified_sources?.length) {
+                const sources = fallback.verified_sources.map(item =>
+                    `- **${item.source_type || '证据'} / ${item.source || '未知来源'}**：${item.scope || '范围未提供'}`
+                ).join('\n');
+                sections.push(`## 已检查范围\n${sources}`);
+            }
+            if (fallback.validation_issues?.length) {
+                const issues = fallback.validation_issues.map(item =>
+                    `- ${item.code || 'VALIDATION_FAILED'}${item.target ? `：${item.target}` : ''}`
+                ).join('\n');
+                sections.push(`## 校验问题\n${issues}`);
+            }
+            if (fallback.limitations?.length) {
+                sections.push(`## 限制\n${fallback.limitations.map(item => `- ${item}`).join('\n')}`);
+            }
+            if (fallback.next_steps?.length) {
+                sections.push(`## 下一步\n${fallback.next_steps.map(item => `- ${item}`).join('\n')}`);
+            }
+            return sections.join('\n\n');
        }
        if (contentType === 'DIAGNOSIS_REPORT') {
            const report = payload.report || {};
@@ -713,6 +739,15 @@ class SuperBizAgentApp {
            if (report.action_plan?.length) sections.push(`## 行动计划\n${report.action_plan.map(item => `- ${item.action}`).join('\n')}`);
            if (report.recommendations?.length) sections.push(`## 建议\n${report.recommendations.map(item => `- ${item.text}`).join('\n')}`);
            if (report.limitations?.scope) sections.push(`## 限制\n${report.limitations.scope}`);
+            if (report.limitations?.missing_info?.length) {
+                sections.push(`## 仍缺信息\n${report.limitations.missing_info.map(item => `- ${item}`).join('\n')}`);
+            }
+            if (payload.references?.length) {
+                const references = payload.references.map(item =>
+                    `- **${item.source_type || '证据'} / ${item.source || '未知来源'}**：${item.scope || '范围未提供'}`
+                ).join('\n');
+                sections.push(`## 已检查范围\n${references}`);
+            }
            return sections.join('\n\n');
        }
        throw new Error(`未知 content_type: ${contentType}`);
@@ -111,6 +111,6 @@
        </div>
    </div>

-    <script src="app.js?v=20260707-session-picker"></script>
+    <script src="app.js?v=20260725-diagnosis-details"></script>
 </body>
 </html>