Design

Data Flow

Gatekeeper rule catalog
  -> ExecutorGatekeeperService.validate(...)
  -> gatekeeper_result.rule_set_version + rule metadata summary
  -> DiagnosisTraceEvaluator fixture checks
  -> baseline report / demo runbook

Stable demo scenarios
  -> request payloads and docs
  -> optional live run for main path
  -> saved trace fixtures for deterministic matrix
  -> mvp/eval baseline report

Eval Matrix

The diagnosis eval matrix remains offline and deterministic. It should cover these rows:

Matrix row	Expected signal
Positive supported evidence	`PASS`, Gatekeeper `pass`, Composer `valid`
Narrow-scope observation	`PASS`, one or minimal claims, no forbidden over-expansion
No-evidence negative observation	`PASS` or allowed non-reject verdict, `$.no_evidence`, no overstatement
Unsupported claim filtering	`LOW_CONFID`, unsupported claim not in final answer
Gatekeeper fabricated reference	`REJECT` or `LOW_CONFID`, Gatekeeper `fail`
Composer fallback	no raw Executor protocol leakage

The evaluator should validate rule set version only for cases that opt into the new check. This keeps old fixtures readable while allowing the new matrix to prove Gatekeeper metadata persistence.

Stable Demo Scenarios

Stable demo scenarios are source-controlled payloads and documentation, not a live-only test harness. The demo set should include:

A supported positive path.
A no-evidence / negative-observation path.
A safety/reject path explained through fixed fixture or evaluator output.

Only the main path needs a live script in this phase. Other scenarios may be represented by payloads, fixture names, and expected trace fields.

Gatekeeper Rule Catalog

Gatekeeper rules stay local and lightweight:

{
  "version": "gatekeeper-rules-v1",
  "rules": [
    {
      "id": "evidence.raw_path",
      "description": "raw_path must exist in retrieval_details.evidence_refs",
      "default_severity": "reject",
      "enabled": true
    }
  ]
}

The first implementation may use an in-memory default catalog or a classpath JSON resource. It must expose:

rule set version
enabled rule ids
rule descriptions
severity defaults or threshold parameters when present

Gatekeeper validation logic remains deterministic Java code. The catalog is metadata/config, not a dynamic scripting engine.

Audit Persistence

gatekeeper_result should include:

{
  "rule_set_version": "gatekeeper-rules-v1",
  "rules": [
    {
      "id": "evidence.raw_path",
      "description": "raw_path must exist in retrieval_details.evidence_refs",
      "enabled": true,
      "default_severity": "reject"
    }
  ]
}

Existing fields remain:

status
severity
checked_bindings
failed_rules
warnings
errors

Interface Impact

Impact level: L2 internal contract change.

This expands internal audit JSON and eval case/result fields. It does not change external HTTP APIs, database schema, Planner output, or public DTO contracts.

Risks

Too much Gatekeeper flexibility could weaken safety. This phase only adds metadata/configuration and keeps rule implementations fixed in code.
Demo scenarios should not promise deterministic LLM behavior. Deterministic claims should point to fixture-backed eval results.
Baseline updates must be made together with new fixtures and tests.

3.3 KiB Raw Blame History