111 lines
3.3 KiB
Markdown
111 lines
3.3 KiB
Markdown
# Design
|
|
|
|
## Data Flow
|
|
|
|
```text
|
|
Gatekeeper rule catalog
|
|
-> ExecutorGatekeeperService.validate(...)
|
|
-> gatekeeper_result.rule_set_version + rule metadata summary
|
|
-> DiagnosisTraceEvaluator fixture checks
|
|
-> baseline report / demo runbook
|
|
```
|
|
|
|
```text
|
|
Stable demo scenarios
|
|
-> request payloads and docs
|
|
-> optional live run for main path
|
|
-> saved trace fixtures for deterministic matrix
|
|
-> mvp/eval baseline report
|
|
```
|
|
|
|
## Eval Matrix
|
|
|
|
The diagnosis eval matrix remains offline and deterministic. It should cover these rows:
|
|
|
|
| Matrix row | Expected signal |
|
|
|---|---|
|
|
| Positive supported evidence | `PASS`, Gatekeeper `pass`, Composer `valid` |
|
|
| Narrow-scope observation | `PASS`, one or minimal claims, no forbidden over-expansion |
|
|
| No-evidence negative observation | `PASS` or allowed non-reject verdict, `$.no_evidence`, no overstatement |
|
|
| Unsupported claim filtering | `LOW_CONFID`, unsupported claim not in final answer |
|
|
| Gatekeeper fabricated reference | `REJECT` or `LOW_CONFID`, Gatekeeper `fail` |
|
|
| Composer fallback | no raw Executor protocol leakage |
|
|
|
|
The evaluator should validate rule set version only for cases that opt into the new check. This keeps old fixtures readable while allowing the new matrix to prove Gatekeeper metadata persistence.
|
|
|
|
## Stable Demo Scenarios
|
|
|
|
Stable demo scenarios are source-controlled payloads and documentation, not a live-only test harness. The demo set should include:
|
|
|
|
- A supported positive path.
|
|
- A no-evidence / negative-observation path.
|
|
- A safety/reject path explained through fixed fixture or evaluator output.
|
|
|
|
Only the main path needs a live script in this phase. Other scenarios may be represented by payloads, fixture names, and expected trace fields.
|
|
|
|
## Gatekeeper Rule Catalog
|
|
|
|
Gatekeeper rules stay local and lightweight:
|
|
|
|
```json
|
|
{
|
|
"version": "gatekeeper-rules-v1",
|
|
"rules": [
|
|
{
|
|
"id": "evidence.raw_path",
|
|
"description": "raw_path must exist in retrieval_details.evidence_refs",
|
|
"default_severity": "reject",
|
|
"enabled": true
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
The first implementation may use an in-memory default catalog or a classpath JSON resource. It must expose:
|
|
|
|
- rule set version
|
|
- enabled rule ids
|
|
- rule descriptions
|
|
- severity defaults or threshold parameters when present
|
|
|
|
Gatekeeper validation logic remains deterministic Java code. The catalog is metadata/config, not a dynamic scripting engine.
|
|
|
|
## Audit Persistence
|
|
|
|
`gatekeeper_result` should include:
|
|
|
|
```json
|
|
{
|
|
"rule_set_version": "gatekeeper-rules-v1",
|
|
"rules": [
|
|
{
|
|
"id": "evidence.raw_path",
|
|
"description": "raw_path must exist in retrieval_details.evidence_refs",
|
|
"enabled": true,
|
|
"default_severity": "reject"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Existing fields remain:
|
|
|
|
- `status`
|
|
- `severity`
|
|
- `checked_bindings`
|
|
- `failed_rules`
|
|
- `warnings`
|
|
- `errors`
|
|
|
|
## Interface Impact
|
|
|
|
Impact level: L2 internal contract change.
|
|
|
|
This expands internal audit JSON and eval case/result fields. It does not change external HTTP APIs, database schema, Planner output, or public DTO contracts.
|
|
|
|
## Risks
|
|
|
|
- Too much Gatekeeper flexibility could weaken safety. This phase only adds metadata/configuration and keeps rule implementations fixed in code.
|
|
- Demo scenarios should not promise deterministic LLM behavior. Deterministic claims should point to fixture-backed eval results.
|
|
- Baseline updates must be made together with new fixtures and tests.
|