22 lines
1.1 KiB
Markdown
22 lines
1.1 KiB
Markdown
## MODIFIED Requirements
|
|
|
|
### Requirement: Diagnosis eval SHALL validate trace fixtures deterministically
|
|
The diagnosis eval harness SHALL evaluate saved trace fixtures without invoking an LLM judge.
|
|
|
|
#### Scenario: prompt audit assertions are enforced
|
|
- **WHEN** an eval case sets `requirePromptAudit=true`
|
|
- **THEN** the evaluator SHALL require `verifier_evaluation.prompt_audit.version`
|
|
- **AND** when `expectedPromptAuditVersion` is configured, it SHALL match exactly
|
|
- **AND** when `expectedPromptVersions` is configured, each configured prompt name SHALL appear with the expected version
|
|
|
|
#### Scenario: Gatekeeper rule metadata assertions are enforced
|
|
- **WHEN** an eval case sets `requireGatekeeperRules=true`
|
|
- **THEN** the evaluator SHALL require `verifier_evaluation.gatekeeper_result.rules` to be non-empty
|
|
- **AND** each rule item SHALL include `id`, `enabled`, and `default_severity`
|
|
|
|
#### Scenario: expanded baseline remains passing
|
|
- **WHEN** the committed fixture set is evaluated
|
|
- **THEN** every case SHALL pass
|
|
- **AND** baseline JSON and Markdown reports SHALL reflect the expanded case count and verdict distribution
|
|
|