29 lines
1.5 KiB
Markdown
29 lines
1.5 KiB
Markdown
## ADDED Requirements
|
|
|
|
### Requirement: Evaluation harness SHALL expose an evidence-pipeline matrix
|
|
The evaluation harness SHALL include fixed cases that demonstrate the current Chat evidence pipeline across positive evidence, narrow-scope observation, no-evidence negative observation, low-confidence filtering, reject safety, and composer fallback behavior.
|
|
|
|
#### Scenario: Matrix cases are fixture backed
|
|
- **WHEN** the fixed diagnosis case file is evaluated
|
|
- **THEN** each matrix case SHALL resolve to an offline trace fixture
|
|
- **AND** evaluation SHALL not require a live LLM or running application
|
|
|
|
#### Scenario: Matrix cases preserve V2 audit closure
|
|
- **WHEN** a matrix case requires V2 audit closure
|
|
- **THEN** its fixture SHALL include `gatekeeper_result`
|
|
- **AND** it SHALL include `claim_checks`
|
|
- **AND** it SHALL include `composer_output`
|
|
|
|
### Requirement: Evaluation harness SHALL validate Gatekeeper rule set version when requested
|
|
The evaluation harness SHALL be able to assert the Gatekeeper rule set version recorded in a fixture.
|
|
|
|
#### Scenario: Expected rule set version matches
|
|
- **WHEN** an evaluation case declares `expectedGatekeeperRuleSetVersion`
|
|
- **AND** the fixture has the same `selfEvaluation.verifier_evaluation.gatekeeper_result.rule_set_version`
|
|
- **THEN** the rule set version check SHALL pass
|
|
|
|
#### Scenario: Expected rule set version mismatches
|
|
- **WHEN** an evaluation case declares `expectedGatekeeperRuleSetVersion`
|
|
- **AND** the fixture has a different or missing rule set version
|
|
- **THEN** the case SHALL fail with a clear failed check
|