1.5 KiB
1.5 KiB
ADDED Requirements
Requirement: Evaluation harness SHALL expose an evidence-pipeline matrix
The evaluation harness SHALL include fixed cases that demonstrate the current Chat evidence pipeline across positive evidence, narrow-scope observation, no-evidence negative observation, low-confidence filtering, reject safety, and composer fallback behavior.
Scenario: Matrix cases are fixture backed
- WHEN the fixed diagnosis case file is evaluated
- THEN each matrix case SHALL resolve to an offline trace fixture
- AND evaluation SHALL not require a live LLM or running application
Scenario: Matrix cases preserve V2 audit closure
- WHEN a matrix case requires V2 audit closure
- THEN its fixture SHALL include
gatekeeper_result - AND it SHALL include
claim_checks - AND it SHALL include
composer_output
Requirement: Evaluation harness SHALL validate Gatekeeper rule set version when requested
The evaluation harness SHALL be able to assert the Gatekeeper rule set version recorded in a fixture.
Scenario: Expected rule set version matches
- WHEN an evaluation case declares
expectedGatekeeperRuleSetVersion - AND the fixture has the same
selfEvaluation.verifier_evaluation.gatekeeper_result.rule_set_version - THEN the rule set version check SHALL pass
Scenario: Expected rule set version mismatches
- WHEN an evaluation case declares
expectedGatekeeperRuleSetVersion - AND the fixture has a different or missing rule set version
- THEN the case SHALL fail with a clear failed check