SuperBizAgent-java/openspec/changes/archive/2026-07-09-interview-demo-quality-audit/specs/diagnosis-eval-harness/spec.md

MODIFIED Requirements

Requirement: Diagnosis eval SHALL validate trace fixtures deterministically

The diagnosis eval harness SHALL evaluate saved trace fixtures without invoking an LLM judge.

Scenario: prompt audit assertions are enforced

• WHEN an eval case sets requirePromptAudit=true
• THEN the evaluator SHALL require verifier_evaluation.prompt_audit.version
• AND when expectedPromptAuditVersion is configured, it SHALL match exactly
• AND when expectedPromptVersions is configured, each configured prompt name SHALL appear with the expected version

Scenario: Gatekeeper rule metadata assertions are enforced

• WHEN an eval case sets requireGatekeeperRules=true
• THEN the evaluator SHALL require verifier_evaluation.gatekeeper_result.rules to be non-empty
• AND each rule item SHALL include id, enabled, and default_severity

Scenario: expanded baseline remains passing

• WHEN the committed fixture set is evaluated
• THEN every case SHALL pass
• AND baseline JSON and Markdown reports SHALL reflect the expanded case count and verdict distribution