feat(demo): add interview quality audit

2026-07-09 11:18:49 +08:00
parent a6c2d4459c
commit 9c9a0024d4
37 changed files with 1162 additions and 86 deletions
@@ -195,3 +195,22 @@ The evaluation harness SHALL be able to assert the Gatekeeper rule set version r
 - **WHEN** an evaluation case declares `expectedGatekeeperRuleSetVersion`
 - **AND** the fixture has a different or missing rule set version
 - **THEN** the case SHALL fail with a clear failed check
+
+### Requirement: Diagnosis eval SHALL validate trace fixtures deterministically
+The diagnosis eval harness SHALL evaluate saved trace fixtures without invoking an LLM judge.
+
+#### Scenario: prompt audit assertions are enforced
+- **WHEN** an eval case sets `requirePromptAudit=true`
+- **THEN** the evaluator SHALL require `verifier_evaluation.prompt_audit.version`
+- **AND** when `expectedPromptAuditVersion` is configured, it SHALL match exactly
+- **AND** when `expectedPromptVersions` is configured, each configured prompt name SHALL appear with the expected version
+
+#### Scenario: Gatekeeper rule metadata assertions are enforced
+- **WHEN** an eval case sets `requireGatekeeperRules=true`
+- **THEN** the evaluator SHALL require `verifier_evaluation.gatekeeper_result.rules` to be non-empty
+- **AND** each rule item SHALL include `id`, `enabled`, and `default_severity`
+
+#### Scenario: expanded baseline remains passing
+- **WHEN** the committed fixture set is evaluated
+- **THEN** every case SHALL pass
+- **AND** baseline JSON and Markdown reports SHALL reflect the expanded case count and verdict distribution