osiman/SuperBizAgent-java
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(eval): add evidence pipeline acceptance closure

Browse Source
This commit is contained in:
aruo
2026-07-09 00:47:48 +08:00
parent a77c947cd4
commit db0f229285
46 changed files with 1434 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
openspec/specs/diagnosis-eval-harness/spec.md
+27 -2
View File
@@ -2,9 +2,7 @@
## Purpose
Provide a repeatable offline evaluation harness for MVP diagnosis Agent behavior, so prompt, tool, retrieval, and verifier changes can be checked against fixed trace-based regression cases.
## Requirements
### Requirement: Evaluation harness SHALL define fixed diagnosis cases
The system SHALL provide a small fixed set of MVP diagnosis evaluation cases with explicit expected trace and answer criteria.
@@ -170,3 +168,30 @@ The evaluation harness SHALL detect configured unsafe or unsupported claim text
#### Scenario: Composer fallback still avoids raw JSON leakage
- **WHEN** a trace records Composer fallback rendering
- **THEN** the evaluator SHALL still enforce final-answer raw JSON leakage checks
### Requirement: Evaluation harness SHALL expose an evidence-pipeline matrix
The evaluation harness SHALL include fixed cases that demonstrate the current Chat evidence pipeline across positive evidence, narrow-scope observation, no-evidence negative observation, low-confidence filtering, reject safety, and composer fallback behavior.
#### Scenario: Matrix cases are fixture backed
- **WHEN** the fixed diagnosis case file is evaluated
- **THEN** each matrix case SHALL resolve to an offline trace fixture
- **AND** evaluation SHALL not require a live LLM or running application
#### Scenario: Matrix cases preserve V2 audit closure
- **WHEN** a matrix case requires V2 audit closure
- **THEN** its fixture SHALL include `gatekeeper_result`
- **AND** it SHALL include `claim_checks`
- **AND** it SHALL include `composer_output`
### Requirement: Evaluation harness SHALL validate Gatekeeper rule set version when requested
The evaluation harness SHALL be able to assert the Gatekeeper rule set version recorded in a fixture.
#### Scenario: Expected rule set version matches
- **WHEN** an evaluation case declares `expectedGatekeeperRuleSetVersion`
- **AND** the fixture has the same `selfEvaluation.verifier_evaluation.gatekeeper_result.rule_set_version`
- **THEN** the rule set version check SHALL pass
#### Scenario: Expected rule set version mismatches
- **WHEN** an evaluation case declares `expectedGatekeeperRuleSetVersion`
- **AND** the fixture has a different or missing rule set version
- **THEN** the case SHALL fail with a clear failed check