4.1 KiB
MODIFIED Requirements
Requirement: Deterministic Draft and evidence validation
For a Draft with a non-null Conclusion, the Harness SHALL deterministically reject it unless every Analysis has a unique non-blank Analysis ID, a supported kind, non-blank text, and at least one Tool Call ID, and every Conclusion, Action Plan item, and Recommendation has non-empty references to existing Analysis IDs. For a Draft with conclusion=null, Release SHALL NOT require normal conclusion structure or invoke EvidenceRepair; any supplied Tool references SHALL still resolve to current-Run READY canonical invocations and SHALL obey positive/negative evidence semantics.
Scenario: Duplicate or missing Analysis ID in concluded Draft
- WHEN a Draft with a Conclusion contains a blank or duplicate Analysis ID
- THEN EvidenceGuard returns violations and SemanticGuard is not invoked
Scenario: Broken report reference in concluded Draft
- WHEN a Conclusion, Action Plan item, or Recommendation has an empty or unknown Analysis reference
- THEN EvidenceGuard rejects the Draft before semantic review
Scenario: No-conclusion Draft has valid negative observation
- WHEN a
conclusion=nullDraft cites a current-Run READYNO_EVIDENCEcall asNEGATIVE_OBSERVATION - THEN Release accepts the reference authenticity without running EvidenceRepair or SemanticGuard
Scenario: No-conclusion Draft fabricates a Tool reference
- WHEN a
conclusion=nullDraft cites a missing, cross-Run, incomplete or ERROR Tool call - THEN the reference is excluded and cannot be published as an observed fact
Requirement: Fail-closed release policy
The release use case SHALL publish the unchanged verified Draft only when a non-null Conclusion passes EvidenceGuard and SemanticGuard returns SUPPORTED. It SHALL publish fixed SafeFallback content for evidence failure, semantic unsupported, final semantic technical failure, a valid no-conclusion Draft, information saturation, or handled budget termination. No-conclusion and controlled-stop release SHALL be deterministic from verified references and ProgressSnapshot and MUST NOT invoke a repair or semantic model call. A fallback result MUST NOT include an unsupported Draft, full verified snapshot, internal stop counters, or SemanticGuard reason.
Scenario: Supported report release
- WHEN EvidenceGuard succeeds for a concluded Draft and SemanticGuard returns
SUPPORTED - THEN release outcome is
SUCCESSand the same verified Draft semantics are returned without summarization or partial editing
Scenario: Unsupported report fallback
- WHEN SemanticGuard returns
UNSUPPORTED - THEN release outcome is
FALLBACK, type isSEMANTIC_UNSUPPORTED, and verified sources are derived only from the snapshot
Scenario: Semantic review remains unavailable
- WHEN all permitted technical attempts fail
- THEN release outcome is
FALLBACK, type isSEMANTIC_UNAVAILABLE, and no Draft or internal failure reason is exposed
Scenario: Evidence validation fallback sources
- WHEN evidence repair fails or the second EvidenceGuard rejects a concluded Draft
- THEN release outcome is
FALLBACK, type isEVIDENCE_VALIDATION_FAILED, andverified_sourcesis empty
Scenario: Missing context ends without Tool calls
- WHEN a valid no-conclusion Draft has no Tool calls and identifies required missing context
- THEN release outcome is
FALLBACK, type isMISSING_REQUIRED_CONTEXT, and no guard model call occurs
Scenario: Finite checks do not support a conclusion
- WHEN a no-conclusion Draft or controlled stop has a non-empty verified ProgressSnapshot
- THEN release outcome is
FALLBACK, type isINSUFFICIENT_EVIDENCE, and observed facts describe only actual completed checks
Scenario: Invalid Draft has publishable progress
- WHEN the Agent's final Draft is rejected by strict parsing but its bounded ProgressSnapshot contains current-Run verified observed facts
- THEN Release publishes
FALLBACKwith typeINSUFFICIENT_EVIDENCEusing only that snapshot and MUST NOT use any content from the invalid Draft