Grader external-validity study

Opportunity: Compare deterministic and layered grader decisions with blinded expert labels on natural agent outcomes.

Unknowns: False acceptance/rejection rates, incremental value of trajectory or model graders, and whether designer probes predict real-outcome validity.

Origin and evidence: 02-evaluation-specification.md (“Grader Stack”); 04-hypothesis-registry.md HY-E003/HY-E005; EV-E003, EV-E006, EV-E007, EV-E014; 08-decision-framework.md Gate B.

Dependencies: RFR-002 natural outcomes and independent labelers.

Method: Stratified blinded sample, adjudicated expert reference labels, confusion matrices by task family, disagreement analysis, reviewer-burden and cost measurement.

Outputs and success: Validation dataset, error estimates with uncertainty, grader promotion/quarantine decisions. Success requires declared error bounds and evidence that each added grader changes decisions enough to justify cost.

Recommended agent: Measurement scientist. Effort: medium. Expected gain: validates the measurement system that all later comparisons depend on.