Cycle 003 — Grader Calibration and Isolation

Objective

Determine whether a minimal deterministic grader can classify the declared pilot probes and whether agent-readable fixtures can exclude gold outcomes and grader code.

Commands

python3 tools/evaluation/evalctl.py all
python3 tools/evaluation/evalctl.py prepare --output <new-disposable-directory>

Results

• Contracts valid: 2/2.
• Declared calibration classifications matched: 7/7.
• Complete/alternative-correct outcomes accepted: 4/4.
• Partial, regressive, or persuasive-wrong outcomes rejected and classified: 3/3.
• Blind fixtures exported: 2/2.
• Gold outcome, expected-label, or grader files exposed by export: 0.

Interpretation

The tool is useful for preflight validation. It proves neither agent capability nor grader generalization. The calibration set was designed by the same researcher who wrote the grader, so it is vulnerable to confirmation bias.

Assumptions Challenged

• “Trajectory grading is needed immediately.” Not supported for these state-observable pilots.
• “Automated calibration means grader-ready.” Rejected; unseen outcomes and independent labels remain necessary.
• “Security follows evaluation.” Rejected after current incident evidence; containment and gold separation move into the foundation.
• “Token cost is the economic metric.” Rejected; total verified-outcome cost is the decision measure.

Architecture Decision

Adopt a small local architecture now: versioned Markdown/JSON contracts, blind disposable fixtures, an external Python grader, and immutable reports. Defer databases, dashboards, distributed scheduling, knowledge graphs, and broad model matrices.

Diminishing-Returns Boundary

More internal design is now less valuable than:

1. independent review;
2. actual isolated agent attempts;
3. unseen outcome grading;
4. human-time and resource measurements.

The next cycle must produce one of those evidence types rather than more framework prose.