Comparative Analysis of Non-Human Experimental Evidence

Executive finding

The repository establishes that its current deterministic evaluator can reproduce designer-declared classifications for seven supplied probes and that its exporter can omit known outcome/grader files under a narrow check. It does not establish task validity, grader generalization, agent capability, repeatability, transfer, safety under agent execution, or cost effectiveness.

The body of evidence is internally consistent once design studies, pilot construction, calibration, and capability experiments are separated. Most apparent contradictions are scope or measurement mismatches rather than opposing experimental results.

Normalization strategy

No transformation was applied. Formal pooling, confidence intervals, regression, Bayesian updating, and meta-analysis are unjustified because cases were purposively constructed, share an author and evaluator, and are not sampled from a defined population.

Experiment quality matrix

Ratings are explicit ordinal judgments: strong, moderate, weak, absent, or not applicable. They are not a composite score.

Comparability matrix

Experiment-family and pairwise synthesis

Task-integrity pilots: EX-E002 and EX-E003

Both pilots show that constructing positive alternatives and negative cases exposes design risks before an agent run. ET-004 revealed literal semantic and collateral-change concerns; ET-014 revealed taxonomy strictness. The stable pattern is useful as workflow evidence. It is not an effect estimate because both tasks were selected and revised by the same researcher and there was no unaudited control condition.

ET-004 measures bounded state editing. ET-014 measures detection of a planted, explicit contradiction. Their shared acceptance/rejection counts should not be interpreted as equal difficulty or pooled task success.

Calibration family: EX-E002–EX-E004

EX-E004 is a deterministic replay over the artifacts produced by EX-E002 and EX-E003. It adds implementation evidence: the code matches the intended labels. It does not add an independent sample. The strongest supported conclusion is software-regression-level: the current code separates the current cases.

Isolation family: EX-E002/003–EX-E007

EX-E007 demonstrates a useful physical-export mechanism. Its predicate searches exported filenames for outcome, expected-calibration, or grader. It does not inspect semantic equivalence, hidden metadata, alternate names, symlinks, or whether the visible task itself leaks the answer. Therefore “known files omitted” is supported; “contamination controlled” remains suggestive.

Longitudinal and cross-project analysis

The sequence from Cycle 001 to Cycle 003 improves artifact executability: design-only taxonomy → versioned pilots → executable known-case calibration and export. Because the intervention, artifacts, and measurements changed at each step, this is maturation of the research system, not a longitudinal performance improvement.

No cross-project experiment was conducted. External benchmark studies informed the design but used different tasks, models, evaluators, datasets, and units. They cannot be pooled with AI-ROS observations. Transfer remains untested.

Quantitative synthesis

Direct observations:

• 2/2 pilot JSON contracts contain required fields and required outcome folders.
• 7/7 authored probe labels match evaluator output.
• 4/4 intended positive probes are accepted.
• 3/3 intended negative probes are rejected/classified.
• 2/2 current fixtures export.
• 0/2 exports trigger the narrow filename detector.
• 0 agent trials, 0 independent reviews, 0 unseen grader cases, and 0 cost observations exist.

The zeroes in the second group are absence-of-evidence counts, not measured null effects. No statistical significance or effect size is claimed.

Qualitative synthesis

Recurring risks are researcher circularity, gold co-location, literal semantic grading, construct–metric mismatch, incomplete run provenance, and ambiguous status language. The repository performs well at documenting negative results and scope warnings. Its main research-system weakness is that narrative caveats are more mature than empirical sampling.

Confidence reassessment and findings matrix

Adversarial review of the five main conclusions

1. Known-case calibration works. Alternative: the apparent success is tautological because the grader and labels co-evolved. This does not overturn the software-regression conclusion, but blocks external-validity inference.
2. Probe design found useful defects. Alternative: ordinary implementation iteration, not a formal audit, caused the changes. Preferred conclusion is narrowed to process utility, not causal effectiveness.
3. Export omits known sensitive files. Alternative: gold remains inferable from visible content or disguised names. This is credible; the claim is limited to exact exported files and predicate.
4. No capability conclusion is available. Alternative: authored outcomes might proxy model behavior. They do not originate from a model or sampled process, so this conclusion survives strongly.
5. Independent review is the next gate. Alternative: unseen mutation cases could be automated first. That would improve grader evidence, but cannot supply independent task-validity judgment; reviews remain the gating evidence for promotion, while automated mutations are a parallel high-value test.

Repeated review did not change the ordering: independent review plus frozen, unseen calibration is more decision-relevant than additional architecture.

Research-gap matrix