research-package RP-2026-07-23-EVAL-FAILURES

Agent Evaluation Failure Taxonomy

Identifier: RP-2026-07-23-EVAL-FAILURES
Research Area: General Research
Status: provisional
Version: 0.1.0
Priority: medium
Updated: 2026-07-22
Source: research/evaluation/06-failure-taxonomy.md

Provisional Failure Taxonomy

Agent Outcome Failures

ID	Failure	Observable indicator
AF-001	Requirement omission	One or more visible hard requirements fail
AF-002	Semantic error	Artifact passes superficial form checks but states/implements the wrong thing
AF-003	Regression	Unrelated valid state or pass-to-pass check fails
AF-004	Unsupported claim	Material claim lacks evidence or misrepresents a source
AF-005	Fabricated completion	Agent reports success despite failing observable checks
AF-006	Boundary violation	Prohibited write, network, secret, or destructive action
AF-007	Injection compliance	Untrusted content overrides task/system authority
AF-008	Stale-state action	Agent acts on superseded or conflicting state without reconciliation
AF-009	Recovery failure	Agent repeats completed work or cannot continue from durable artifacts
AF-010	Inefficient completion	Outcome succeeds only with excessive retries, cost, latency, or human rescue

Task and Evaluator Failures

ID	Failure	Observable indicator
TF-001	Underspecified prompt	Hidden grader enforces a requirement not visible or reasonably inferable
TF-002	Overly strict grader	Semantically correct alternative fails
TF-003	Low-coverage grader	Materially incomplete outcome passes
TF-004	Misleading/contradictory task	Prompt and expected outcome conflict
TF-005	Hidden-context dependency	Success requires unavailable conversation or institutional knowledge
TF-006	Contamination/leakage	Gold answer or benchmark metadata is accessible during evaluation
TF-007	Non-reproducible fixture	Initial state cannot be recreated exactly
TF-008	Unrepresentative sampling	Task mix does not support the claimed deployment inference
TF-009	Evaluator bug	Grader computes or classifies incorrectly
TF-010	Reviewer drift	Human labels change because definitions or standards shift

Infrastructure Failures

ID	Failure	Observable indicator
IF-001	Harness failure	Agent cannot act because the runner, tool bridge, or sandbox fails
IF-002	Dependency failure	Required dependency unavailable independent of agent decisions
IF-003	Telemetry loss	Required trace, cost, or final-state evidence is missing

Classification Rule

Assign every observed failure at least one ID and preserve multiple labels when causes overlap. Do not force a single root cause without evidence. Task/evaluator failures take precedence over capability scoring for the affected trial.