research-document RFR-004
Representative repository task corpus
Representative repository task corpus
Opportunity: Build and audit a minimum suite spanning production-like coding, documentation, recovery, provenance, security, and cross-file work.
Unknowns: Population of intended tasks, sampling frame, documentation bias, task-family balance, and transfer from public benchmarks.
Origin and evidence: 01-task-suite-manifest.md; EV-E001, EV-E004, EV-E008, EV-E009; HY-E001, HY-E002, HY-E008; 12-research-and-engineering-roadmap.md A4.
Dependencies: RFR-003, RFR-009, sufficient real repository history.
Method: Define target population; sample from held-out history and synthetic-but-audited cases; record provenance and leakage risk; independently audit every contract; report by task rather than only aggregate.
Outputs and success: Versioned corpus, sampling report, audit labels, held-out fixtures. Success requires explicit coverage targets and demonstrated inclusion of real coding workloads.
Recommended agent: Evaluation dataset lead. Effort: large. Expected gain: external validity and reusable infrastructure for most roadmap questions.