Planned benchmark facet · ranking not published

Which models are best at testing and debugging?

Find failures, write useful tests, and repair root causes.

This is the evaluation plan, not a model leaderboard.

No model order or score is shown. Candidate external benchmarks and planned first-party comparisons become evidence only after their exact release gates pass; retired benchmark grades are not reused.

First-party pairwise plan

Six distinct tasks for this facet

exploration tasks

Used for comparison coverage and calibrated refinement.

sealed holdout task

Kept out of model and judge tuning until release validation.

tasks in total

One of four equally weighted facets in this collection.

First-party, task-local pairwise comparisons with deterministic validators acting as hard gates. The final task remains sealed from model and judge tuning until release validation.

Collection-level external evidence map

Candidate benchmarks that can inform this skill

These sources are mapped to Software Engineering & Coding, then identity-partitioned and lineage-deduplicated. They are supporting inputs, not a substitute for direct task-local pairwise results, and no score is reproduced on this page.

Artificial Analysis

Coding and Coding Agent indices

Mapped and runnable

Evidence role: primary
Maximum directness: 70%
Declared subject: harnessed model
Usable lineages mapped: 4

Capabilities: coding, repository engineering, terminal work, repository understanding.

Mapped fields: SciCode, DeepSWE, Terminal-Bench v2 (Coding Agent Index), SWE-Atlas-QnA.

• source redistribution is conditional: assumed
• metric subject identity differs from the editorial mapping; publish and calibrate each identity partition separately

Microsoft BenchPress Score Matrix

BenchPress observed coding rows

Mapped and runnable

Evidence role: supporting
Maximum directness: 60%
Declared subject: harnessed model
Usable lineages mapped: 4

Capabilities: code editing, code generation, issue resolution, terminal work.

Mapped fields: BenchPress verified canonical row — Aider Polyglot (diff mode), BenchPress verified canonical row — LiveCodeBench v6, BenchPress verified canonical row — SWE-bench Pro, BenchPress verified canonical row — Terminal-Bench Hard.

OpenHands Index

Mapped and runnable

Evidence role: primary
Maximum directness: 85%
Declared subject: agent
Usable lineages mapped: 3

Capabilities: issue resolution, greenfield implementation, testing.

Mapped fields: OpenHands Index — Issue Resolution, OpenHands Index — Greenfield, OpenHands Index — Testing.

• metric subject identity differs from the editorial mapping; publish and calibrate each identity partition separately

Current blockers

Why no ranking is live

No approved combined collection artifact has been released.
The parent collection has not passed its publication gate.
No approved facet-level result artifact has been released.

Publication gate

What must pass first

1Combine at least one eligible external result with at least one first-party pairwise result.
2Retain at least 2 independent evidence lineages and 8 eligible model configurations.
3Complete at least 3 facet tasks with at least 80% required-task coverage.
4Show that this facet is materially distinct, then pass reliability and sealed-release stability checks.

Software Engineering & Coding

Explore all four planned skills

Repository issue resolution Diagnose and fix real repository issues under an exact harness. Implementation Build correct, scoped, maintainable changes. Testing and debugging Find failures, write useful tests, and repair root causes. Code review and security Spot correctness, security, and maintainability risks.

Back to Software Engineering & Coding Sources and safeguards