The question we want to answer
Which models are best at software engineering and coding?
A useful category rank must measure this outcome, not merely a broad intelligence score or a stylistically similar task.
Category ranking not yet published
Software Engineering & Coding: evidence status
Repository-level implementation, debugging, testing, review, and maintenance with functional correctness and engineering quality kept distinct.
This page is not a model ranking.
We do not yet have enough defensible evidence to rank models for Software Engineering & Coding. No model order is shown here, and this category contributes nothing to the Spring Prompt overall leaderboard.
Why we are not ranking it yet
Evidence gap
This V3 category is defined, but no model order is published until eligible external evidence and SpringPrompt pairwise evidence pass the coverage, independence, reliability, and stability gates.
Collection structure
Explore the four skills in this benchmark
Each skill has its own task plan and evidence status. A collection ranking cannot go live by silently substituting one facet for another.
Repository issue resolution →
Diagnose and fix real repository issues under an exact harness.
25% weight · objective evaluation
Implementation →
Build correct, scoped, maintainable changes.
25% weight · hybrid evaluation
Testing and debugging →
Find failures, write useful tests, and repair root causes.
25% weight · hybrid evaluation
Code review and security →
Spot correctness, security, and maintainability risks.
25% weight · hybrid evaluation
Promotion gate
What must improve before a ranking goes live
- 1At least one eligible external result and one first-party pairwise result.
- 2At least two independent evidence lineages and eight eligible configurations.
- 3At least 80% required-task coverage for every headline row.
- 4Reliability and sealed-release stability checks must both pass.
We do not fill these gaps with the retired internally judged scores. Saved outputs can still illustrate the task, but their old judge grades never determine a new rank.