🔍 Read the full analysis: FrontierCode: Getting An AI-Written Pull Request Ready For Review on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis evaluated 23 AI models on FrontierCode, a benchmark that scores whether repository changes meet maintainers’ review criteria, not just whether tests pass. Claude Opus 5.5 led with a 52.2% score and cleared every blocking check in 57.7% of attempts, while the results show wide cost differences and leave many models untested.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that asks whether a change is fit for a maintainer to accept, rather than simply whether it passes tests. Claude Opus 5.5 led with a 52.2% score, but cleared all blocking checks in 57.7% of attempts, indicating that even the top result does not make human review unnecessary.
Artificial Analysis added the benchmark to its evaluations on October 12, 2026, running models through the same open-source agent harness, Stirrup. Each model could make up to three attempts per task, with a limit of 250 turns per attempt. The evaluation covered the 100-task Main set from FrontierCode’s larger collection of 150 tasks.
After Opus, GPT-6.1 Sol scored 49.1%, followed closely by Claude Fable 5.1 at 49.0%. The models’ pass rates for all blocking checks were lower than or similar to their overall scores: Sol cleared them in 54.7% of attempts, and Fable in 54.3%. Nine of the 100 tasks were not solved by any model on any attempt, according to the source report.
FrontierCode checks patches for correctness, regressions, whether the agent’s tests distinguish the changed code from the original, and whether the patch stays within limits on files and diff size. It also uses adaptive tests and an LLM-based code review against maintainers’ criteria. Failing any blocking check gives an attempt a score of zero; passing all blocking checks produces a score from 70% to 100%, adjusted for quality checks. The review judge in this evaluation was GPT-6 Luna (medium).
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
The results address a practical question for teams considering AI-generated pull requests: how much work remains before a proposed change can be reviewed and merged? FrontierCode’s design treats reviewability and scope control as part of code quality, rather than counting successful test runs alone. The leading model’s 57.7% blocking-check pass rate means a substantial share of attempts still failed at least one required check in this benchmark.
Cost also varied sharply. Artificial Analysis reported per-task costs ranging from $0.09 to $12. GPT-6.1 Sol scored 49.1% for $0.81 per task, while Opus scored 52.2% for $10.15. That is a 3.1-point score difference at about 12.5 times the reported cost. These figures may help teams compare tested configurations, but they do not establish what a model would cost or achieve on a different codebase or workflow.
The rankings also differed from general intelligence scores. For example, Claude Sonnet 5.5 ranked second among the listed models on the Intelligence Index but fifth on FrontierCode, while DeepSeek V4.1 Flash placed higher on FrontierCode than its general-index position. That gap suggests general capability rankings are not a substitute for evaluations focused on repository changes.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Repository Changes
FrontierCode was developed by Cognition and open-source maintainers, and announced in June 2026. Version 1.1 followed in July with rules intended to prevent agents from looking up solutions online. The source report says more than 20 maintainers created tasks drawn from 36 established repositories, including Celery, Budibase, uppy and Mattermost, and spent more than 40 hours on each task. The tasks are private, limiting the chance that models encountered them in training.
The benchmark’s checks are intended to capture problems that conventional test-based scoring can miss. A patch can pass tests yet add irrelevant files, expand the change unnecessarily, or include tests that would also pass against the original code. FrontierCode’s scope limits, reverse tests and maintainers’ review criteria penalize those patterns. Attempts flagged by Cognition’s trajectory scanner for looking up a solution receive zero, according to the report.
Artificial Analysis ran the benchmark on its Main set of 100 tasks, rather than the full 150-task collection. Nine models from six labs scored between 43% and 52%, according to the published results. Xiaomi’s MiMo-V2.6-Pro led the open-weights group listed in the report at 45.4%, followed by DeepSeek V4.1 Flash at 44.0% and Kimi K3 at 43.6%.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, Celery maintainer, in Cognition’s announcement
Limits of the Published Results
The results cover a defined benchmark, harness and set of model configurations; they do not establish how often each model would produce acceptable changes across all real-world repositories. The source report does not provide enough information here to assess variation across different task types or how performance changes under other prompting and review workflows. The reported pass rates should not be treated as universal production success rates.
Several prominent models had no FrontierCode result as of October 13, according to the report, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published rankings are incomplete. The source also describes the benchmark and its results but does not independently establish whether its scoring fully reflects every organization’s standards for accepting code.
More Models and Tasks to Watch
The next useful development will be additional results for models not included in this evaluation, along with reruns as model versions and agent setups change. Any comparison will need to keep the task set, attempt limits, harness and cost basis clear so readers can tell whether differences reflect model capability or a changed testing setup.
For engineering teams, the findings support testing candidate models against their own repositories and review requirements before assigning them substantial coding work. FrontierCode offers one structured comparison, but the source does not report a timeline for the next leaderboard update or confirm when missing models will be evaluated.
Key Questions
What is FrontierCode measuring?
It evaluates whether an AI-generated repository change satisfies blocking checks and quality criteria, including correctness, regressions, test usefulness, scope and maintainers’ review standards. It is designed to assess more than whether a patch passes tests.
Which model ranked highest?
Claude Opus 5.5 (max with fallback) led the published results with a 52.2% FrontierCode score. It cleared every blocking check in 57.7% of attempts.
Does the result mean AI-generated pull requests can be merged without review?
No. The benchmark’s leading model failed to clear all blocking checks in 42.3% of attempts. The result is specific to FrontierCode’s tasks and setup, but it does not support treating human review as unnecessary.
How much did the tested models cost per task?
Artificial Analysis reported costs from $0.09 for GPT-6 Luna (max) to $10.15 for Claude Opus 5.5 in the listed cost-frontier results. Costs depend on the tested configuration and should not be assumed to match other workloads.
Were all major AI models included?
No. The source report said Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode results as of October 13, 2026. Their performance remains unreported in this evaluation.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
