The code compiles. The tests pass. The function returns the correct output for every input you throw at it. By every metric the industry has relied upon for years, this is a success. And yet, when a developer looks at the generated code, something feels wrong. The variable names are cryptic. The documentation is missing. The error handling is non-existent. The style conventions the team spent months establishing have been cheerfully ignored. The code works, but it is not the code anyone asked for.
This gap between “functionally correct” and “actually good” has been hiding in plain sight for years, papered over by benchmarks that never thought to look for it. Now, a team of researchers led by Ming Zhong at the University of Illinois Urbana-Champaign and Google DeepMind has given this gap a name, a framework, and a set of numbers that should make every AI lab and engineering organisation sit up and pay attention.
Their paper, published at ICML 2026 as “SWE-IF: Aligning Code Evaluation with Human Preference” and circulated in preprint under the catchier title “Vibe Checker,” reveals something professional developers have long suspected: when it comes to judging AI-generated code, instruction following is the primary differentiator separating models humans prefer from models that merely produce working output. Even more troublingly, the research demonstrates that Claude 4 Opus, a frontier model rather than an also-ran, manages only a 46.75% success rate when asked to follow five instructions simultaneously. That is less than a coin flip.
The timing could not be more pointed. According to the 2025 Stack Overflow Developer Survey, 84% of developers now use or plan to use AI tools in their development process, yet more developers actively distrust the accuracy of those tools (46%) than trust them (33%). Two-thirds, 66%, report spending more time fixing AI-generated code that is “almost right, but not quite.” The trend has not reversed since. Stack Overflow's own follow-up analysis, published in February 2026, tracked trust in AI accuracy falling to 29% from 40% the previous year, even as adoption climbed. The 2026 Developer Survey opened on 23 June 2026 and had not reported at the time of writing. These numbers describe exactly the problem the SWE-IF research has now quantified: models that pass functional tests but fail the requirements that matter most to the humans using them.
The Benchmark That Measured the Wrong Thing
To understand why this matters, you need to understand what pass@k actually measures, and what it does not.
Since OpenAI introduced the HumanEval benchmark in 2021, the industry has treated functional correctness as the gold standard for code generation evaluation. The pass@k metric works like this: generate k code samples for a problem, run them against a test suite, and check whether at least one passes. There is no middle ground, no partial credit, no assessment of anything beyond “does it work?”
That binary approach made sense when getting a model to produce syntactically valid Python was itself an achievement. But contemporary models routinely achieve pass@1 rates above 90% on HumanEval, and the benchmark is, for practical purposes, saturated. Research from EvalPlus found its original test suites so insufficient that pass@k scores drop by 19.3% to 28.9% once more rigorous test cases are applied, and the problems skew overwhelmingly easy: 84.8% classified as “Easy,” only 0.6% as “Hard.”
The deeper problem is not test quality or difficulty distribution. It is that functional correctness captures a single dimension of what makes code good. HumanEval says nothing about maintainability, runtime performance, or whether code follows established conventions, includes proper documentation, or handles edge cases gracefully. Even BigCodeBench, which pushed evaluation towards realistic tasks involving diverse function calls across 139 libraries, found the best model of its day solving merely 60% of complete tasks against human performance of 97%. The gap between benchmark performance and real-world capability is not small. It is a chasm.
What none of these benchmarks measure is the constellation of non-functional requirements that occupy the bulk of a professional developer's attention: style conventions, documentation standards, error handling patterns, API usage constraints, and the dozens of other specifications that transform raw functionality into maintainable software. The ISO/IEC 25010 standard recognises this directly, treating structural quality as distinct from functional suitability. When a developer asks an assistant to “write a function that parses this JSON, use type hints throughout, add docstrings in Google style, handle KeyError exceptions explicitly, and keep line length under 88 characters,” pass@k cares about exactly one of those requirements. The rest are invisible.
Thirty Rules, Five Categories, One Reckoning
The SWE-IF research team, which includes senior research scientist Jiao Sun at Google DeepMind and is supervised by Jiawei Han at UIUC, set out to make these invisible requirements visible. Their approach was systematic, grounded in existing software engineering practice, and deliberately designed to be deterministic rather than subjective.
The centrepiece is VeriCode, a taxonomy of 30 verifiable code instructions organised into five categories: Coding Style and Conventions, covering the rules linters and formatters enforce, such as line length and naming; Logic and Code Patterns, addressing structural requirements like maximum function branches and complexity thresholds; Documentation and Commenting, dealing with docstring formats and documentation completeness; Error Handling and Exception Management, capturing requirements around try-except blocks and specific exception types; and Library and API Constraints, specifying which libraries or API patterns should or should not be used.
These categories are not arbitrary. They map to the dimensions of code quality professional developers care about daily. Qodo's “State of AI Code Quality” report found that the single most requested improvement to AI coding tools was not raw capability but improved contextual understanding, cited by 26% of developers and rising to roughly 30% once customisation to team standards is folded in. Developers are not, in the main, asking for models that can solve harder problems. They are asking for models that will do what they were told, the way their own team does it. That is a request for instruction following, and it is precisely what VeriCode was built to measure.
Twenty-seven of the 30 instructions are implemented as checks in Ruff, the Rust-based Python linter that has become the de facto standard for Python code quality verification, and which implements over 800 built-in rules at 10 to 100 times the speed of its predecessor, Flake8. The remaining three verifiers sit outside what an off-the-shelf linter covers, a small detail worth dwelling on: it means the taxonomy is not simply a repackaging of Ruff's rule book but a deliberate attempt to describe what developers actually specify, including a few things no linter ships with.
Crucially, every instruction comes with a deterministic verifier. There is no ambiguity, no subjective judgement, no need for another language model to act as judge, an approach that introduces exactly the noise and subjectivity earlier attempts at measuring code quality struggled with. Either the code follows the instruction or it does not. A machine can check. And because parameters can be varied (line length from 79 to 120 characters, docstring format from Google to NumPy style), the 30 base rules generate hundreds of distinct instruction variants, making memorisation nearly impossible and keeping the evaluation robust against the contamination that has plagued benchmarks like HumanEval.
Building the Testing Ground
With VeriCode in hand, the researchers constructed two complementary benchmarks designed to cover the spectrum of programming tasks developers actually encounter.
Big-SWE-IF extends BigCodeBench, a collection of 1,140 real-world programming tasks involving diverse function calls and complex instructions across seven domains. BigCodeBench was itself built through systematic human-LLM collaboration: starting from real developer intents harvested from Stack Overflow, twenty human experts, most with more than five years of Python experience, refined and validated every task inside an execution-based sandbox, producing an average of 5.6 test cases per task at 99% branch coverage.
Live-SWE-IF extends LiveCodeBench, which draws 1,055 algorithmic tasks from competitive programming platforms like LeetCode, AtCoder, and CodeForces. Its critical advantage is that new problems are continuously collected after model training cutoff dates. Problems are annotated with release dates, so for any model with a known cutoff, scores can be computed exclusively on problems it could not have seen during training.
For each task, an LLM-based selector chooses relevant, non-conflicting instructions from the VeriCode taxonomy, so models are never asked to follow arbitrary or contradictory rules. They receive instructions a reasonable developer might actually specify. The evaluation runs in two modes: single-turn generation, where all instructions are presented at once, and multi-turn editing, where they are added in stages. Both test functional correctness and instruction following simultaneously.
The researchers then evaluated 31 leading language models from 10 model families, spanning Gemini, Claude, OpenAI, DeepSeek, Qwen, Grok, Gemma, Mistral, MiniMax, and Kimi. The results were sobering.
The Numbers That Should Worry Everyone
When models were asked to follow a single instruction alongside producing functionally correct code, performance was reasonable. Most leading models handled one constraint without significant difficulty. But as the number of simultaneous instructions increased, performance degraded in ways that reveal fundamental limitations in how these systems process and prioritise requirements.
The clearest evidence comes from the multi-turn editing condition, where instructions arrive in stages rather than all at once, much as they do in a real code review. On Big-SWE-IF, adding five instructions this way cut the average pass@1 rate by 5.85%. That is not a trivial drop. It represents a measurable loss of functional correctness caused by nothing more than the presence of additional non-functional requirements. The models were not being asked to do harder computational work. They were being asked to write the same code while also adhering to style and documentation conventions, and the effort of following those conventions caused them to break the code itself.
On Live-SWE-IF, the pattern holds but distributes unevenly across models, which is arguably more troubling than a uniform decline would be. For some systems the degradation is modest. For others, o4-mini and Kimi K2 among them, it exceeds 10%. A drop of that magnitude is not sampling noise. It means that for particular models, telling them how you want the code written measurably reduces their chance of writing code that works at all. And because the effect is concentrated in specific models rather than spread evenly, it is invisible to any evaluation reporting only an average.
The headline numbers are worse still. With five instructions applied simultaneously, the best result on Big-SWE-IF belongs to Claude 4 Opus, at 46.75%. On Live-SWE-IF the ceiling is 40.95%. These are not mid-tier models struggling with an unfair test. This is the frontier. And with three or more instructions, most advanced models fall below 50 across both benchmarks. Consider what that means in practice: give one of the best code generation models in the world a moderately complex task with five reasonable constraints (use type hints, add docstrings, handle exceptions, follow a naming convention, keep functions under a certain length) and it will satisfy all of them less than half the time. For models outside the top tier, failure rates are considerably worse.
This phenomenon, which the researchers term “functional regression,” is particularly insidious. Adding perfectly reasonable, non-conflicting instructions does not merely cause the model to miss those instructions. It actively degrades the model's ability to produce correct code in the first place. The instructions are not just ignored; they interfere with the core capability. Think of it as asking a chef to prepare a dish while also specifying plating, garnishing, and seasoning. The additional requirements should not make the food taste worse, yet with language models the analogous degradation is measurable and consistent.
Ninety-Five Per Cent, and Under Fifty
It would be reasonable to assume a result like this ages badly. Benchmarks fall. Models improve. A 46.75% score recorded against the frontier of late 2025 ought to look quaint within a year, overtaken by the next generation the way HumanEval was overtaken.
That is not what happened. What happened instead is that the two halves of the problem came apart.
On functional correctness, the past year has been a rout. The top of the SWE-bench Verified leaderboard now sits at 95% and above: Claude Fable 5 records 95.0%, with Claude Opus 5, released on 24 July 2026, reported higher still, and Claude Opus 4.8 at 88.6% before it. Gemini 3.1 Pro sits at 80.6%. GPT-5.6 Sol reached general availability on 9 July 2026. Kimi K3 ranks third on the Artificial Analysis Intelligence Index and first on Frontend Code Arena. On SWE-bench Pro, a deliberately harder successor built to resist exactly this kind of saturation, the leaders have already reached roughly 80%.
Those figures deserve one caveat. Leaderboard positions shift monthly, and published scores frequently fail to distinguish between standardised harnesses and vendor-specific scaffolding, a difference that can move a number by several points. But the direction of travel is not in dispute. Resolving real GitHub issues, a task considered a serious open research problem as recently as 2023, is now something the best models do roughly nineteen times out of twenty.
Now set that against the other number. Ninety-five per cent on functional correctness. Under fifty on instruction following at five constraints. Functional correctness has been substantially solved at the frontier. Instruction fidelity has not moved with it.
This is the whole argument, and the past year has widened it rather than closed it. The industry poured extraordinary resources into the dimension it could measure and received extraordinary returns. The dimension it was not measuring stayed roughly where it was. Every point of SWE-bench progress since has been earned on the axis that was already winning, which means the distance between what these models can do and what developers actually ask them to do is now greater than at any previous point in the history of code generation. We have built systems that can solve the problem and cannot reliably be told how.
Lost in the Middle of Your Prompt
Perhaps the most revealing finding is what the researchers call the “lost-in-the-middle” effect for instruction following. The phenomenon was first characterised in the broader language model context by Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang in their influential 2024 paper in the Transactions of the Association for Computational Linguistics. Working at Stanford, they demonstrated a distinctive U-shaped performance curve: model performance was highest when relevant information appeared at the very beginning or the very end of the input context, and degraded significantly when models had to retrieve information from the middle. Subsequent work has connected this to architectural properties of transformers, specifically the interaction between positional embeddings and causal attention masks, with some researchers reframing the effect not as a bug but as an emergent property of autoregressive pre-training.
The SWE-IF team found the same U-shaped curve applies specifically to code generation instructions. Models are less likely to follow constraints appearing in the middle of a prompt than those at either end. This transforms an abstract observation about attention patterns into a concrete software engineering problem.
The implications for daily development work are profound, because in professional software development requirements are rarely ordered by importance. They are organised by category or logical grouping. A developer asking for “type hints, Google-style docstrings, exception handling for network errors, maximum line length of 88, and use of the requests library” has no reason to expect the exception handling requirement to be treated as less important simply because it sits third in a list of five. A competent human programmer would read all five, understand them as a single specification, and satisfy every one regardless of position. But language models systematically deprioritise middle-positioned constraints. The instruction governing how the code handles failure, arguably the most consequential requirement for production reliability, is the one most likely to be silently discarded.
What Humans Actually Want
The most consequential finding emerges from the comparison with real human preferences. The team analysed over 800,000 human votes from the coding subset of LMArena (formerly Chatbot Arena), where users compare outputs from different models in blind pairwise comparisons, aggregated into Elo ratings. This is not a small or synthetic dataset. It represents the accumulated preferences of real developers making real choices about code they intend to use.
They found that combining functional correctness and instruction following produced a composite score substantially more predictive of human choice than either measure alone. Traditional benchmark rankings, the researchers noted, often showed little or even negative correlation with what human evaluators actually prefer. This is a striking claim. It means the leaderboards the industry uses to compare models are not merely incomplete; in some cases they are actively misleading. An organisation choosing its AI coding tool on the basis of HumanEval rankings might systematically select the model least aligned with what its developers want. Copilot Arena, a Visual Studio Code extension built by researchers at Carnegie Mellon, UC Berkeley, MIT, and Cornell, reached the same conclusion from a different direction: across more than 25,000 code completion battles, it found rankings drawn from real developer preferences correlate poorly with most traditional benchmarks, with smaller models that overperform on static evaluations frequently underperforming when actual developers judge their output.
The correlation data also reveals an important contextual distinction. For everyday programming, the work most developers do most of the time, involving web development, data processing, API integration, and utility scripting, instruction following emerged as the main differentiator among advanced models. Once models clear a threshold of functional correctness, what separates the ones developers prefer is how well they follow the non-functional requirements embedded in the prompt. For competitive algorithmic problems, by contrast, functional correctness still dominates: when the task has a single correct answer, style matters less than output. But competitive programming is a tiny fraction of real-world software development. The vast majority of code written on any given day is building applications, maintaining systems, integrating services, and extending existing codebases. For that work, instruction following is what matters.
The Training Pipeline Problem
These findings expose a fundamental misalignment in how models are trained for code generation. The dominant paradigm, Reinforcement Learning with Verifiable Rewards (RLVR), uses pass@k as its primary reward signal. Models are trained to maximise functional correctness because that is what the verifier can check. The reward is binary, and the optimisation pressure is entirely focused on producing code that works.
This has been remarkably effective. Models like DeepSeek R1 scaled RLVR with rule-based rewards for mathematics, code, and logic, and every subsequent generation of reasoning models has pulled the same lever harder, coupling reinforcement learning with tool use to produce exactly the SWE-bench numbers described above. The lever works. That is the problem. It works on one axis, and the industry has spent a year pulling it.
When RLVR trains a model to maximise pass@k, it implicitly teaches that nothing matters except getting the tests to pass. Style conventions, documentation, error handling, API constraints: all orthogonal to the reward signal. At best the model learns them incidentally from training data. At worst the optimisation pressure works against them, because following additional constraints reduces the probability of producing functionally correct code, exactly as the functional regression finding demonstrates.
VeriCode offers a path forward. Because each instruction carries a deterministic verifier, the taxonomy can be integrated directly into RLVR pipelines as an additional reward signal. Instead of rewarding models solely for code that passes tests, training could reward code that passes tests while also following the specified instructions. The verifiers are automated, scalable, and objective: precisely the properties reinforcement learning rewards require.
That proposal has begun to be acted upon. Multi-component RLVR reward designs now in circulation use Ruff-detected lint, style, and vulnerability signals as reward components alongside test-passing, treating code quality as a first-class training objective rather than a hoped-for side effect. A forward-looking suggestion buried in the discussion section of a 2025 preprint has, inside a year, become an active line of work.
It also carries a hazard the original proposal named only in passing. The moment a linter becomes part of a reward function, it becomes a target, and Goodhart's law applies to reinforcement learning with unusual force. RLVR is already known to be prone to over-optimisation, in which models exploit verification shortcuts that satisfy the checker without satisfying the intent behind it: reward hacking, in the field's terminology. Work such as IFDecorator addresses this directly for instruction following, wrapping RLVR training in intent-alignment checks and deliberately planted “trip wire” instructions designed to catch a model in the act of gaming its verifier. The lesson is not that verifiable rewards for instruction following are a bad idea. It is that a model trained to satisfy Ruff will learn to satisfy Ruff, and whether it has also written good code remains, stubbornly, a separate question.
What This Means for Engineering Organisations
For teams relying on AI coding assistants, these findings carry immediate practical implications. The first is that prompt engineering is more consequential than most teams realise. Because of the lost-in-the-middle effect, ordering matters: placing the most critical non-functional requirements at the beginning and end of prompts, rather than burying them in the middle, can meaningfully improve compliance. This costs nothing to implement.
The second is that teams should not trust AI-generated code to follow specifications without verification. A 46.75% success rate at five simultaneous instructions means that more than half the time, even the best models will miss at least one requirement. Automated verification, using linters like Ruff configured to match team standards, becomes not a nice-to-have but a necessary component of any AI-assisted workflow. Code review needs to check specifically for instruction compliance, not just functional correctness. In most organisations the infrastructure to do this already exists. What needs to change is the focus of the review it performs.
The third concerns tool selection. If traditional benchmarks correlate poorly with human preference, organisations making purchasing decisions on HumanEval scores are optimising for the wrong thing. Teams should evaluate tools against their own standards and conventions, testing whether a model produces code meeting their particular requirements for style, documentation, error handling, and API usage. A model scoring five points lower on a public leaderboard but consistently following your team's conventions may be the better choice.
There is also an organisational design consideration. As AI handles more routine code generation, the role of senior developers shifts towards specification and review, and the ability to catch the instructions a model missed becomes the primary quality assurance function. The Atlassian 2025 State of Developer Experience report found developers spend only 16% of their time coding, with 50% losing ten or more hours per week to non-coding tasks and organisational inefficiencies. The picture has improved since: Atlassian's 2026 research into AI-native development, drawing on 3,400 repositories across 2,500 customers, found teams merging 19% more pull requests per month and saving two to three hours per developer per week, with 99% reporting some time saving and 68% saving ten or more hours weekly. But time returned at the point of generation can be spent again at the point of review. If these tools are to genuinely improve productivity, they need to reduce the review burden, not relocate it. That means following instructions the first time.
A Subfield Forms Around the Gap
When this work first appeared, it read as an isolated finding: one team, one taxonomy, one uncomfortable number. It no longer does. In the months since, instruction following in code generation has acquired the unmistakable features of a research subfield, complete with independent replication, competing benchmarks, and a workshop of its own.
The most important corroboration came from outside the original group. CodeAlignBench, released by a team at Apple in October 2025, took a deliberately different route to the same question. Rather than deriving instructions from a linter's rule set, its authors ran a user study with working developers across three programming languages and built the benchmark from the adjustments those developers actually asked for. It evaluates both adherence to constraints specified up front and the ability to act on follow-up refinements, and it agrees with human judges 87% of the time on whether an instruction was followed. Its findings are hard to wave away: frontier model scores spread across a range of roughly 30 percentage points, and, crucially, the resulting ranking does not mirror the ranking those same models achieve on functional correctness. Two independent teams, different methodologies, different instruction sources, same conclusion. The models developers prefer are not the models the leaderboards promote.
CIFE, published in December 2025, sharpened the question by asking not whether models follow instructions but how nearly they do. Its 1,000 Python tasks carry an average of seven developer-specified constraints across thirteen categories, and its authors evaluated fourteen open and closed models against a composite C2A Score designed to capture correctness and constraint compliance jointly rather than trading one against the other. The result is perhaps the most diagnostically useful finding in the entire literature: there is a large gap between partial and strict constraint satisfaction, with strong models clearing 90% on partial adherence. Read that slowly. These models are not ignoring instructions. They are very nearly following them, satisfying most of what was asked, missing some fraction of it, and producing output that is correct in outline and wrong in detail. Which is, almost word for word, the complaint two-thirds of developers make about AI-generated code: almost right, but not quite. The benchmark has found the mechanism behind the survey response.
Then the question moved into the environment where most professional AI coding now actually happens. OctoBench, accepted at ACL 2026, dropped the single-prompt framing entirely and asked how models handle instructions inside agentic, repository-grounded work: 34 environments and 217 tasks instantiated across three scaffold types, scored against 7,098 objective checklist items, over eight representative models. It found the same systematic gap between solving the task and complying with the constraints surrounding it. This matters more than it might first appear. The deficit was originally measured where a human writes a prompt and reads an answer. OctoBench establishes that it survives translation into agent harnesses, where constraints are heterogeneous, persist across many turns, and go unenforced by anyone until something breaks. The deficit follows the models into the tools built on top of them, and in 2026 those tools are where the code comes from.
The institutional apparatus has caught up too. VeriCodeGen, a full-day NeurIPS 2026 workshop on AI for verifiable coding, convenes in Atlanta this December. The gap now has a venue.
Rethinking Evaluation From First Principles
What SWE-IF added to a landscape already in flux was a rigorous, deterministic framework for measuring the dimension that most strongly predicts human preference. Its own history since publication contains a small and telling irony. The paper began life in October 2025 as “Vibe Checker,” a title trading on the coinage Andrej Karpathy had introduced on 2 February 2025 to describe developers accepting AI-generated code without fully comprehending its functionality, an approach Karpathy allowed was “not too bad for throwaway weekend projects.” By its second revision in June 2026, on the way to peer review at ICML, the vibes were gone. The framework had become SWE-IF, BigVibeBench and LiveVibeBench had become Big-SWE-IF and Live-SWE-IF, and the paper presented itself as what it had always actually been: a software engineering instruction-following benchmark. Only VeriCode, the taxonomy at its centre, kept its name.
The renaming is worth a moment's attention, precisely because nothing else changed. The phenomenon the work measures was entirely unaffected by the loss of the branding. The 46.75% did not move. What the rename marks is a shift in how the field regards the problem: not a cultural observation about how people are using these tools, worth a knowing joke in a title, but an engineering deficiency with a number attached, submitted for peer review under a name that simply describes it. Vibe coding was a mood. Instruction-following fidelity is a measurement. The eight months between the two titles are roughly the period in which the industry stopped finding the first framing funny.
What remains is a challenge nobody should mistake for a rounding error. A sub-50% success rate at five instructions is not a gap to be closed by incremental improvement. It is a structural problem in how language models process and prioritise competing requirements, and closing it will likely require architectural innovation, changes to training methodology, and evaluation frameworks that go well beyond tweaking what already exists.
For the broader industry, the message is clear: the benchmarks we use shape the models we build. When pass@k is the only metric that counts, we get models excellent at producing code that passes tests and mediocre at everything else. When instruction following enters the evaluation framework, we get models that write code the way developers actually ask for it. The question is not whether this shift will happen, but how quickly, and how much accumulated code we live with in the meantime.
The code compiles. The tests pass. But does it follow the instructions? That, it turns out, is the question that actually matters.
