Sixteen point two per cent. That is how often people in one 2026 experiment caught a machine's mistake once it had explained itself in fluent, confident prose. Swap the explanation for a bare number, the model's own estimate of how likely its answer was to be right, and the rate more than doubled, to 37.5 per cent.
The figures come from controlled studies by Ruth Cohen and Sarit Kraus of Bar-Ilan University, Lu Feng of the University of Virginia and Ayala Bloch of Ariel University, posted to arXiv as paper 2604.03237. The first version had a title built for the conference circuit: “The Persuasion Paradox: When LLM Explanations Fail to Improve Human-AI Team Performance”. Its abstract did not hedge: “We identify a Persuasion Paradox: fluent explanations systematically increase user confidence and reliance on AI”. The authors argued that interfaces should put calibrated reliance and recovery from errors ahead of persuasive fluency.
Then something unusual happened. On 18 August 2026 the authors posted a revision with a new title, “Supporting Calibrated Reliance in Human-AI Collaboration: Different Strategies for Different Tasks”. The word “persuasion” has vanished from the text. The 16.2 and 37.5 per cent figures remain, but the revised statistics report that differences in error recovery across the four support conditions did not reach significance (p = .055). The new abstract closes on a far less quotable line: “no single support strategy is universally effective.”
Hold both versions in your head, because between them they contain the whole argument. The first is the story everyone wants to tell: explanations are a trap, and the explainable AI movement has been building persuasion machines. The second is the story the evidence supports: explanations reliably raise confidence, do not reliably raise accuracy, and help or hurt depending on whether the reader can check what they say. A paper about the dangers of persuasive fluency was, in its first draft, a little too persuasive. Its authors noticed and pulled it back. That is calibration in action, and it is exactly the behaviour designers should be trying to produce in everyone else.
So the useful question is not whether explanations are good or bad. It is narrower and harder. When should a system withhold its explanation, or restructure it? And what would an interface look like if its job were not to help people understand the AI, but to help them know when to ignore it?
Sixteen Per Cent and a Change of Title
Cohen and colleagues ran three experiments with participants recruited through university mailing lists, mostly people with STEM backgrounds, who were paid a bonus for each correct answer. Two tasks carried the weight: RAVEN matrices, abstract visual puzzles in which you choose the shape that completes a pattern, and logical reasoning questions from the LSAT, the American law school admission test.
The cleverest design was the multi-stage reveal. Twenty-seven participants answered each RAVEN puzzle three times: first alone, then after seeing the AI's predicted answer, then after reading an explanation, either a written rationale from a large language model or a heatmap showing which parts of the image the model weighted. Alone, they were right 37.0 per cent of the time. With the AI's answer, 49.8 per cent. With the explanation on top, 48.8 per cent. The explanation bought nothing in accuracy.
It did buy something else. In the paper's words, “confidence remained largely unchanged after the prediction stage and increased only after explanations were presented”. The prediction made people more accurate without making them feel surer. The explanation made them feel surer without making them more accurate. If you wanted one result to sum up the worry about fluent AI, that is it.
The between-subjects RAVEN study, with 100 participants across five conditions, is where the famous numbers live. Every form of AI support beat working alone by a wide margin: unaided participants managed 24.6 per cent, while the four AI-assisted groups landed between 53.5 and 60.5 per cent, with predicted probabilities at the top. On overall accuracy, though, the AI-supported conditions were statistically indistinguishable. The difference appeared in how people handled the AI's mistakes. “LLM explanations showed the lowest recovery rate (16.2%) despite high agreement (84.2%),” the authors write. The probability condition showed the highest recovery, 37.5 per cent, with 75.8 per cent agreement on correct predictions.
Read those pairs together. The explanation group agreed with the AI more when it was right (a gap that, like the recovery gap, fell short of significance) and caught it less when it was wrong. That is the signature of reliance that has stopped discriminating: trust that rises evenly across good and bad advice instead of tracking which is which.
The caveats matter, and the revised paper is admirably blunt about them. There was no power analysis in advance, and twenty people per condition is a small sample. The RAVEN probabilities were raw, uncalibrated softmax outputs. A “selective automation” strategy, in which the AI decides confident cases and hands uncertain ones to the human, is now presented only as a benchmark derived after the fact from the probability group's data. The pattern is consistent and striking. It is not yet proof.
The Same Explanation Wins at Logic
The LSAT study rarely makes it into social media summaries, and it is the part that turns a scare story into a design problem.
Eighty participants worked through ten LSAT questions with help from an AI whose answers were right 60 per cent of the time. One group saw only the AI's answer, one saw the answer plus a stated probability, one saw an expert-written explanation adapted from earlier research, and one saw an explanation generated by a large language model.
The language model's explanation won outright. That group reached 72.5 per cent accuracy, against 55.5 per cent for expert explanations, 48.5 per cent for the bare prediction and 47.0 per cent for probabilities. It was the only group to beat the AI's own 60 per cent, and it recovered best from errors, at 47.5 per cent against 35.0 per cent for probabilities. It even beat the human experts' explanations, which rules out the comforting idea that machine prose is simply worse prose. Here the fluent explanation was not a sedative. It was a tool.
Why would the same kind of support hurt in one task and help in another? The authors are careful not to overclaim, noting that task, model and explanation style all changed between studies. But the contrast lines up with an idea that runs through the field. An LSAT explanation lives in the same medium as the problem. If a model says an argument assumes that correlation implies causation, you can reread the passage and see whether it does. The explanation offers a claim the reader can test against evidence already in front of them.
A verbal rationale for a visual puzzle is different. It translates a pattern into words, and the reader has to translate it back. That is laborious, so most people do not bother. They register that the paragraph sounds like reasoning and move on. The explanation becomes a vibe rather than an argument.
In 2023, Helena Vasconcelos and colleagues at Stanford University and the University of Washington, including Michael Bernstein, Tobias Gerstenberg and Ranjay Krishna, gave this intuition a formal shape. Across five studies with 731 participants navigating a maze with a simulated AI, they argued that people make a strategic cost-benefit choice about whether to engage with an explanation at all. Task difficulty, explanation difficulty and money all shifted how much people over-relied on the AI. Earlier studies that found explanations useless, they suggested, may simply have used explanations that did not make checking the AI any cheaper. Their title made the counterintuitive point: “Explanations Can Reduce Overreliance on AI Systems During Decision-Making”.
Put Cohen's LSAT result beside that framework and a first principle falls out. An explanation earns its place when it lowers the cost of verification. When it raises that cost, or supplies the feeling of verification without the substance, it should be withheld or rebuilt.
Editing the Machine's Mind Feels Like Power
If reading an AI's reasoning can make you overconfident, perhaps the fix is to let users get their hands on it. Interactive reasoning, where people inspect and change a model's chain of thought, has become a fashionable idea in human-computer interaction. A short paper at CHI 2026, the field's flagship conference, held in Barcelona in April, suggests the fashion deserves scrutiny.
Jiwon Moon of the University of Chicago, with Chien-Ming Huang and Ziang Xiao of Johns Hopkins University, published “Understanding the Affordances of Control in AI Reasoning for Human-AI Decision-Making” in the conference's Extended Abstracts, the venue for shorter, often preliminary work. They compared three interfaces in a decision task: no chain of thought, a read-only chain of thought, and an editable one that let users alter the AI's displayed reasoning.
Their abstract captures the gap between how an interface feels and what it does. “Editability significantly increases perceived power, control and satisfaction, but does not improve decision accuracy relative to a no-rationale baseline.” The read-only version was worse: it “increases agreement with the AI and induces inappropriate reliance when the AI is wrong”, reducing accuracy compared with editing and trending below the baseline in which people saw no reasoning at all.
The mechanism is the interesting part. Mediation analyses, the authors report, suggest that editability raises perceived power, which in turn predicts higher confidence and greater over-reliance on incorrect advice, a pattern “consistent with illusion of control.”
The phrase comes from a 1975 paper by the psychologist Ellen Langer in the Journal of Personality and Social Psychology. Across experiments built on games of pure chance, Langer argued that features associated with skill, such as choice, familiarity, involvement and competition, lead people to expect success beyond what the odds justify. Her best-known demonstration, in which people who picked their own lottery ticket asked far more to sell it than people who were handed one, has not aged well: in 2021 Joowon Klusowski, Deborah Small and Joseph Simmons reported in Psychological Science that experiments with more than 10,000 participants found choice alone did not make people think they were more likely to win.
That wrinkle is instructive rather than fatal. Moon, Huang and Xiao measured perceived power directly and linked it to confidence and over-reliance in their own data. But it is a reminder that a vivid label can travel further than the evidence beneath it, which is the subject of this article. Because the CHI paper is an extended abstract, its effect sizes should be treated as provisional. The direction of the result is still a warning to anyone who treats “user control” as an automatic good. Control that is not connected to better outcomes is a feeling, and feelings are what persuasion is made of.
Thirty Years of Leaning on Machines
None of this is new in kind. It is new in fluency.
In 1997, Raja Parasuraman and Victor Riley published “Humans and Automation: Use, Misuse, Disuse, Abuse” in the journal Human Factors. Misuse, in their taxonomy, is over-reliance on automation, producing monitoring failures and decision biases. Disuse is neglect of automation that works, often because it has cried wolf too often. Abuse is automating without regard for the consequences for human performance. All three map onto modern AI assistants, but misuse is the one fluent explanations feed.
In 2004, John Lee and Katrina See gave the field its central goal in the title of their review in the same journal: “Trust in Automation: Designing for Appropriate Reliance”. The key idea is calibration. Trust should match a system's actual trustworthiness, neither overshooting nor undershooting. And it has to work at a fine grain: trusting a system the right amount on average is not enough. You need to trust it more where it is good and less where it is bad.
Medicine has the clearest data on what happens when that fails. A 2012 systematic review in the Journal of the American Medical Informatics Association by Kate Goddard, Abdul Roudsari and Jeremy Wyatt drew on 74 studies of automation bias, which they defined as “the tendency to over-rely on automation”. Pooling four healthcare studies in which decision support systems gave wrong advice, they found the systems increased the risk of an incorrect decision by 26 per cent. The accelerants were familiar: high trust, low self-confidence, workload, complexity, time pressure. Among the design mitigations, they listed attaching updated confidence levels to the system's output, and offering information rather than direct recommendations.
A review published years before large language models went mainstream recommended, in effect, what the RAVEN probability condition did: show people how sure the system is, case by case.
The idea is now law. Article 14 of the European Union's AI Act, on human oversight of high-risk systems, requires that overseers be enabled “to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)”, particularly where systems provide information or recommendations for decisions taken by people. Those obligations do not bite yet, either: the Digital Omnibus on AI, in force since July 2026, pushed the high-risk deadlines back to December 2027 for standalone systems and August 2028 for AI built into regulated products. The regulation names the problem. It does not specify the interface that solves it. That is the designer's job, and the evidence suggests a fluent paragraph of reasoning is not, by default, the answer.
Explanations Help Most When You Need Them Least
The most cited study on this question predates the language model boom and predicted much of what followed. In 2021, Gagan Bansal and colleagues from the University of Washington and Microsoft Research, among them Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro and Daniel Weld, presented “Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance” at CHI.
Their target was complementary performance: a human-AI team that beats both the human alone and the AI alone. Earlier studies showing that explanations helped had mostly used AI far more accurate than people, so teams improved simply by deferring to a better decision-maker. Bansal's group used AI roughly as accurate as humans, across three tasks (sentiment in beer reviews and book reviews, plus LSAT questions), with 1,626 participants.
Teams did achieve complementary performance, but explanations did not improve on simply showing the AI's confidence. The mechanism was an asymmetry. “Explanations often increased accuracy when the AI system was correct but, worryingly, decreased it when the AI erred, resulting in a minimal net change.” Explanations made people more likely to accept the AI's recommendation whether or not it was right.
Bansal's team tried a fix. Their “adaptive explanation” explained only the top answer when the AI was confident, but laid out the top two alternatives when confidence fell below a threshold, inspired by an efficient assistant that divulges more, “confessing doubts and arguing for alternatives”, only when unsure. It did not significantly improve team performance, though the authors saw suggestive evidence that it nudged agreement in the right direction. Keep it in mind: it is the seed of a better idea.
In 2023, Max Schemmer and colleagues at the Karlsruhe Institute of Technology gave the asymmetry two names. At the Intelligent User Interfaces conference, they proposed measuring “appropriateness of reliance” on separate axes. Relative AI reliance captures how often people switch to correct AI advice after starting with a wrong answer. Relative self-reliance captures how often people keep their own correct answer when the AI is wrong. In their 200-person experiment, explanations raised relative AI reliance from 29.59 to 38.87 per cent. Relative self-reliance did not significantly change.
That is the pattern in miniature. Explanations help people accept the AI when it is right. They do far less to help people resist it when it is wrong. And resistance is the whole point of a human in the loop. If the machine were never wrong, you would not need the person.
Why Fluency Reads as Truth
Large language models sharpen the problem because their explanations are not heatmaps or feature weights. They are prose, and prose is the native format of human persuasion.
In 2025, Mark Steyvers and colleagues at the University of California, Irvine published “What large language models know and what people think they know” in Nature Machine Intelligence. They measured the calibration gap: the difference between how confident people are in a model's answers and how confident the model itself is. Shown default explanations, people overestimated the model's accuracy, and longer explanations raised their confidence even when the extra length did nothing for accuracy. “There's a disconnect between what LLMs know and what people think they know,” Steyvers said in the university's announcement. “We call this the calibration gap.” When the team rewrote explanations so their language reflected the model's internal confidence, that gap narrowed, and people got better at telling right answers from wrong ones.
A year earlier, Sunnie Kim of Princeton University, with Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard and Jennifer Wortman Vaughan of Microsoft, tested something similar. In a pre-registered experiment with 404 participants answering medical questions with a simulated LLM-powered search engine, first-person expressions of uncertainty, along the lines of “I'm not sure, but...”, reduced people's confidence in the system and their agreement with it, and improved their accuracy. Impersonal hedges had weaker effects that did not reach significance. Wording matters enough, the authors concluded, that it needs user testing before deployment.
Back in 2020, Yunfeng Zhang, Q. Vera Liao and Rachel Bellamy of IBM Research found that confidence scores helped people calibrate their trust, while explanations of individual predictions did not do the same job. Their caveat still applies: calibrated trust does not guarantee better decisions unless the human brings knowledge that can compensate for the model's errors. Calibration tells you when to be suspicious. It does not tell you the right answer.
Fluency, in other words, is not a neutral property. Length, confident tone and smooth reasoning all signal reliability without being connected to it. A well-designed system should make those signals honest, or remove them.
Friction Works and Nobody Likes It
If people skim explanations and defer, the obvious response is to make deferral harder. Zana Buçinca, Maja Barbara Malaya and Krzysztof Gajos of Harvard University tested exactly this in “To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making”, published in 2021 in the Proceedings of the ACM on Human-Computer Interaction.
Borrowing from research on diagnostic error in medicine, they built three interventions into a task in which 199 participants judged how to cut the carbohydrate content of meals. In the “on demand” condition, the AI's suggestion stayed hidden until people clicked to see it. In “update”, people decided first and could revise after seeing the AI's advice. In “wait”, a message saying the AI was processing the image appeared for 30 seconds before the suggestion arrived.
All three significantly reduced over-reliance compared with simple explainable AI interfaces. None eliminated it: the interventions “reduced, but not yet eliminated, overreliance on the AI”, and the human-AI teams still underperformed the AI alone. Then the twist. When the AI was wrong, “people performed best in conditions that they preferred and trusted the least”, which they also rated most difficult. The interventions also helped people high in “need for cognition”, a measure of how much someone enjoys effortful thinking, more than others, raising the question of who friction actually protects.
This is the commercial problem in one finding. Every metric a product team watches, from satisfaction scores to stated trust, rewards the smooth, confident explanation. The metric that matters, whether people catch the machine's errors, rewards friction. Moon, Huang and Xiao found the same split from the other side. And a 2026 CHI review of reliance research by Muhammad Raees, Vassilis-Javed Khan, Ioanna Lykourentzou and Konstantinos Papangelis made the general point: user engagement is valuable for appropriate reliance, but it comes at a cost.
When to Withhold the Explanation
So when should a designer leave the explanation out? The evidence supports no blanket rule, and the revised Cohen paper is explicit that no single strategy wins everywhere. But it does point to conditions under which a default explanation is more likely to hurt than help.
The first is when the explanation cannot be checked against evidence the user can see. The RAVEN rationale asked people to verify a verbal account of a visual pattern; the LSAT rationale asked them to verify a verbal account of a verbal argument. Where explanation and evidence live in different media, or the evidence is hidden, the explanation tends to function as a confidence signal rather than an argument. Vasconcelos's cost-benefit framing predicts exactly this.
The second is when fluency is decoupled from reliability. A language model can write an equally polished rationale for a right answer and a wrong one, and Steyvers's results show that people read polish and length as accuracy. If a system cannot make the style of its explanation track its actual confidence, the explanation is broadcasting a false signal.
The third is when the user cannot bring independent knowledge. Zhang, Liao and Bellamy's caveat applies: suspicion helps only if the human can act on it. Goddard's review identified time pressure, workload and complexity as amplifiers of automation bias, precisely when people are least able to scrutinise a paragraph of reasoning. In those conditions, a persuasive explanation mostly accelerates deference.
The fourth is before the user has formed a view. Buçinca's update condition and Cohen's multi-stage design both suggest timing matters. An explanation shown before someone has thought for themselves anchors them. Shown afterwards, as a challenge rather than a verdict, it does a different job.
None of this means explanations should vanish. Withholding an explanation from the moment of decision is not the same as withholding it from the record. Engineers debugging a model, auditors checking it and people contesting its verdicts may all need a full account. The question here is narrower: what to put in front of someone in the seconds when they decide whether to accept the machine's answer. The default, an explanation attached to every answer and displayed immediately, is the wrong default for a large class of tasks.
How to Rebuild an Explanation
Withholding is the blunt option. Restructuring is more interesting, and the literature offers several patterns.
Lead with uncertainty rather than reasoning. In Cohen's RAVEN study, the probability display recovered best from errors and was rated clearest by participants, despite explaining nothing. Goddard's review flagged confidence displays more than a decade ago, and Kim's work shows uncertainty can be expressed in natural language, if the phrasing is tested. A number or a well-chosen hedge is a cue to look harder, cheap to read and hard to mistake for an argument.
Make explanations conditional, in the spirit of Bansal's adaptive design. When the model is confident and historically reliable on this kind of case, a short answer may suffice. When it is uncertain, it should do the opposite of what fluent assistants usually do: argue against itself, present the strongest alternative, and show what evidence would change its answer.
Stop recommending and start evaluating. In a 2023 paper at the ACM Conference on Fairness, Accountability and Transparency, titled “Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven decision support”, Tim Miller, then at the University of Melbourne, argued that recommendations plus explanations undermine human agency and do not match how people reason. He proposed “evaluative AI”, tools that “provide evidence for and against decisions made by people” rather than handing them a decision to accept or reject. The human proposes; the machine stress-tests. There is nothing to be persuaded of, because the machine is not selling an answer.
Sequence the interaction. Ask for the human's judgement first, reveal the AI's view second, and explain only where they disagree. That combines Buçinca's update intervention with adaptive explanation, concentrating scarce attention where it could change the outcome.
Route the cases. Machine learning researchers have long studied “learning to defer”; a 2020 paper at the International Conference on Machine Learning by Hussein Mozannar and David Sontag, for instance, developed methods for training a model that either predicts or hands the decision to a human expert. Cohen's selective-automation benchmark gestures at the same idea, and the authors note that humans can keep final authority under such schemes. Human scepticism is a scarce resource and should be spent where the model is weakest.
A Framework Built Around Distrust
Pull these threads together and a framework for calibrated reliance emerges. Its goal is not understanding but discrimination: helping people tell the machine's good answers from its bad ones, case by case. What follows is a synthesis of the research above, not a standard anyone has yet adopted.
Measure the right thing. Evaluate a system separately on cases where the AI is right and cases where it is wrong, which is the first recommendation in the revised Cohen paper. Schemmer's two axes offer one method; error recovery offers another. Overall accuracy and trust ratings on their own are close to useless, because the Cohen, Bansal, Buçinca and Moon studies all show satisfaction and confidence moving independently of decision quality.
Make every confidence signal honest. If explanations sound equally sure whether the model is right or wrong, their style is lying. Either calibrate the language to the model's real confidence, as Steyvers's team did, or strip out the persuasive styling and show a calibrated number. One caveat: Cohen's RAVEN probabilities were not carefully calibrated, and the authors warn that uncertainty alone can miss errors the model makes confidently. Probability displays are a starting point, not a solution.
Match support to the task's evidence. Where users can verify an explanation against material in front of them, as with the LSAT, rich explanations can pay their way. Where they cannot, prefer uncertainty cues, contrastive alternatives or nothing at all. This is the core lesson of the revised Cohen paper, and a direct rejection of one-size-fits-all explainability.
Protect independent judgement. Elicit the human's view before revealing the machine's wherever time allows, and present explanations as challenges to disagreement rather than justifications for agreement.
Budget friction. Buçinca's results show friction works and people dislike it. Applied everywhere, it wears people down and pushes them back towards blind acceptance. Applied where the model is uncertain, the stakes are high and the human might know better, it earns its cost.
Distrust the feeling of control. Moon, Huang and Xiao's work warns that editing, steering and customising can raise users' sense of power without improving outcomes. Every control affordance should be tested against accuracy on cases where the AI is wrong, not just against satisfaction.
Treat oversight law as a design brief. Article 14 asks that overseers “remain aware” of automation bias. Awareness is not a pop-up warning. It is an interface that makes the machine's fallibility visible at the moment of decision.
The Researchers Who Took Their Own Advice
The revision of the Cohen paper is worth dwelling on. The first version made a vivid, general claim with a memorable name. The second narrowed it, flagged its statistical limits, demoted its most appealing benchmark to a reference point, and concluded that the right answer depends on the task. The authors noticed that their framing was more confident than their evidence, and they recalibrated. The Langer replication story is the same lesson at a fifty-year scale.
That is precisely the behaviour a calibrated-reliance interface tries to produce, and it is hard. Product teams have every commercial reason to keep the fluent, confident explanation, which tests well, demos well and makes users feel smart. The evidence from Bansal in 2021 to Moon in 2026 says that feeling is not the same as being right, and the two can pull in opposite directions.
The explainable AI movement rested on an appealing premise: if people understand why a machine says what it says, they will use it better. The premise is not wrong so much as incomplete. Understanding helps when it gives people something to check. When it gives them only something to believe, it becomes another form of persuasion, and a more effective one than a bare answer, because it looks like reasoning.
The goal, then, is not more explanation or less, but better-aimed explanation: cheap to verify, honest about uncertainty, sequenced so it does not pre-empt human judgement, and most forthcoming when the machine is least sure. Judge it on one question above all: when the model was wrong, did the person notice?
In the RAVEN study, people reading fluent explanations caught roughly one AI error in six. Shown a number instead, they caught more than one in three. Neither is good enough, and the gap between them is statistically fragile. But it points towards where better designs will come from: interfaces built not to win the user's trust, but to earn it one case at a time, and to surrender it gracefully when it has not been earned.
