In June 2022, in an operating room in Fort Worth, Texas, a 44-year-old patient named Erin Ralph went under for what was meant to be a routine sinuplasty. The surgeon, Dr Marc Dean, was using the TruDi Navigation System, a piece of kit originally manufactured by Acclarent, a Johnson & Johnson subsidiary, that in 2021 had been augmented with a machine-learning algorithm designed to map the bony architecture of the sinuses in real time. The promise was straightforward: a digital second pair of eyes, overlaying anatomical landmarks on the surgeon's view so that the delicate corridors between the nose and the brain could be navigated with something closer to mathematical certainty. What happened instead, according to a lawsuit Ralph later filed, was that the system “misled and misdirected” the surgeon. Her carotid artery was injured. She had a stroke on the operating table. Surgeons had to remove part of her skull to manage the swelling. She is still in therapy.
Eleven months later, another patient of Dr Dean's, Donna Fernihough, was undergoing the same procedure with the same device. Mid-operation, her carotid artery “blew”, in the description that appears in the court filings, blood spraying from the wound. She had a stroke that day too.
These were not isolated mishaps. In February 2026, Reuters published an investigation that pulled together the FDA's adverse event database with court records, internal correspondence, and interviews with surgeons, regulators, and patients. Before the TruDi system was given its AI upgrade in late 2021, the FDA had received seven unconfirmed reports of device malfunctions and one injury across the device's lifetime. In the four years after the upgrade, that figure rose to at least 100 unconfirmed malfunctions and adverse events, with at least 10 documented injuries. The investigation widened to take in other AI-integrated devices: Samsung Medison's Sonio Detect, used for prenatal ultrasound; Medtronic's LINQ implantable cardiac monitor with its AccuRhythm AI module. In one case, an AI overlay meant to highlight critical anatomy during a laparoscopic procedure failed to flag a structure in the surgical field; cerebrospinal fluid began leaking from the patient's nose. In another, a surgeon “mistakenly punctured the base of a patient's skull”. By the time the piece went to press, there were 1,357 FDA-authorised AI-enabled medical devices on the US market, more than double the number authorised by the end of 2022, with 182 product recalls already linked to 60 of them. Forty-three per cent of those recalls had occurred within a year of approval.
The investigation made clear that part of the problem was regulatory. Dr Alexander Everhart of Washington University was quoted as saying that the FDA's traditional approach was “not up to the task of ensuring AI-enabled technologies are safe and effective”. The agency's AI review unit, the Division of Imaging, Diagnostics and Software Reliability, had been cut from around 40 scientists to about 25 under the Trump administration's cost-cutting initiative, and the Digital Health Center of Excellence had lost roughly a third of its 30-strong staff. An anonymous former FDA employee put it plainly: “If you don't have the resources, things are more likely to be missed.”
But there is another layer to the Reuters story, one that is harder to legislate around and that has begun, in the months since the piece appeared, to draw the attention of a much wider research community. It concerns not the machine but the human standing next to it. In every one of these cases, including the catastrophic ones, the device was nominally under the supervision of a trained clinician. The AI was an assistant. The surgeon, the radiologist, the obstetrician was meant to be the safeguard.
That is the architecture of clinical AI deployment as it has been understood since the field's first regulatory frameworks were drafted. The algorithm advises; the human verifies; the patient is protected by the redundancy. It is a model so deeply entrenched that it now functions less as a deliberate design choice than as a cultural default, repeated in white papers, manufacturer disclaimers, professional society guidelines, and informed-consent forms. Human-in-the-loop. Clinician-led. AI-augmented. The vocabulary is reassuring in roughly the way the architecture is meant to be: a single human pair of eyes, attached to a single human brain trained over years of residency and fellowship, can be relied upon to catch what the machine gets wrong.
The question the Reuters investigation forced open, and that a growing body of research has been picking at for the last three years, is whether this model can survive its own success. If the clinician's role is to check the AI, and the AI is good enough to make that checking feel mostly redundant, and the clinician has built her expertise alongside the AI from her earliest training, then what exactly is the safeguard checking with, and against what reference?
The Faith Problem
The Guardian, in November 2025, ran a piece that crystallised a mood that had been thickening in American medicine for at least two years. The headline framed it as a “dangerous faith in AI” sweeping the country's hospitals. The reporters had spoken to physicians across multiple specialties who described what one of them called a “creeping deference”, a tendency among colleagues, and sometimes themselves, to nod along with algorithmic recommendations in cases where, five years earlier, the same physician's clinical instincts would have prompted independent scrutiny.
There was nothing especially surprising about the pattern. It has a name in the human-factors literature: automation bias, the tendency of humans operating alongside automated decision-support systems to over-rely on the automation, particularly under cognitive load. The term was coined in the late 1990s in studies of aviation cockpit automation, and the foundational synthesis remains a 2010 paper by Raja Parasuraman and Dietrich Manzey, two cognitive psychologists who argued that automation bias and a related phenomenon, automation complacency, were two facets of the same underlying mechanism: a redistribution of attentional resources away from a task once the operator has come to trust that the machine is handling it. In the cockpit context, the most quoted example is the crew that flies a serviceable aircraft into terrain because the autopilot has not flagged a problem and they have stopped watching the altimeter.
Medicine has been late to this literature, but it has been arriving steadily. A 2012 systematic review by Kate Goddard and colleagues at City University London, published in the Journal of the American Medical Informatics Association, pulled together what was then a small but consistent body of evidence that clinicians using computerised decision-support systems made worse decisions when the system was wrong than they would have made without the system at all. The review identified workload, task complexity, time pressure, and user trust as the main mediators. Training, accountability framing, and design choices like where the recommendation appeared on the screen were among the few mitigations that showed any consistent effect.
Since then, the evidence has piled up. In 2023, a study in Radiology by a German group examined what happened when 27 breast imaging radiologists were given AI prompts that were deliberately incorrect. The radiologists' false-positive recall rates rose by up to 12 per cent, with experienced readers affected almost as much as the less experienced. A separate multi-reader study on cerebral aneurysm detection using time-of-flight MR angiography found that false-positive AI findings drove inexperienced readers to recommend significantly more aggressive follow-up examinations; reading times were shorter with AI present at every level of experience, a marker of the attentional shortcut the Parasuraman framework predicts. A 2023 chest radiography study found that incorrect AI results increased both false-negative and false-positive interpretations relative to the same cases read without AI, and the effect was strongest in less experienced clinicians.
The Guardian's contribution was to describe what this dynamic feels like from inside the practice. Physicians spoke of an erosion they could feel but not quite locate. One quoted clinician said that when the AI's read agreed with their own, they felt confirmed; when it disagreed, they paused; and increasingly often, the pause did not resolve in their favour. It is the kind of subjective account human-factors researchers have learned to take seriously, not because individual testimony is reliable evidence of underlying cognitive change, but because the language of “deference” and “creeping” maps onto exactly the attentional patterns the laboratory studies have measured.
The Polyp That Was Not Found
If the laboratory studies pinned down the in-the-moment dynamics of automation bias, the question of what happens to clinicians over the longer arc of their careers required a different kind of investigation. The most striking attempt came not from radiology but from gastroenterology, published in The Lancet Gastroenterology & Hepatology in 2025. The paper, an observational study from a multicentre Polish trial called ACCEPT (Artificial Intelligence in Colonoscopy for Cancer Prevention), looked at what happened to endoscopists' performance on unassisted colonoscopies after the same endoscopists had been routinely using an AI polyp detection system.
The mechanics of the study were unusually clean. Four endoscopy centres in Poland had introduced AI tools for polyp detection in late 2021. Between September 2021 and March 2022, 1,443 patients underwent non-AI assisted colonoscopies; 795 of those were performed before the AI system was introduced at the centres, and 648 afterwards, with the AI deliberately switched off for those cases. The crucial comparison was not between AI-assisted and unassisted colonoscopy, which prior literature had explored extensively, but between unassisted colonoscopy by clinicians who had never used AI and unassisted colonoscopy by clinicians who had been using AI as a matter of routine.
The adenoma detection rate, the percentage of screening colonoscopies that identify at least one precancerous polyp and the most validated quality metric in colorectal cancer prevention, fell from 28.4 per cent before AI exposure to 22.4 per cent afterwards. An absolute drop of six percentage points may not sound seismic until you start translating it into lives. Adenoma detection rate is one of the few clinical metrics in any specialty that has been directly linked, in large cohort studies, to long-term cancer mortality: a one percentage point increase in ADR is associated with a roughly three per cent decrease in interval colorectal cancer incidence. A six-point fall is not a rounding error.
The authors were careful with their causal claims. The study was observational; the periods being compared were not identical; the endoscopists knew which cases were being read without AI. But the inference the authors did draw was that continuous exposure to AI might “reduce the skills of the endoscopist”, a phrasing chosen because it was the most parsimonious explanation the data would support.
What the ACCEPT paper offered was something the laboratory studies could not: a population-scale glimpse of what happens to clinical performance when an entire department's daily practice is reshaped around an AI assistant, and then the AI is taken away. The finding was not that clinicians became unable to find polyps. It was that they found fewer, by a margin that, if replicated, would erase years of quality-improvement gains in cancer screening.
The Lancet study is currently a single paper in a single specialty, and its limitations are real. But it landed in a research community that had been waiting for exactly this kind of empirical anchor. A scoping review published in ESMO Real World Data and Digital Oncology in 2026 concluded that evidence of clinical deskilling, although still scarce, was already consistent across specialties: skills faded not because they were unnecessary but because they were no longer practised. The authors framed it, drawing on a much older literature on motor and perceptual skill, as a use-it-or-lose-it problem rather than a fundamentally novel phenomenon. What was new, they suggested, was the speed at which AI was being woven into routine practice, and the question of whether the institutions that train clinicians would respond fast enough to preserve the underlying competencies.
The Pipeline Question
This is where the question stops being one about working clinicians and becomes one about the next generation. A radiologist who finished her training in 2010, used unassisted reads for a decade, and then started working with AI assistance in 2020 carries inside her the reference signal against which the AI's behaviour can be assessed. She knows what an unassisted read feels like; she can notice, in herself, the moment when the AI's overlay nudged her toward a decision she would otherwise have questioned. The radiologist who finishes her training in 2028, by contrast, will have built her pattern recognition alongside the AI from her first residency rotation. She will have no reference signal of her own. The question of what unassisted reading feels like will not be answerable from the inside, because she has never done it.
This is the structural concern Fortune surfaced, in a different register, in May 2026. The piece was framed as a kind of victory lap for the radiology profession, ten years after Geoffrey Hinton's much-quoted 2016 prediction that the specialty was doomed. Hinton, the Turing Award and Nobel laureate whom the press routinely calls the “Godfather of AI”, had told an audience at the Machine Learning and the Market for Intelligence conference in Toronto that “people should stop training radiologists now”, because it was “completely obvious” that within five years, ten at most, deep learning would do a better job than humans. His most-quoted line was the image of the coyote that had already run off the cliff but had not yet looked down.
A decade later, the coyote is still in the air. Fortune, drawing on Medscape's 2026 physician compensation report, put the average US radiologist salary at $571,000, up 9 per cent on the previous year. The number of active radiologists in the United States grew by roughly 10 per cent across the decade. Case loads, according to data from the Journal of the American College of Radiology, climbed 25 per cent between 2018 and early 2025. As of March 2026, there were around 4,333 active job listings for radiologists, with an average time-to-fill of 130 days. Hinton, in a New York Times interview in 2025, retracted the timing if not the direction: he had been speaking only about image analysis, he said, and human radiologists would work with AI to be more efficient and more accurate, not to be replaced.
The Fortune piece treated this as straightforward vindication for the specialty. It is not quite that, or not only that. What the headline numbers obscure is that the radiologist of 2026 is not doing the same job that the radiologist of 2016 was doing. The case load is up by a quarter, and the time available per scan has shrunk correspondingly. AI is part of how that case load is being absorbed; not by replacing the radiologist, but by changing the nature of what reading a scan means. Christoph Herpfer, an economist at the University of Virginia's Darden School of Business quoted in the Fortune piece, made the point that AI in radiology had behaved less like a substitute than a complement, expanding the volume of imaging the system could process rather than shrinking the workforce that processed it. Jeff Chang, a former emergency radiologist who co-founded Rad AI, was quoted to similar effect: the productivity gains had absorbed the demand.
That is true. It is also a description of an entire profession being restructured around a tool, with the tool inside the loop of every trainee from their first day on a workstation. The question the Fortune piece does not ask, because it is not within the brief of a workforce-optimism story, is what kind of expertise that workforce will carry in twenty years. If the value of the human radiologist in 2046 is partly that she can catch what the AI gets wrong, the value depends on the human reading skill that was built up across her career. If that skill is now built alongside the AI from residency onwards, the loop is closed in a particular way: the radiologist's expertise is shaped from its earliest stages by the tools it is meant to be checking.
Educational researchers have started to map this concern empirically. A 2024 paper in Insights into Imaging on AI-supported training for radiology residents, which used the disruptions of the COVID-19 pandemic as a natural experiment, found that AI increased residents' immediate accuracy on chest X-ray interpretation but did not produce enduring gains once the AI was removed. The residents who had learned with the tool performed worse when the tool was taken away than those who had learned without it. A multi-institutional survey of US radiology residents published in 2023 found that 83 per cent thought AI education should be part of residency, but only a minority of programmes had an established curriculum that took the deskilling concern seriously. The gap between the speed of clinical deployment and the speed of pedagogical adaptation is now wide and widening.
The ACGME, the body that accredits US graduate medical education, has begun, slowly, to ask radiology programmes to document how they preserve unassisted reading practice. The European Society of Radiology issued guidance in 2025 recommending a structured minimum of supervised, AI-free reads during the early years of training. None of these interventions is yet underpinned by the kind of evidence that would tell programme directors how many unassisted hours per week or per month constitute an adequate dose. The honest answer is that no one knows, because the cohort of clinicians who have trained entirely alongside AI is still small enough that the longitudinal data has not arrived.
Mechanism
It is worth pausing, before reaching for mitigations, to look at the cognitive machinery underneath all of this. The 2010 Parasuraman and Manzey paper proposed that automation bias and automation complacency could be unified under what they called an attentional framework. When an automated system performs a task reliably enough that the operator comes to trust it, the operator's attention is reallocated; the cognitive resources that would have gone to monitoring the task are spent elsewhere. The shift is not deliberate, and it is not, in the usual sense, irrational; it is a sensible economisation of finite attention. The trouble is that the reallocation is invisible to the operator, and it persists even when the automation, in a given instance, is wrong.
Apply that to clinical practice and the picture sharpens. A radiologist who has read 10,000 AI-assisted scans has had her attentional pattern shaped, over thousands of repetitions, around the assumption that the AI will catch what she might miss. Each scan is not a fresh act of unassisted vigilance; it is a collaboration in which her attentional resources have learned to redistribute themselves around the algorithm's apparent strengths and weaknesses. This is not a moral failing. It is the same process by which an experienced driver stops actively scanning the dashboard once she has internalised the rhythms of the car. It is what skilled human-machine teaming looks like from the inside.
The problem is that when the machine is removed, or when the machine is wrong in a way it does not flag, the redistributed attention does not snap back into place automatically. The 2025 Lancet study, in this reading, is the empirical correlate of the Parasuraman attentional model: endoscopists who had been working with AI had restructured their attentional patterns around it, and their unassisted ADR fell because the redistribution did not reverse the moment the screen went dark.
The same framework predicts something less often discussed: the deskilling effect should be most severe for the skills least often consciously practised. A surgical resident who deliberately performs a portion of an operation unassisted, against the resistance of the workflow, retains the muscle memory and the perceptual chunking the operation requires. A radiologist who reads the AI overlay first and then “checks” the image is performing the unassisted skill not at all; she is performing a different skill, that of reviewing an AI annotation, which is a real skill but not the same one. Over a career, the second skill grows and the first one shrinks. This is what the ESMO scoping review meant by “use-it-or-lose-it”: the deskilling is not a failure of clinician dedication but a structural consequence of where the workflow puts the human attention.
There is a deeper version of this concern that has been pressed most clearly by James Reason, the British human-error scholar whose Swiss-cheese model has been the dominant metaphor in patient safety for a generation. The model imagines layers of defence against error, each with holes; an accident occurs when the holes line up. In a clinical AI deployment, the AI is one layer and the clinician is another. The safeguard model assumes the holes in the two layers are independent, that the things the AI gets wrong are not the same things the clinician gets wrong. If automation bias reshapes the clinician so that her holes start to align with the AI's, the two layers collapse into one. The defence-in-depth is not depth at all. It is one layer, twice drawn.
What Mitigations Look Like
The interventions the literature has proposed cluster into three rough categories, none yet supported by the kind of trial evidence that would let a hospital trust it.
The first is preserved unassisted practice. The Polish endoscopy data, combined with the ESMO review, has driven the most concrete version of this proposal: that clinicians using AI tools should be required to perform a structured minimum number of unassisted reads or procedures, distributed across their working time, as a maintenance activity in the same way that pilots maintain hand-flying hours alongside autopilot use. The Royal College of Radiologists in the UK floated a proposal along these lines in late 2025, suggesting that one in ten screening mammograms be read without AI as a matter of departmental policy. The American College of Radiology has held back from a specific number but has endorsed the principle. The objection from hospitals has been straightforward: every unassisted read is a read that takes longer, and the productivity case for AI deployment was built on the assumption the time was being recovered.
The second is simulator hours. In aviation, the response to autopilot-induced skill atrophy was not to take the autopilot out of the cockpit but to require pilots to spend a defined number of hours per year in simulators practising the hand-flying skills the autopilot displaced. The clinical analogue would be high-fidelity simulator practice, with real anonymised cases, that exercises the unassisted diagnostic muscles. There is now a small industry of radiology and surgical simulator vendors selling exactly this proposition, and a smaller body of evidence that it can preserve perceptual skill if the dose is high enough. What is missing is a regulatory regime that mandates the dose.
The third, and the most interesting, is structured disagreement. The Stanford radiology group, in 2025, published work on AI monitoring methods that explicitly flag cases in which the AI's confidence has dropped or in which the case lies outside the distribution of training data; their argument is that the clinician should not be asked to second-guess the AI on every case, but should be alerted when the AI itself is unsure. A related but distinct proposal is to engineer workflows so that the clinician records her independent read before seeing the AI's output, with the system then revealing the AI read and forcing an explicit reconciliation when the two disagree. This blind-read-first protocol has been tested in some breast imaging settings with promising early results, but it has the same productivity cost as the first proposal: it slows everything down.
What these proposals share is an acknowledgment that the safeguard model as currently conceived is not self-sustaining. If the value of the human safeguard depends on the human carrying expertise that the AI does not have, then expertise has to be actively maintained as a separate variable in the system, not assumed to persist as a by-product of clinical work. The mitigations are attempts to insert a different kind of redundancy into the workflow: not a second pair of eyes but a second mode of attention, exercised on a schedule independent of the AI's daily presence.
The Coherence Problem
There is a more uncomfortable possibility, which the mitigations sidestep without quite addressing, and which the Reuters investigation, the Guardian piece, the Fortune story, and the Lancet paper all point at obliquely. It is the possibility that the safeguard model is not coherent in the form in which it has been described.
The model says: AI assists, clinician verifies, patient is protected by redundancy. The model works if and only if the clinician's verification is causally independent of the AI's recommendation, which is what makes the redundancy meaningful. If the clinician's expertise has been shaped, over the years of her training and practice, by the AI she is supposed to be checking, the independence assumption fails. The clinician is not a second, independent observer; she is a co-product of the same system. The patient is being protected by a single integrated decision process that has been presented, in regulatory documents and informed-consent forms, as if it were two.
This is the question the editorial accompanying the Polish study in The Lancet Gastroenterology & Hepatology was reaching toward when it asked whether AI-assisted colonoscopy was producing better colonoscopy or simply a different practice altogether, in which the AI's outputs and the endoscopist's behaviour were no longer separable. The same question can be asked of every other specialty where deployment is far enough along to begin generating longitudinal data. It is the question Erin Ralph's lawyers were implicitly raising in the TruDi litigation when they argued the navigation system “misled and misdirected” the surgeon: at what point does the system stop being a tool that the surgeon uses and start being part of the cognitive process by which the surgeon decides?
There is no clean answer, because the boundary is genuinely blurry. Every diagnostic tool, from the stethoscope onwards, has shaped the clinical reasoning of the clinicians who use it. The radiologist who came of age with digital radiography reasons differently from the one who came of age with film, and the difference is not nothing. The difference between an AI-assisted clinician and her unassisted predecessor is a difference of degree, not of kind. But the degree matters. A stethoscope does not learn from millions of prior auscultations and update its outputs in real time; an AI system does, and the rate at which the AI updates, and the opacity of the updates, sets a pace of integration that prior tools did not.
The clean answer would be to say we should not deploy AI tools where the integration risks are this deep, and that is a position some researchers hold, in the limit. It is not, realistically, where the field is going. The economic and clinical pressures behind AI deployment are large enough, and the gains in image-by-image and case-by-case accuracy real enough, that the deployment will continue. The question is what the safeguard model means once we have admitted that the human in the loop is being shaped, day by day, by the loop she is part of.
Sitting With It
It would be more satisfying to end with a recommendation. The literature contains plenty. Preserve unassisted practice. Mandate simulator hours. Engineer structured disagreement. Invest in AI literacy curricula. Build monitoring tools that flag the AI's uncertainty. Track adenoma detection rates and mammography false-positive rates and surgical adverse event rates as drift indicators, with department-level interventions triggered when the numbers move in the wrong direction. Each of these is being tried, somewhere, and each is plausible.
What none of them quite does is answer the underlying question. If the value of human clinical expertise lies partly in its capacity to serve as a check on AI error, and that expertise is itself shaped from its earliest stages by the tools it is supposed to be checking, the safeguard model is not just under-resourced or poorly implemented. It is, in some structural sense, in tension with itself. The mitigations are attempts to hold the tension open, to preserve enough independence between the human and the machine that the redundancy retains meaning. Whether they will be enough, at the dose at which they are likely to be implemented, against the gradient of productivity pressure pulling the workflow in the other direction, is not knowable now. It is barely knowable in principle.
In Fort Worth, Erin Ralph is still in therapy. In Poland, the endoscopists who took part in the ACCEPT trial are back at work, with AI mostly switched on, the lower unassisted ADR a number in a paper rather than a feature of their daily practice. The radiologists Fortune profiled in May are earning their $571,000 and reading more scans per shift than their predecessors did a decade ago. Geoffrey Hinton has retracted his prediction without quite retracting its premise. The 1,357 AI-authorised medical devices on the US market are joined every month by more. The trainees who will inherit this system are being shaped by it now, in their first year of residency, in ways none of them can step outside to see.
The honest version of the question is not what we should do about this. It is whether we have given ourselves the conceptual tools to know what we are doing. The safeguard model, as it stands, presumes a kind of independence between the human and the machine that the evidence is steadily eroding. What we put in its place will determine, more than any single mitigation, what patient safety means in the decade ahead.
