I don't trust that.
Everybody has said it about something. The navigation app that routed you onto a road that was closed. The tool that was fluently, confidently wrong about the one thing you happened to know. You didn't file a complaint. You just stopped relying on it.
That's the ordinary way people handle anything new. We check it before we lean on it, we lean on it more as it keeps being right, and one bad miss can undo it all.
We had a saying when I ran a health information exchange across 21 hospitals: "data moves at the speed of trust." Connecting the systems to the HIE took real work. Getting a physician to rely on a record that came from another hospital, and to believe it in the moment a patient was in front of them, took even more. Whether the exchange got used came down to trust more than technology.
Healthcare AI is running into the same wall, and it's worth a deeper look, because the stakes are even higher now and AI tools are arriving fast.
In the AMA's own tracking, physician use of AI in practice went from 38 percent in 2023 to 66 percent in 2024 to 81 percent in the survey it fielded last January. Confidence hasn't kept pace. Just over a third of physicians say their enthusiasm outweighs their concern, and 40 percent remain equally excited and worried, a number that hasn't moved across three waves of the survey. Patients are even more guarded. When Pew asked in late 2022, six in ten Americans said they would be uncomfortable if their own provider relied on AI to diagnose them and recommend treatment, and three-quarters said their bigger worry was that healthcare would adopt AI too fast, before the risks were understood.
None of these concerns are new. In 2021, an independent team validated a sepsis prediction model that was already running in hundreds of hospitals. It missed 67 percent of sepsis cases, and its real-world accuracy came in well below what the vendor had reported. No one had published an independent validation before that, with the model already in widespread use.
Two years earlier, researchers took apart a population-health algorithm used to decide which patients received extra care. It predicted cost as a stand-in for need, and because the system had historically spent less on Black patients at the same level of illness, it steered less help to them. Fixing the flaw would have raised the share of Black patients flagged for extra support from under 18 percent to over 46 percent.
Nobody in either story set out to do harm. The tools were put into practice without independent, local validation and without a clear line of accountability. And recovery is harder than it would be from a person making the same mistake. In forecasting experiments, people lost confidence in an algorithm faster than in a human after both got it wrong, and kept choosing the human even after seeing the algorithm outperform it. A tool doesn't get the benefit of the doubt a colleague gets. CHAI's own guide lists algorithm aversion among the end-user tendencies a health system should evaluate. When that's the track record, distrust is normally deemed to be good judgment.
So what changes it? When you ask physicians what would raise their confidence in AI, they don't describe a more capable model. In the AMA's 2026 survey, the two things they rated most important were validation of safety and effectiveness by a trusted entity, with monitoring over time (88 percent), and assurance that patient data is protected (86 percent). Close behind came a designated channel to flag the tool when it's wrong, coverage under standard malpractice insurance, and clean integration into the workflow. Asked which regulatory action would most increase their trust, they put clear liability frameworks first, ahead of everything else, with post-market surveillance second.
Look closely at that list. Validation. Monitoring. Data protection. A feedback channel. Liability. Integration. None of those describes the model itself; every one of them describes the system built around it, which is what governance means.
This is what gets lost in the noise about model benchmarks and infrastructure. Transparency helps, but an explanation screen doesn't earn a clinician's trust on its own, and a confident-looking score can win trust it hasn't earned. Anand Chowdhury, who directs AI informatics at Duke, put it well on a CHAI panel last month: Google Maps has not built trust by giving us screenshots from an atlas. What earns durable trust is slower work: validation on your own patients, not the vendor's, which at minimum means running the tool against your own records before it touches a workflow and checking how it performs across the populations you actually serve; monitoring that catches drift before it becomes a safety event; a model that can say it doesn't know rather than guess; a human who stays responsible for the decision; and an honest answer to the question physicians just put at the top of their list, which is who is liable when the AI makes a mistake.
For a note a clinician signs, that answer is settled: they signed it. The harder case is the one the sepsis model and the population-health algorithm share, where nothing gets signed at all. A score that never fires leaves no artifact and no moment of approval, and the patient who wasn't flagged never lands in anyone's queue. So the question is narrower than it sounds. Who reviews that model's performance, by name, how often, and who covers when that person is out? Most organizations can't answer that. Writing it down is part of the work.
I made a version of this argument last year, after running the same federal policy question through several leading AI models and getting answers that ranged from exactly right to confidently wrong, sometimes from the same model on a second try. A pilot trusts the aircraft because of the ground crew: the people who designed it, ran the simulations, and did the pre-flight checks long before anyone boarded. Health IT leaders are the ground crew for clinical AI, and the validation, the monitoring, and the liability answer are the pre-flight checks.
The governance scaffolding already has a shape. CHAI has published a Blueprint for Trustworthy AI, certifies independent assurance providers, and runs a public registry of model "nutrition labels" (model cards) so a health system can see how a tool was built and tested before relying on it. It opened a cybersecurity working group this month alongside that governance work. NIST's AI Risk Management Framework turns the same work into four steps: decide who's accountable, map the risks, measure them, and keep managing them. The Joint Commission and CHAI issued initial guidance in 2025 on the responsible use of AI in hospitals. All of that is voluntary. The one piece with enforcement behind it, ONC's rule requiring certified decision-support tools to disclose how they were built, is now proposed for removal in the deregulatory rule ONC published in December. Voluntary or not, it's the machinery that lets a clinician, a patient, and a board believe a tool does what it claims, and if the federal floor drops out, that work moves to the health system rather than going away.
In most conversations, governance is characterized as the thing that slows AI down, the compliance drag you handle after the technology works. But that's backwards; governance is not the tax you pay once the model is good, it's how the model earns the right to be used at all.
CHAI put numbers on that last month. Fifteen health systems pooled their AI intake processes and found reviews taking two weeks to two months, one of them with more than 200 tools in the queue. The time went less to analysis than to chasing vendors for the basics: how the tool was validated, what the vendor monitors after go-live, how it performed across patient subgroups, and what the plan is when something goes wrong. That's the vendor's half of the job, and the health system's half, validating on its own patients and deciding who watches the tool, can't start without it. On the same call, one vendor's security lead said publishing those answers up front had fast-tracked her reviews and won deals. Adoption moves at the speed the trust evidence arrives.
Data still moves at the speed of trust, and so does AI. More than twenty-five years after we were teaching hospital staff to use a mouse, the technology has changed beyond recognition, and whether people trust what they've been handed still decides whether it gets used. The organizations that scale AI in healthcare will be the ones whose clinicians trust it, because the ground crew did the work to make it trustworthy.
For most of you the AI is already in front of your clinicians: the ambient scribe, the drafted portal replies, the risk score inside the EHR, some of it switched on by a vendor update rather than a governance decision. So the question applies to what's running now, not only to what's next.
What would it take for your clinicians to trust the AI running in your organization?
Sources: AMA 2026 Physician Survey on Augmented Intelligence (fielded January 15 to February 2, 2026, n=1,692; with the 2023 and 2024 waves); Pew Research Center, "60% of Americans Would Be Uncomfortable With Provider Relying on AI in Their Own Health Care" (fielded December 2022, published February 2023); Wong et al., JAMA Internal Medicine (2021), external validation of the Epic Sepsis Model; Obermeyer et al., Science (2019); Dietvorst, Simmons and Massey, "Algorithm Aversion: People Erroneously Avoid Algorithms after Seeing Them Err," Journal of Experimental Psychology: General 144(1), 2015; Coalition for Health AI (Blueprint for Trustworthy AI, Assurance Resource Provider certification, Applied Model Card registry, Health AI Cybersecurity Work Group, August 2026); NIST AI RMF 1.0; Joint Commission and CHAI, Responsible Use of AI in Healthcare guidance (September 2025); ONC HTI-1 final rule (2024) and HTI-5 proposed rule (Federal Register, December 29, 2025); CHAI webinars: "Panel on Generative and Agentic AI Governance in Healthcare" (July 28, 2026; Anand Chowdhury, Duke) and "Evaluating Clinical AI" fireside chat with Nabla (July 30, 2026; Brenton Hill, CHAI; Brittany Harrell, Nabla), including CHAI's AI intake framework findings from 15 health systems.
AI disclosure: I used AI for research, fact-checking, and drafting this piece. Every statistic was verified against its primary source, and I edited and approved the final text. That's the human-in-the-loop standard this article argues for, and it seemed fair to hold myself to it.
