What Should Our Board Be Asking About AI Risk?

Key Takeaways

·Board-level fiduciary duty extends to AI risk. Under the Caremark standard, directors can face personal liability for failing to implement adequate oversight of known organizational risks, and AI is now a known risk in healthcare.
·The foundational board question is inventory: do we know what AI tools are deployed, by whom, and for what purpose? Without an inventory, oversight is structurally impossible.
·Only 61% of hospitals validate AI tools on their own patient population before deployment, and less than 50% test for bias. Boards should ask who reviews validation results and whether those reviewers are qualified to evaluate what the results mean.
·Roughly 10-15% of large health system boards have formal AI oversight structures. The gap between AI deployment rates and board-level governance is widening, and accreditors are beginning to notice.
·The Joint Commission and CHAI RUAIH framework states explicitly that the fiduciary board should be regularly updated on AI use and its outcomes. That expectation is becoming a practical standard.

The short answer

Boards should be asking five categories of questions: Do we know what AI tools we have? Are those tools validated on our patient population? Is there a named individual accountable for AI governance with authority to halt deployment? Would we detect a problem if one developed? And what is our liability exposure if an AI-related patient safety event occurs and we have no governance documentation? Organizations that can answer all five with specifics are in a defensible position. Most can't answer more than two or three, and that gap is exactly what regulators, accreditors, and plaintiffs' experts are starting to probe.

Why Boards Are Now in the Accountability Chain

The question of board-level AI oversight isn't about being forward-thinking. It's about existing fiduciary duty applied to a category of organizational risk that has grown faster than governance structures have adapted. AI tools are making clinical recommendations, flagging patient deterioration, routing care pathways, and generating documentation in hospitals across the country. In most cases, those tools arrived through IT procurement processes that never touched the board's agenda.

The Caremark standard in Delaware corporate law holds that directors can face personal liability for failing to implement adequate board-level oversight of known organizational risks. Courts extended this reasoning to cybersecurity risk after several high-profile breaches. Legal scholars and risk managers are now applying the same framework to AI. The accountability chain runs from vendor to health system executive leadership to board via fiduciary duty, and that chain is getting tested in litigation, enforcement actions, and accreditation surveys simultaneously.

The Joint Commission and Coalition for Health AI (CHAI) RUAIH framework is explicit: "the fiduciary board of the healthcare organization should be regularly updated on AI use and its outcomes." That language isn't aspirational. It reflects the direction accreditation expectations are moving, and health systems that can't show board-level visibility into their AI governance will be answering for that gap in surveys over the next several years.

The Gap Between Deployment and Oversight

·Only 10-15% of large health system boards have formal structures specifically for AI oversight, while AI deployment rates continue to accelerate across clinical and administrative functions
·Only 61% of hospitals validate AI tools against their own patient population before deployment — the remaining 39% are relying exclusively on vendor validation data
·Less than 50% of hospitals test deployed AI tools for bias, meaning racial and demographic disparities in AI performance may be going undetected
·AI-related malpractice claims increased 14% between 2022 and 2024, with automation bias — clinician failure to question AI output — as a leading contributing factor

Five Areas Where Board Questions Are Overdue

The AHA, NACD, Deloitte, and the Joint Commission/CHAI framework converge on five areas where board questions should be directed. These aren't audit committee questions or IT committee questions in the sense of technical detail. They're governance questions about whether the right structures exist and whether accountability is clear. Boards don't need to understand model architecture to ask them.

Inventory and awareness

The foundational question is whether the organization knows what AI tools are deployed, by whom, and for what purpose. AHA guidance, Deloitte, and the NACD all identify this as the first question boards should ask. Without an inventory, the board cannot exercise oversight over what it doesn't know exists. Boards should ask not just whether an inventory exists, but who maintains it, how often it's updated, and whether it includes tools acquired at the department level without central IT involvement. Shadow AI, meaning tools staff are using without formal procurement, is a category many inventories don't capture. The answer to this question should come from a maintained register, not from someone's recollection.

Validation and performance

FDA clearance is point-in-time and based on the vendor's testing population. Local validation, meaning testing AI performance against your own patient population with your own data, is essential because performance can differ dramatically across settings. Boards should ask how tools are validated before deployment, who reviews validation results, and whether the reviewers have the clinical and technical qualifications to evaluate what the results mean. The follow-on question is what happens when a tool's post-deployment performance diverges from pre-deployment expectations. Only 61% of hospitals currently validate AI tools against their own patient data. Most organizations are relying on vendor claims and FDA clearance alone, which leaves a material evidence gap in the event of patient harm.

Governance structure and accountability

The NACD recommends boards ensure AI risk oversight is clearly assigned, either to the full board or to a specific committee such as audit or quality, with a defined reporting cadence. Boards should ask who the named individual accountable for AI governance is and whether that person has actual authority to halt deployment if a safety concern arises. Committees can advise and recommend, but individual accountability for AI-influenced decisions requires a named person at the end of the chain. If the answer to 'who explains what happened if this tool caused harm?' is 'the committee,' that's a governance gap, not a governance structure.

Risk monitoring and incident detection

The Optum racial bias case is the instructive example: an AI tool that systematically underestimated the health needs of Black patients was in wide deployment for years before the problem was identified and published in a peer-reviewed journal. Boards should ask whether the organization would detect a comparable problem if one developed, and specifically what monitoring is in place for model drift, bias, and safety events. The absence of reported AI-related incidents is not evidence that AI is performing well. It may mean monitoring processes aren't finding what's there.

Liability and insurance exposure

Malpractice and cyber insurers are introducing AI-specific policy riders and coverage conditions. Some are beginning to require demonstration of 'reasonable governance' as a coverage condition, and what counts as reasonable is converging toward alignment with CHAI and JC guidance. Boards should ask whether current insurance policies cover AI-related incidents, whether those policies have been reviewed by counsel specifically for AI liability language, and what the organization's exposure looks like if a patient safety event occurs and the board can't produce documentation that AI governance was being exercised. Most current policies predate widespread clinical AI deployment and don't address it clearly.

The Overarching Question

Every other question on this list builds toward one: can the organization demonstrate that it is governing AI in a way that is reasonable, defensible, and visible at the board level?

Organizations that can show alignment with the RUAIH framework will be positioned to demonstrate reasonable oversight in front of regulators, accreditors, insurers, and plaintiffs' attorneys. Those that cannot face increasing exposure from all four directions simultaneously. The Joint Commission's emerging AI-related survey questions, the FDA's post-market surveillance expectations for software as a medical device, state consumer protection frameworks used in actions like the Texas AG's settlement with Pieces Technologies, and malpractice plaintiff experts are all developing variations of the same core argument: you deployed AI that contributed to harm and you either didn't know or didn't check.

That argument is harder to make against an organization that has board-level AI governance documentation, a named accountable executive, a maintained tool inventory, a validation process, and a monitoring program. That documentation isn't just a compliance exercise. It's the primary defense when something goes wrong despite the organization's best efforts.

What Board-Level AI Reporting Should Include

Boards that are exercising meaningful AI oversight typically receive regular reporting covering these elements. These aren't technical briefings. They're accountability updates that connect AI deployment to board-visible risk.

·Current AI tool inventory with risk classification by tool — high-impact clinical decisions carry different board visibility requirements than administrative automation
·Validation status for each high-impact tool, including dates, methodology, and summary findings from local validation against the organization's own patient population
·Summary of AI-related incident reports, near-misses, and bias monitoring results from the prior period
·Named accountable executive with a brief update on governance structure maturity and any significant changes since the prior reporting period
·Insurance coverage review status for AI-specific liability, including any open questions about whether current policies address AI-related incidents

Committee Delegation vs. Full Board Oversight

The NACD guidance gives boards two structural options: keep AI risk oversight at the full board level, or formally delegate it to an existing committee, most often audit or quality. Both approaches work if the delegation is explicit and reporting back to the full board is regular and substantive.

What doesn't work is informal delegation, where AI risk questions are generally assumed to belong to IT governance or the quality committee but no one has formally defined scope, authority, or reporting requirements. That's not governance. It's hoping someone else is watching.

Health systems building board-level AI oversight from scratch often find it useful to start with a gap assessment: map what's currently deployed against what governance processes exist, identify where accountability is unclear, and present that gap analysis to the board before proposing a governance structure. It's harder for governance investment to be deprioritized when the board has seen the gap in concrete terms.

Full board oversight

Appropriate when AI deployment is extensive, when clinical AI tools carry high patient safety stakes, or when the board has determined AI risk is material enough to warrant the same treatment as financial or cybersecurity risk. Requires dedicated agenda time and board education investment, but produces the clearest accountability signal.

Delegated committee oversight

Appropriate when an existing committee has the right membership and capacity to take on AI risk, and when formal charter language, reporting requirements, and escalation triggers are clearly defined. The full board still needs regular summary reporting. The key is that delegation is documented, not assumed.

Frequently Asked Questions

Common questions healthcare board members and executives ask about AI risk oversight.

Does a board have a legal obligation to oversee AI risk, or is this just best practice?

The Caremark standard in Delaware corporate law holds that directors can face personal liability for failing to implement adequate oversight of known organizational risks. Courts applied this to cybersecurity, and legal scholars and risk managers are extending the same reasoning to AI. Whether or not a court eventually imposes director liability in the AI context, the insurance and regulatory exposure from the absence of governance documentation is real now. The Joint Commission and CHAI guidance explicitly names board-level visibility as a component of reasonable AI oversight. Organizations that treat this as merely aspirational are making a bet that enforcement and litigation move slowly. That bet is becoming harder to justify.

What does the Joint Commission expect from boards regarding AI?

The RUAIH framework states that the fiduciary board should be regularly updated on AI use and its outcomes. Surveyors are currently using that guidance as a reference in Leadership and Quality Improvement categories. Joint Commission surveyors are developing AI-related questions, and organizations that can show board-level governance documentation, not just clinical or IT-level policies, are better positioned for those surveys. Formal binding standards are expected to follow the current guidance as expectations harden.

How often should boards receive AI governance reporting?

The emerging practice is quarterly reporting to whichever body has formal AI oversight responsibility, with an annual comprehensive review. Quarterly reports should cover material changes to the tool inventory, any AI-related incident reports or near-misses, and validation status for high-impact clinical tools. The annual review should address insurance coverage adequacy, governance structure maturity, and alignment with current JC and CHAI guidance. More frequent reporting may be appropriate during periods of rapid AI deployment or following a safety event.

What's the difference between AI governance at the board level and AI governance at the operational level?

Board-level governance addresses whether oversight structures exist, whether accountability is clear, and whether material risks are visible to directors. Operational governance addresses the specific policies, workflows, and monitoring processes that govern how individual AI tools are evaluated, deployed, and monitored. Both are necessary. Organizations that have detailed operational AI policies but no board visibility into the program are missing the accountability layer that would matter in litigation or an accreditation survey. Organizations that have board-level declarations of AI governance without operational substance are performing theater.

What should a board do if it discovers the organization has been deploying AI without any governance structure?

Start with inventory. Direct executive leadership to produce a current accounting of all AI tools in clinical or operational use, with a risk classification for each. That inventory becomes the basis for a gap assessment, which becomes the basis for a governance structure proposal. The gap assessment should include validation status, accountability assignments, and an insurance coverage review. Boards don't need to design the governance system themselves. They need to be explicit about requiring one and to set a timeline for seeing it presented. The organizations most at risk are not those building governance from scratch, but those not building it at all.

How does AI governance connect to existing quality and patient safety committee work?

The cleanest integration point is the quality committee or patient safety committee, because AI governance in the clinical context is fundamentally a question of whether tools being used in patient care meet the organization's safety standards. AI validation, bias monitoring, and incident detection are extensions of existing patient safety processes, not separate programs. The distinction is that AI risk requires additional technical expertise that quality committees may not have, and the vendor contract dimension requires legal and risk management involvement that quality committees don't typically handle. The best structures connect AI governance to quality while adding the technical and legal expertise quality committees currently lack.

Sources

Joint Commission and Coalition for Health AI (CHAI). Responsible Use of Artificial Intelligence in Healthcare (RUAIH) Framework, Version 2. 2024.
National Association of Corporate Directors (NACD). AI Oversight for Boards: Key Questions and Governance Considerations. 2024.
American Hospital Association. Trustworthy AI in Health Care: A Framework for AI Governance. 2024.
Deloitte. AI Governance in Healthcare: Board and Executive Accountability. 2024.
In re Caremark International Inc. Derivative Litigation, 698 A.2d 959 (Del. Ch. 1996).
Obermeyer, Z. et al. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 447-453. 2019.
State of Texas v. Pieces Technologies, Inc. Office of the Texas Attorney General. 2024.
MLT AI Decision Impact Classification Framework v1. Mosaic Life Tech. 2025.

What Should Our Board of Directors Be Asking About AI Risk?