What to Require from AI Vendors Before Deployment in Your Health System

Key Takeaways

·ONC's proposed HTI-5 rule would remove federal model card requirements, making vendor contracting the primary accountability mechanism for health systems evaluating AI tools.
·Require vendors to supply validation data including demographic subgroup performance, not just headline accuracy metrics from ideal test conditions.
·Contracts must include transparency obligations, model change notification, monitoring data access, rollback provisions, and the right to suspend without financial penalty.
·The PCCP framework allows vendors to update FDA-cleared algorithms without notifying deploying organizations. Contracts need explicit notification requirements to close this gap.
·Standard BAA language doesn't address whether patient data can be used for vendor model improvement. That needs to be addressed explicitly before you sign.
·Vendors who refuse to provide demographic subgroup data or resist audit rights provisions are telling you something important about how they'd respond to a post-deployment problem.

The short answer

Before any AI tool deploys in your health system, require vendors to supply detailed validation data including demographic subgroup performance, not marketing metrics from best-case conditions. Write contracts that include transparency obligations, model change notification, monitoring data access, rollback provisions, and the right to suspend without penalty if performance deviates materially from specifications. Update your Business Associate Agreement to address AI-specific data use. Name someone responsible for receiving vendor notifications. And treat vendor refusal to negotiate on audit rights or liability terms as a governance signal, not a minor commercial disagreement.

Why Vendor Contracting Has Become the Primary Governance Mechanism

For most of the past decade, health systems could rely on a combination of FDA clearance, federal regulatory standards, and vendor-supplied documentation as a baseline for AI tool governance. That baseline is eroding. ONC's proposed HTI-5 rule would remove the federal model card requirements that previously obligated vendors to disclose AI tool performance, limitations, and training data characteristics. If that proposal is finalized, the accountability gap falls to health systems to fill through contracts.

This isn't a hypothetical. The combination of reduced federal disclosure requirements and the PCCP framework, which allows vendors to update FDA-cleared algorithms without new submissions or notifications to deploying organizations, means health systems need contractual mechanisms to know when the tool they approved is no longer the tool they're running. Federal deregulation doesn't reduce organizational accountability for AI safety. For health systems, it increases it.

The practical implication: governance committees that treat vendor contracts as an afterthought in the procurement process are accepting risk that should be allocated explicitly before signature. The questions worth negotiating over are well understood by leading health systems. The challenge is building the internal capacity to ask them consistently.

What to Require Before You Agree to Deploy

Governance committees should approach vendor due diligence with a consistent set of requirements that go beyond marketing collateral. The documentation that matters most:

Model documentation equivalent to a CHAI Applied Model Card

The Coalition for Health AI's Applied Model Card is a standardized format for describing AI tool performance, training data characteristics, known limitations, and bias evaluation. Requiring vendors to complete this format, or supply documentation covering the same elements, gives your governance committee a consistent basis for comparison across different tools. The elements that matter most: intended use and out-of-scope uses, training data sources and demographic characteristics, validation methodology and population, known failure modes, and bias evaluation results across demographic subgroups.

Demographic subgroup performance data

Headline accuracy metrics from vendor test conditions don't tell you how the tool performs on your patient population. Require vendors to provide performance data broken out across demographic subgroups: race, ethnicity, age, sex, socioeconomic status, and any other dimensions relevant to your patient mix. If the vendor's validation evidence comes from a population that doesn't match yours, that's a signal to require local validation before full deployment, not after.

Independent validation evidence where available

Leading health systems, including Emory Healthcare, have built their own de-identified test datasets to benchmark vendor algorithms rather than accepting vendor-provided performance summaries. Ask whether independent validation data exists, who conducted it, and what population it covered. Vendors who've had their tools independently validated should have no difficulty sharing those results. Vendors who cite proprietary methodology as a reason not to share performance evidence are declining to give you information you need to make a governance decision.

Known limitations and failure mode documentation

A vendor with a mature safety culture maintains documentation of how their AI fails: what conditions trigger errors, what patient characteristics increase risk, what workflow contexts produce unexpected outputs. Request this documentation explicitly. A vendor that claims to have none hasn't found their failure modes. A vendor that refuses to share documentation of known limitations is telling you something about how they'd respond to a post-deployment problem.

What Contracts Need to Say

Standard software contracts weren't written with AI governance in mind. Health systems that accept vendor-provided templates without AI-specific modifications are leaving significant gaps. Six areas that need explicit contract language:

Transparency obligations

Require vendors to supply model documentation including intended use, training data characteristics, known limitations, and validation results. Vague commitments to 'work collaboratively on governance' aren't enforceable. Define specifically what documentation the vendor is obligated to provide, in what format, and on what timeline.

Model change notification with a named recipient

The PCCP framework allows vendors to update FDA-cleared algorithms without new FDA submissions. Contracts must require advance notification when vendors materially update the underlying AI model, define what constitutes a material change, and name a specific role at your organization who receives that notification. Notifications that go to a generic inbox or a departed employee provide no governance value.

Local validation support and monitoring data access

Require vendors to cooperate with your team in conducting in-house performance evaluation, including providing access to model outputs or model details sufficient to support that evaluation. Also require ongoing monitoring data access: the ability to assess post-deployment performance against the specifications the vendor warranted. Without both, you're dependent on the vendor's own self-assessment.

Rollback provisions

When a vendor updates a model and the new version underperforms, you need the contractual right to revert to the prior version without penalty while remediation is underway. Standard software contracts don't include this. It needs to be negotiated explicitly, and it needs to define what 'underperforms' means in measurable terms.

Suspension rights without financial penalty

If real-world performance fails to match the specifications the vendor warranted, your organization needs to be able to suspend use without being locked into a contract that treats performance failure as a termination-for-cause issue. Vendors who refuse any form of suspension right are indicating they don't expect to stand behind their performance claims.

Incident reporting obligations

Require the vendor to notify your organization within a defined timeframe when they identify performance issues, known defects, or material changes that could affect patient safety. Define the notification window, the documentation required, and the vendor's obligation to cooperate in any incident investigation involving their tool.

Data Governance: What Standard BAAs Miss

A standard Business Associate Agreement protects PHI under HIPAA's framework, but it wasn't written with AI-specific data use in mind. The questions that matter most in an AI vendor relationship go beyond standard BAA language, and they need to be addressed explicitly before you sign.

Patient data use for model training

AI vendors frequently include provisions, explicitly or through broad language, allowing them to use de-identified patient data to improve their models for other customers. This should be addressed directly. Patient data generated in your deployment should not be used for vendor model improvement without your explicit consent. This is not standard BAA language and needs to be added.

Rights to model outputs

If the vendor's system generates outputs from your patient data, clarify who owns those outputs and what the vendor can do with aggregated or de-identified versions of them. This question has become more complex as vendors have begun using interaction data to improve their systems.

Data minimization requirements

Define what data the vendor actually needs to provide the contracted service, and require them to limit collection to that scope. The minimum necessary standard applies to Business Associate relationships, but AI vendor contracts often allow broad data collection beyond what the service requires.

Re-identification prohibitions and audit rights

Explicitly prohibit re-identification of de-identified data, with penalties for non-compliance. Reserve the right to audit vendor data handling practices, with defined procedures and timelines. CHIME has flagged that some vendors refuse provider terms in BAAs or attempt to cap liability at levels that shift risk disproportionately to health systems. These provisions are worth examining carefully.

Where Liability Actually Falls

This is the piece that surprises most health system leaders when they look at it carefully. In the current legal environment, liability for AI-related patient harm falls primarily on hospitals and physicians, not vendors, absent an explicit product defect. Vendors have carefully written their contracts to limit indemnification obligations, cap damages at nominal amounts relative to potential clinical harm exposure, and disclaim warranty for clinical outcomes.

The practical implication: your health system is holding most of the risk for AI tool performance regardless of what the vendor said in the sales process. Contracts that explicitly allocate risk and require vendor cooperation in incident investigation are your primary lever for changing that equation. At minimum, contracts should require the vendor to cooperate fully in any incident investigation involving their tool, provide timely access to model logs, version history, and change documentation relevant to the incident, and share in liability exposure when performance deviated from specifications they warranted.

Building your own test datasets, demanding audit rights, and ensuring contractual leverage to suspend or terminate if real-world performance fails to match warranted performance are the institutional practices that give contracts teeth. Contracts without operational enforcement mechanisms are difficult to rely on when something goes wrong.

Red Flags in Vendor Negotiations

·Refusal to provide demographic subgroup performance data, citing proprietary methodology
·Resistance to monitoring data access provisions, framed as a security or intellectual property concern
·Flat refusal to include any form of suspension rights without financial penalty
·Inability to define what constitutes a 'significant' model change requiring notification
·BAA language that implicitly or explicitly permits patient data use for model training without consent
·Indemnification caps that are nominal relative to potential clinical harm exposure
·Resistance to rollback provisions for model updates that underperform

Any of these in isolation might have a legitimate explanation. Multiple of them together, or any single one combined with resistance to discuss the underlying concern, warrants a serious conversation before the contract is signed.

Frequently Asked Questions

Common questions from health system leaders navigating AI vendor contracts and due diligence.

What is a CHAI Applied Model Card and should we require it from vendors?

A CHAI Applied Model Card is a standardized documentation format developed by the Coalition for Health AI for describing AI tool performance, training data, known limitations, and bias evaluation. It gives health systems a consistent way to compare what different vendors are disclosing. Requiring vendors to complete this format, or provide documentation covering the same elements, is a reasonable due diligence ask. Vendors unwilling to complete it should explain specifically what prevents them from doing so.

What does the PCCP framework mean for our vendor contracts?

The Predetermined Change Control Plan framework, finalized by the FDA in December 2024, allows vendors with FDA-cleared algorithms to pre-approve categories of model updates without submitting a new application for each change. This means vendors can update the AI model you deployed without any notification to your organization, unless your contract requires it. The PCCP framework was designed to reduce regulatory burden on vendors as models improve over time. The side effect is that health systems need explicit contractual notification requirements to know when the tool they approved is no longer the tool they're running.

Is it reasonable to ask a vendor for suspension rights without penalty?

Yes, and it's becoming more common among leading health systems. The right to suspend a tool without financial penalty if real-world performance deviates materially from warranted specifications is a legitimate governance protection, not an unusual ask. The negotiation is typically around defining 'material deviation' in measurable terms, which is a reasonable conversation to have explicitly before deployment rather than arguing about it after a performance problem emerges.

What should the incident reporting obligation in an AI vendor contract look like?

A well-drafted incident reporting obligation requires the vendor to notify a named contact at your organization within a defined window, typically 48 to 72 hours, when they identify performance issues, known defects, or changes that could affect patient safety. It also requires the vendor to cooperate in any internal investigation involving their tool, provide access to relevant logs and documentation within a defined number of days of the request, and produce a post-incident report documenting root cause and remediation steps.

Do we need AI-specific language in our BAA or a separate data use agreement?

In most cases, yes. Standard BAA language was designed to address HIPAA's requirements around PHI handling, not the AI-specific questions that matter most in vendor relationships: whether patient data can be used for model training, who owns model outputs, and what audit rights you have over vendor data handling practices. These provisions can be added to the BAA itself or addressed in a supplemental data use agreement, but they need to be addressed explicitly in one of those documents. Off-the-shelf BAA templates almost never cover them.

What should we do when a vendor refuses to negotiate on core governance provisions?

Document the refusal and escalate it to your governance committee as a procurement risk, not a routine contracting disagreement. Vendors who decline to provide demographic subgroup performance data, resist monitoring data access provisions, or refuse any form of suspension rights are declining governance protections that have become standard expectations among leading health systems. The decision to proceed without those protections can be made, but it should be made explicitly by the governance committee with the specific risk acknowledged and accepted, not by the procurement team in the course of routine contract negotiation.

Sources

Office of the National Coordinator for Health Information Technology (ONC). HTI-5 Proposed Rule. 2025.
U.S. Food and Drug Administration. Predetermined Change Control Plans for Machine Learning-Enabled Medical Devices: Guidance for Industry and FDA Staff. December 2024.
Coalition for Health AI (CHAI). Applied Model Card Framework and Algorithmic Impact Assessment Guidance. 2025.
The Joint Commission and CHAI. Responsible Use of Artificial Intelligence in Healthcare. 2024.
CHIME. AI Principles for Health Information and Technology. 2025.
Emory Healthcare. AI Vendor Evaluation and Local Validation Protocol. Internal practice cited in CHIME AI Governance guidance. 2024.
American Hospital Association. Trustworthy AI in Health Care: A Framework for AI Governance. 2024.

What Should We Require from AI Vendors as a Condition of Deployment in Our Health System?