How Do We Evaluate an AI Vendor's Claims and What Questions Should We Ask Before Signing a Contract?

The Texas Attorney General's office settled with Pieces Technologies, a healthcare AI documentation company, over allegations that the company marketed its AI with materially misleading accuracy claims. Specifically, the company claimed an error rate below 1 in 100,000 and that its AI performed better than humans — claims that were based on a methodology measuring individual data-point accuracy rather than the clinical summary accuracy that actually matters clinically. The AI also produced hallucinations, and the company continued using those marketing claims after receiving error reports from clients. The settlement matters because it's the first major enforcement action establishing what misleading AI marketing in healthcare looks like, and it signals that regulators are willing to act. Every governance committee evaluating a vendor should read it.

Local validation means testing a vendor's AI tool against your own patient population, using your own data, before relying on it clinically. It's distinct from accepting the vendor's published validation studies, which may have been conducted on a different population under different conditions. Emory Healthcare is the frequently cited example — they built de-identified test datasets from their own patient records and used them to benchmark vendor algorithms rather than accepting vendor-provided summaries. The practical implementation involves your clinical informatics and quality teams working with the vendor to run the AI against a representative sample of your historical cases where the correct answer is already known, then comparing the AI's output to the documented clinical truth. This requires vendor cooperation, which is why a local validation clause in the contract matters.

No. FDA clearance for a medical device means the FDA determined the device is substantially equivalent to a predicate device — it's a regulatory threshold, not a clinical performance guarantee for your specific patient population. FDA-cleared AI tools have been recalled, suspended, and discontinued by hospitals after post-deployment validation revealed performance problems. FDA clearance is one input to your evaluation, not a substitute for it. The Baptist Health vascular ultrasound case is instructive here — an FDA-cleared tool was missing aneurysms in Baptist Health's patient population. Regulatory clearance didn't prevent the problem, and Baptist Health's own monitoring detected it.

Four consistently: local validation clauses (vendor commitment to support in-house performance evaluation), suspension rights (ability to halt use without penalty if performance deviates from specifications), notification requirements (vendor obligation to inform the organization of model changes, issues, or material performance changes), and data use restrictions (explicit prohibition on using patient data for vendor model improvement without consent, backed by a current and AI-specific Business Associate Agreement). Most standard vendor contracts are written to protect the vendor. The burden is on your legal counsel and governance committee to negotiate these provisions before signing.

A suspension right is a contractual provision that allows your organization to stop using a vendor's AI tool — and stop paying for it — without financial penalty if the tool's performance deviates materially from the vendor's stated specifications. Without this provision, you may be contractually obligated to continue paying for a contract even after you've stopped using the tool for patient safety reasons. This matters because AI model performance can degrade over time, particularly as patient populations shift or as the underlying model is updated by the vendor. Having the right to suspend use if performance falls below a contractually defined threshold is a basic protection that many organizations don't have.

Refusal to share validation data or failure mode documentation should be a significant red flag in your procurement evaluation. A vendor with genuine confidence in their product's safety and performance should welcome local validation — it protects them from liability and builds customer confidence. When vendors decline to share performance data, the most likely explanations are that the data doesn't support their marketing claims, or that the vendor's legal team has assessed the disclosure risk and decided the risk of sharing outweighs the sales benefit. Neither is a comfortable position for a health system to be in. If you're proceeding with a vendor who won't share performance data, document that decision carefully and ensure your governance committee has explicitly accepted that risk.