AI Governance in Healthcare: EU Rules and Practical Controls

AI governance in healthcare assigns responsibility for how systems are selected, tested, used and reviewed. The legal assessment depends on each system’s intended purpose: medical-device rules, the EU AI Act and GDPR have different scope tests. Clinical AI is not automatically an Annex III high-risk system, and administrative uses still need an assessment. AI Act, Article 6 and Annex III; MDR, Article 2; GDPR, Articles 6 and 9

This guide reflects EU legal sources checked on 14 September 2026. It proposes practical records and review steps for hospitals and other healthcare organisations. The composition of a governance committee and the operational examples below are recommendations, rather than prescribed legal structures.

Classify the intended use before choosing controls

Record the medical or administrative purpose, intended users, patient population, inputs, outputs and decisions affected. Separate the classification under product law from the AI Act assessment.

These are classification questions, not examples of customer deployments. Sources: MDR, Article 2(1); IVDR, Article 2(2); AI Act, Article 6 and Annex III.

For the AI Act product route, the AI must be a covered product or its safety component and the relevant product legislation must require third-party conformity assessment. MDR and IVDR appear in Annex I, Section A. Article 6(1a)–(1c) clarifies safety components and the assessment trigger. Article 6(3)‘s exception concerns Annex III systems; it is not a general exemption for medical devices. AI Act, Article 6 and Annex I

Where an Annex III exception is proposed, assess the significance of the risk and the specified conditions. A preparatory task can qualify, but profiling of natural persons prevents reliance on that exception. The provider must document the assessment. AI Act, Article 6(3)–(4)

Check medical-device qualification and conformity

MDR qualification turns on the manufacturer’s intended medical purpose. For software that qualifies as a device, Annex VIII Rule 11 classifies software informing diagnostic or therapeutic decisions as class IIa, with class IIb or III applying where the possible consequences meet the stated thresholds. Physiological-monitoring software has separate conditions. Rule 11 does not use a test of whether a clinician can independently verify an output. MDR, Article 2(1) and Annex VIII, Rule 11

Conformity procedures depend on the device class and circumstances. MDR Article 52 provides notified-body assessment routes for class IIa, IIb and III devices and a manufacturer declaration route for ordinary class I devices, with specified exceptions. It is inaccurate to say that every healthcare AI system requires a notified-body certificate. The conditional health-institution exemption in Article 5(5) also needs a specific assessment; developing software internally does not establish eligibility by itself. MDR, Articles 5 and 52

For applicable high-risk medical products, AI Act Article 43(3) incorporates AI requirements into the product-law conformity assessment. Article 2(13) provides for specific limits on overlapping requirements through delegated acts. Check the measures that actually apply to the device before assuming that one regime displaces another. AI Act, Articles 2(13) and 43(3)

Use the current AI Act timetable

AI Act, Article 113. These dates do not delay existing MDR, IVDR or GDPR requirements. For older systems, examine the Article 111 transition rules and subsequent design changes. The separate transition to 2 December 2026 in Article 111(4) concerns certain providers’ Article 50(2) synthetic-content marking obligations, rather than all transparency duties.

AI literacy also needs attention now. Amended Article 4 requires measures supporting relevant staff’s AI literacy, while expressly excluding an obligation to guarantee a specific individual level. Practical training can cover intended uses, limitations, handling patient information and escalation. AI Act, Article 4

Define responsibility for each system

An organisation using purchased AI under its authority will generally be a deployer. It can also be a provider where it develops or commissions a system and puts it into service under its own name. Specified changes or rebranding can trigger provider responsibilities under Article 25. AI Act, Articles 3(3)–(4) and 25

For high-risk systems within the applicable scope and dates, Article 26 requires deployers to assign human oversight to people with the necessary competence, training, authority and support. It also covers use according to instructions, relevant input data where controlled by the deployer, monitoring and escalation. AI Act, Article 26

As a practical arrangement, identify a clinical owner for patient-facing decisions and a technical owner for system operation. Involve regulatory affairs, information security and data protection when the use requires them. Record who can approve use, restrict it and stop it. A committee can resolve issues that cross those responsibilities; it should not leave the person on a clinical shift uncertain about whom to contact.

Assemble a review record before use

The following is a proposed evidence checklist. Adapt it to the device, workflow and applicable duties.

A performance result from another setting may leave local questions unanswered. Document those questions and the evidence needed to resolve them. Avoid a universal accuracy threshold or fixed validation design for every type of clinical software. MDR clinical evaluation and AI Act risk-management requirements depend on the applicable purpose and system. MDR, Articles 5(3) and 61; AI Act, Article 9

Assess health-data processing and patient information

Identify both an Article 6 GDPR lawful basis and an applicable Article 9 condition for processing health data. Explicit consent is one possible Article 9 condition. Healthcare provision, public health and research have other conditions with their own safeguards; a care-related permission does not establish that unrelated vendor model training is permitted. Assess each purpose and any proposed reuse. GDPR, Articles 5, 6 and 9

A DPIA is required where processing is likely to create high risk to people’s rights and freedoms. Article 35 expressly includes large-scale processing of special-category data. It does not state that every tool processing any patient record automatically requires a DPIA; relevant supervisory-authority lists must also be checked. GDPR, Article 35

For bias detection, the amended AI Act’s Article 4a permits certain processing of special-category data only where strictly necessary and subject to detailed safeguards. It is not a general permission to collect sensitive attributes or share them with suppliers. Include data-protection review before adding subgroup data to an evaluation. AI Act, Article 4a

Patient information also needs the correct legal basis. Article 50 covers specified direct AI interactions and other transparency uses. Article 26(11) concerns information to people affected by Annex III high-risk decision systems. GDPR Article 22 concerns solely automated decisions with legal or similarly significant effects, subject to its exceptions and safeguards; it does not create a general right to demand that every AI-assisted clinical decision exclude AI. AI Act, Articles 26 and 50; GDPR, Article 22

Plan monitoring, incidents and changes

MDR Article 83 places post-market surveillance duties on manufacturers, proportionate to the device type and risk class. Article 87 addresses manufacturer reporting of serious incidents and field safety corrective actions. The AI Act separately addresses provider post-market monitoring and deployer monitoring and escalation. Assign the applicable duties to the right actor. MDR, Articles 83 and 87; AI Act, Articles 26 and 72

Operationally, record the deployed version, relevant performance measures, reported problems and who reviews them. Set review frequency around the clinical risk, volume of use and availability of outcome data. Define events that trigger immediate review, such as a serious incident, a changed intended use or a material software update. Test how staff can continue care when the system is unavailable or withdrawn.

For the broader enterprise rules, see the EU AI Act compliance guide.

Frequently Asked Questions

Is all clinical AI high risk under the EU AI Act?

No blanket classification applies. Medical-device AI needs the Article 6(1)/Annex I test, including the third-party conformity-assessment condition. Annex III separately lists uses such as emergency triage. An actual system can require review under both routes. AI Act, Article 6 and Annexes I and III

Does every healthcare AI system need a notified-body certificate?

No. Establish whether the software is a medical device, then its class and conformity route. MDR Article 52 includes a declaration route for ordinary class I devices and different assessment routes for higher classes. Other conditions, including any proposed in-house exemption, need separate examination. MDR, Articles 5 and 52

Does clinician review remove high-risk status?

Clinician review does not remove product qualification or high-risk classification by itself. For Annex III systems, any Article 6(3) exception must meet its conditions. For medical-device AI, the separate Article 6(1) test remains relevant. AI Act, Article 6

Must a hospital complete a FRIA for every medical AI device?

Article 27 concerns specified deployers of Annex III high-risk systems, rather than all medical devices. Public-law bodies and private entities providing public services should check its scope for each Annex III use. A GDPR DPIA can apply separately, and relevant DPIA material can be reused in a FRIA. AI Act, Article 27; GDPR, Article 35

Is patient consent always required for healthcare AI?

GDPR explicit consent is one possible condition for processing health data. Other Article 9 conditions can apply, including healthcare provision subject to the stated safeguards. Identify the actual processing purpose, Article 6 basis and Article 9 condition. This does not resolve separate consent requirements for treatment or research. GDPR, Articles 6 and 9

How often should clinical AI be reviewed?

Set an operational schedule appropriate to the use and risk, with additional reviews when conditions change. MDR Article 83 requires a proportionate surveillance system; it does not prescribe one universal real-time dashboard or quarterly committee schedule. MDR, Article 83

Is €35 million the standard penalty for a healthcare AI compliance failure?

No. The AI Act’s €35 million/7% tier concerns Article 5 prohibited practices. Article 99(4) sets a different ceiling for listed duties, including Articles 16 and 26. Smaller-enterprise rules, the circumstances of the infringement and rules for public bodies also matter. AI Act, Article 99

Contact The Thinking Company.