Healthcare AI Systems
Build healthcare AI systems: medical imaging, regulatory compliance, privacy, and clinical decision support.
What is a healthcare AI system?
A healthcare AI system uses a learned model inside a health-related clinical, operational, research, or administrative workflow. The system is larger than the model: it includes the intended use, data provenance, user interface, human authority, integration, validation evidence, security controls, monitoring, and a way to stop or recover.
This matters because a model can remain online and produce plausible outputs while the population, device, workflow, or meaning of an input has changed. The core invariant is: use the system only inside its validated boundary, make uncertainty and provenance visible, and preserve a safe path when the boundary is not met.
- Treat the intended use as a testable contract, not a marketing sentence.
- Evaluate the human-AI team and downstream workflow, not only an offline score.
- Measure performance on clinically and operationally relevant slices.
- Keep every model or policy change versioned, reviewable, and reversible.
This lesson is educational architecture guidance, not clinical, legal, or regulatory advice. It does not diagnose, treat, clear, certify, or approve any product. Real deployments require qualified clinical, privacy, security, quality, and regulatory professionals in the applicable jurisdiction.
1 Intent
Define the use boundary
Name the user, patient population, setting, input, output, decision, exclusions, and fallback before selecting a model.
2 Develop
Build traceable evidence
Control provenance and leakage, separate patients across splits, and document how labels and reference standards were created.
3 Evaluate
Validate the whole workflow
Test discrimination, calibration, subgroup behavior, usability, integration, and the consequences of errors under realistic conditions.
4 Deploy
Release with containment
Enforce eligibility checks, abstention, human review, versioned outputs, access controls, and a tested fallback.
5 Operate
Monitor and govern change
Join outcomes when available, investigate drift by slice, review overrides and incidents, and revalidate meaningful changes.
Frame the intended use before the model
An intended-use contract answers four questions that architecture alone cannot recover later:
- Who and where: Which trained user, patient population, care setting, geography, and device environment are in scope?
- What input and output: Which source systems, acquisition procedures, data-quality conditions, output semantics, and uncertainty signals are required?
- Which decision: Does the output inform prioritization, recommend options, produce a time-critical alert, or direct a diagnosis or treatment action?
- Who retains authority: Can the user independently review the basis, and what happens when the system abstains, conflicts with other evidence, or is unavailable?
Workflow support
Operational assistance
Forecast staffing, detect scheduling pressure, or summarize administrative queues. These uses can still affect access and equity, so outcome and slice monitoring remain necessary.
Evidence for a professional
Clinical decision support
Present patient-specific information or options with the basis, limitations, and provenance needed for independent review. Intended user and time criticality matter.
Device-relevant function
Medical image or signal analysis
Analyze an image, waveform, or other medical signal for a medical purpose. In the United States, FDA's current policy navigator says these functions do not meet the medical-image or signal criterion for the Non-Device CDS exclusion.
Direct communication
Patient-facing guidance
Outputs sent to patients or caregivers need a deliberately designed escalation path, comprehensible limitations, privacy controls, and jurisdiction-specific regulatory review.
The same model can have a different risk and regulatory posture when its intended user, input, time criticality, or effect on a decision changes. Document the function, not just the algorithm family.
Turn model accuracy into workflow consequences
Sensitivity and specificity describe behavior conditional on the true outcome. They do not tell a team how many alerts a workflow must review or how trustworthy a positive result is in a new population. Prevalence connects the operating point to those consequences.
Use the lab to change the cohort, prevalence, and operating point. Watch the confusion matrix, positive predictive value, and review queue move together. All values are illustrative and are not a clinical threshold recommendation.
See the review queue behind a model score
Loading the illustrative workload model...
Read the four cells before an aggregate score
- True positive: the model flags a positive case; downstream benefit still depends on timely and correct review.
- False positive: the model flags a negative case; the workflow absorbs extra review, anxiety, delay, or follow-up cost.
- False negative: the model does not flag a positive case; the ordinary care pathway and safety net must still exist.
- True negative: the model does not flag a negative case; this is useful only when the input and population were eligible for the model.
Build evidence that matches the deployment claim
No single metric validates a healthcare AI system. Evidence should mirror the intended population, setting, user, input acquisition, workflow, and decision impact.
Patient
Split unit
Prevent the same person's correlated records from leaking across development and evaluation
Slice
Performance view
Report relevant populations, sites, devices, quality bands, and missingness patterns
Team
Evaluation unit
Measure the human-AI workflow, including overrides, delays, and automation bias
Lifecycle
Evidence horizon
Monitor real use and manage retraining or other changes after release
Separate the evidence layers
- Analytical and data validity: verify input integrity, units, acquisition constraints, label quality, reproducibility, and software correctness.
- Model validity: evaluate discrimination, calibration, uncertainty, and error costs with confidence intervals on untouched data.
- External validity: test sites, devices, time periods, and populations that represent the deployment claim rather than the development convenience sample.
- Human factors and workflow validity: observe whether intended users understand, challenge, override, and act on outputs as designed.
- Prospective and post-deployment evidence: use the study design and oversight appropriate to the claim, then monitor outcomes and incidents after release.
Do not convert a retrospective benchmark into a clinical claim. Dataset labels, prevalence, referral patterns, acquisition devices, and missingness can all differ from the real workflow.
Design the runtime around eligibility and review
The serving path should reject or route unsupported inputs before inference, preserve the evidence required for review, and keep patient identifiers out of model telemetry unless explicitly required and protected.
A bounded healthcare inference path
Every stage has an owner and an observable fallback. The model result never bypasses eligibility, review, or audit controls.
Provenance
Acquire and identify
Bind the source, encounter, device, timestamp, consent or authority, and schema version to the request.
Use boundary
Check eligibility
Validate population, setting, modality, quality, required fields, and supported software or device versions.
Versioned model
Infer and calibrate
Return the model version, output, uncertainty or abstention state, and traceable transformation versions.
Human-AI team
Review in context
Show the basis, limitations, alternatives, patient context, and next action without hiding disagreement.
Lifecycle evidence
Record and observe
Log the authorized action, override, latency, fallback, later outcome, and incident link under an auditable retention policy.
Make failure behavior explicit
- Unsupported input: abstain or route to the approved non-AI workflow; do not silently coerce the record.
- Dependency outage: preserve the ordinary workflow and make degraded mode visible to the user.
- High uncertainty: communicate the state in language the intended user understands and route according to the validated protocol.
- Version mismatch: reject mixed preprocessing, model, calibration, or label contracts.
- Delayed outcomes: keep the prediction-time population and policy context so later evaluation is not biased by missing joins.
Treat every meaningful change as an evidence question
A model can become out of scope without changing its weights. A new scanner, EHR mapping, referral pathway, patient population, user interface, alert destination, or decision policy may alter the system's behavior or the claim supported by existing evidence.
Use the change-control lab to select a boundary change, vary the strength of independent evidence, and choose whether a tested fallback is available. The output is an engineering release posture, not a regulatory determination.
Decide how much evidence a system change needs
Loading change-control scenarios...
Monitor outcomes, slices, and the human-AI team
Service health is necessary but not sufficient. Join technical telemetry to workflow and outcome evidence with governance appropriate to the data.
Can the model operate?
Input and eligibility
Track rejected inputs, missingness, acquisition quality, device and site mix, unsupported versions, and changes in the eligible population.
How is it being used?
Model and workflow
Track score and abstention distributions, calibration when labels arrive, review latency, overrides, disagreement, fallback use, and alert fatigue.
Who benefits or is harmed?
Outcome and equity
Track relevant outcomes and error costs by predefined slice, while accounting for delayed labels, missing follow-up, interventions, and selective outcome observation.
Define incident triggers before release
- a safety, privacy, security, or data-integrity event linked to the system;
- a sustained breach of a validated performance or calibration boundary;
- unexplained performance divergence across an important population or site;
- a sharp change in overrides, abstentions, alert volume, or fallback use;
- a pipeline, device, interface, model, or policy change outside the approved plan;
- inability to reconstruct which inputs, versions, and user action produced an outcome.
The response should name the owner, containment action, evidence to preserve, notification path, rollback criterion, and conditions for resuming use.
Protect health data without weakening observability
Privacy and security controls depend on jurisdiction and organizational role. In the United States, the HIPAA Security Rule applies to covered entities and business associates handling electronic protected health information and requires administrative, physical, and technical safeguards for confidentiality, integrity, and availability.
- Minimize data to what the intended function and evaluation plan require.
- Separate direct identifiers from model-development and monitoring datasets where feasible.
- Enforce least privilege, strong authentication, encryption, key rotation, and environment separation.
- Record access, export, model invocation, configuration changes, and privileged administrative actions.
- Threat-model model artifacts, prompts, embeddings, temporary files, debug traces, and vendor integrations as possible disclosure paths.
- Test backup, restore, downtime, and breach-response procedures with the clinical or operational fallback.
De-identification and privacy-enhancing techniques reduce particular risks; they do not automatically make data anonymous, unbiased, secure, or reusable for every purpose.
Use regulatory sources as constraints, not shortcuts
Regulatory status is function- and jurisdiction-specific. Avoid fixed approval timelines or universal pathway claims. Start with current primary guidance and involve qualified specialists.
- The FDA's January 2026 Clinical Decision Support Software guidance explains the agency's current interpretation of the U.S. Non-Device CDS criteria and includes examples for device and non-device functions.
- FDA's current policy navigator emphasizes the intended input, intended user, recommendation versus directive, time criticality, and whether a health care professional can independently review the basis.
- The 2025 IMDRF Good Machine Learning Practice principles emphasize representative datasets, independent test sets, human-AI team performance, clear user information, and lifecycle monitoring.
- FDA's August 2025 final guidance for predetermined change control plans recommends describing planned modifications, the method for developing and validating them, and an impact assessment.
- WHO's ethics guidance frames autonomy, safety, transparency, accountability, equity, and responsiveness as governance responsibilities for AI used in health.
Primary sources
- FDA: Clinical Decision Support Software guidance, January 2026
- FDA: Digital Health Policy Navigator, Step 6
- FDA: Good Machine Learning Practice guiding principles
- FDA: Transparency for machine learning-enabled medical devices
- FDA: Predetermined Change Control Plan final guidance, August 2025
- HHS: HIPAA Security Rule
- WHO: Ethics and governance of artificial intelligence for health
- NIST: Artificial Intelligence Risk Management Framework 1.0
Review the production readiness boundary
Before a healthcare AI release, require evidence that the team can answer each question:
- Is the intended use, population, setting, input, output, user, exclusion, and fallback precise?
- Are provenance, label quality, patient-level splits, and dataset limitations documented?
- Do external, slice-level, calibration, uncertainty, and workflow evaluations match the claim?
- Can the runtime reject unsupported inputs and preserve model, transformation, and policy versions?
- Can intended users understand the output, basis, limitations, and required action without automation bias?
- Are privacy, security, access, retention, downtime, and incident controls tested?
- Are outcomes, overrides, abstentions, drift, and adverse events monitored with named owners?
- Does the change plan state what can change, what evidence is required, and how to roll back?
If a team cannot reconstruct the intended use, evidence, exact runtime versions, and human action for a consequential output, the system is not operationally auditable.