A Classification of Safety Risks in Medical AI

AbstractIn clinical practice, patient safety is paramount, guided by the principle "first, do no harm." For medical AI to become a core component of clinical care, safety cannot be determined by its performance or accuracy alone, but by whether or not it protects patients from harm. However, consequences of medical AI models - used either independently or in combination with clinicians - may result in harms that can emerge years after the initial AI-enabled diagnosis or intervention, are difficult to attribute to the original AI model, and are irreversible (e.g., serious morbidity or mortality). We propose a four-level clinical safety classification for medical AI. S1 refers to narrow, testable uses; S2 to decision support that clinicians can review before acting; S3 to uses in which important harms may appear only after follow-up and therefore require surveillance and outcome monitoring before wider rollout; and S4 to high-risk autonomous actions that should be prohibited or allowed only under exceptional safeguards. Using a real-world prospective study of DeepDR-LLM, an integrated image-language system that combines a large language model with image-based deep learning for primary diabetes care and diabetic retinopathy screening, we illustrate how the same AI function may remain classified as S2 when used as clinician-reviewed decision support, become S3 when safety depends on referral completion and follow-up outcomes, or approach S4 if used to bypass clinical judgment. We propose multiwindow postdeployment surveillance that captures near-term signals, scheduled follow-up checkpoints, and long-horizon monitoring beyond 12 months through registry or claims linkage and postimplementation surveillance studies, strengthening medicine's core commitment to "first, do no harm - both now and in the future."