Beyond Autonomy: Ethical Considerations of AI in Substitute Decision-Making for Incapacitated Patients

The integration of artificial intelligence (AI) into healthcare offers significant potential for enhancing patient care, particularly for incapacitated patients unable to communicate their preferences. Advanced AI systems, such as Personalized Patient Preference Predictors (P4), are increasingly proposed as tools to assist in substitute decision-making, yet their ethical implications remain underexamined.

This paper critically examines the ethical implications of employing AI-driven preference prediction systems in substitute decision-making for incapacitated patients. It evaluates these systems through the ethical principles of autonomy, beneficence, non-maleficence, and justice, and proposes a framework for their ethically sound integration into clinical practice.

The analysis draws on principled ethical reasoning to interrogate key challenges posed by P4 systems, including the distinction between probabilistic preference inference and genuine autonomy, the risk of algorithmic paternalism even in personalized models, and the subtle manifestations of bias within highly individualized datasets.

The analysis reveals that while P4 systems may augment preference prediction, they cannot fully capture the relational, contextual, and value-laden nature of authentic patient preferences. Algorithmic paternalism and latent bias remain salient risks, even in highly personalized models. These findings underscore the indispensability of human surrogates in contextualizing AI-generated data.

AI should function as a decision-support tool that augments, rather than replaces, the holistic judgment of human surrogates. An ethical framework integrating AI predictions within a model of relational autonomy is proposed. Key recommendations include maintaining meaningful human oversight, developing transparent and interpretable AI, and establishing a robust regulatory framework to ensure ethically sound and patient-centered care.