On the risks of depersonalizing consent and the safe implementation of LLMs in medical decision-making

Zohny et al provide a proof of concept for large language model (LLM)-patient communication in medical decision-making, discussing some of the risks and potential downsides of implementing this technology. However, removing human healthcare professionals (HCPs) from medical decision-making carries further risks they do not discuss. These include risks that a conscientious HCP with appropriate training can address, including (1) diminished situational autonomy due to pressure from family members or loved ones, (2) barriers to autonomy due to the situational inflexibility of LLMs and (3) diminished comfort and trust due to lack of human empathy. While the central moral focus in the implementation of this technology should be patient consent and the process that supports it, the dehumanisation of medical decision-making risks broader negative effects on both patients and HCPs, some of which are also discussed in this article. These concerns should be addressed to minimise the harm and maximise the good that LLMs can do to enhance patient decision-making.