"Black box" artificial intelligence for mortality prediction: a mixed-methods study of palliative care team, patient,...

New artificial intelligence (AI)-based mortality prediction algorithms could support both patients' prognostic awareness and person-centered palliative care. Although they promise accuracy, their outputs can be hard to explain-potentially affecting whether patients and care teams use them. To investigate perspectives on the explainability of AI algorithms in palliative care, we conducted a sequential mixed-methods study.

We interviewed 30 palliative care physicians and nurses; 15 social workers, spiritual care providers, psychologists, and others; and 35 patients and caregivers at four U.S. academic centers (total n=80). The 53 interviews containing data on explainability were analyzed thematically to understand reasons for concern or unconcern. We randomly sampled and surveyed n=2,500 palliative care physicians (overall adjusted response rate, 32.6%). The 537 surveys with complete responses on explainability items were analyzed descriptively; a multivariable model examined predictors of concern.

Among 53 interviewees, 18 expressed only concern about black box AI-based prognostication, 17 expressed only unconcern, and 18 interviewees expressed mixed sentiments. Reasons for concern related to: data transparency, mistrust of machines or their creators, patient-clinician communication, bias, and accuracy. Reasons for unconcern related to: inexplicability not unique to AI, greater accuracy, not using AI in isolation, trust in science, and being evidence-based. Notably, "accuracy" and "trust" appeared in both. Overall, 75% of physicians (n=396/528) reported being at least "moderately concerned" about unexplainable AI algorithms. Male physicians were less likely to be strongly concerned [adjusted odds ratio (aOR) 0.57; 95% confidence interval (CI): 0.36, 0.89; P=0.01] about explainability. Those who perceived AI mortality prediction to be inaccurate were more likely to be concerned (aOR 2.06; 95% CI: 1.27, 3.41; P=0.003).

Our findings suggest that if a black box model is perceived as accurate, there may be less demand for explainability. Nevertheless, in palliative care-where communication is key-explainability may still be central. Future efforts should seek to create models that are both accurate and explainable at the point-of-care.