Divergent impacts of explainable AI for dermatological diagnosis on clinicians versus lay people

Artificial intelligence (AI) is increasingly permeating healthcare, from serving as a physician assistant to powering consumer applications. The opacity of AI algorithms makes the ability of humans to interact with AI algorithms challenging. To overcome this limitation, explainable AI (XAI) provides insight into AI decision-making, but evidence suggests that XAI can paradoxically induce bias in the human decision-making process. Here we present results from two large-scale experiments, involving 623 lay people and 153 primary care physicians (PCPs), respectively, in which a fairness-based AI model for dermatological diagnoses and different XAI-based explanations were combined to examine how XAI assistance, particularly multimodal large language models (LLMs), influences diagnostic performance. With fairness-constrained model training, assistance from an AI model that achieved balanced performance across skin tones improved final diagnostic accuracy and reduced skin-tone-related performance disparities among both lay people and PCPs. In this setting, LLM explanations yielded divergent effects: lay users showed higher automation bias-accuracy was boosted when the diagnoses provided by the AI model were correct but was reduced when the model erred-whereas experienced PCPs remained resilient, benefiting irrespective of the AI model's accuracy. In addition, presenting the AI model's diagnosis before human decision-making may lead to stronger anchoring bias. These findings highlight XAI's varying impacts based on human expertise and the timing of when the AI-based prediction is provided, underscoring the concept that LLMs can act as a 'double-edged sword' in medical AI and informing future human-AI collaborative system design.