Quality and safety of large language model generated communication in psychodermatology: a bilingual scenario-based...

Large language models (LLMs) are increasingly used by patients for medical information and reassurance. In psychodermatology, where communication must address psychological distress, stigma, and functional impact, the safety and quality of AI-generated educational content have not been systematically assessed.

To evaluate the quality and safety of patient-facing educational responses generated by contemporary LLMs for psychodermatologic conditions in a bilingual setting, comparing English and a non-English language (Turkish).

This cross-sectional, scenario-based evaluation analyzed responses from five LLMs (GPT-4o, GPT-5, Claude 4 Sonnet, Gemini 2.5 Flash, and LLaMA 3.1 70B) to 16 standardized psychodermatology scenarios. Three blinded clinician-evaluators (two from dermatology and one from psychiatry) independently rated each response across six clinical communication domains. Readability indices and word counts were assessed, and repeated-measures nonparametric analyses were performed.

Gemini 2.5 Flash achieved the highest overall quality scores in both English and Turkish, significantly outperforming LLaMA 3.1, Claude 4 Sonnet, and GPT-4o (P < 0.05). Across models, empathy and stigma-free communication scored highest, whereas actionability and risk management scored lowest. English outputs were longer than Turkish (mean 373.9 vs 269.4 words; P < 0.001). LLaMA 3.1 showed significantly lower quality in Turkish (66.3%) compared with English (77.2%; P < 0.001). Interrater agreement was good (ICC = 0.703).

While LLMs demonstrated strong empathic and stigma-sensitive communication in psychodermatology, they consistently lacked actionable guidance and robust risk framing. These findings support cautious, clinician-supervised use of LLMs as adjunctive tools for patient education in psychodermatologic care.