Evaluating AI Responses to Postoperative Questions in Mohs Reconstruction

Area of Science:

• Dermatology
• Artificial Intelligence
• Medical Informatics

Background:

• Postoperative patient education is crucial after Mohs facial reconstruction.
• Large language models (LLMs) offer potential for improving patient communication.
• Variability in AI-generated medical information requires careful evaluation.

Purpose of the Study:

• To evaluate the quality, accuracy, comprehensiveness, and readability of responses from four leading LLMs to common postoperative questions after Mohs facial reconstruction.
• To compare the performance of Claude AI, ChatGPT, Microsoft Copilot, and Google Gemini in this context.

Main Methods:

• A prospective observational study adhering to STROBE guidelines was conducted.
• Thirty-one common postoperative questions were posed to four LLMs (Claude AI, ChatGPT, Microsoft Copilot, Google Gemini) using a standardized prompt.
• Responses were blindly assessed by facial plastic surgeons using validated tools (EQIP, Likert scales, readability formulas).

Main Results:

• Claude AI demonstrated superior performance in quality (EQIP: 90.3), accuracy (4.55/5), and comprehensiveness (4.60/5).
• All evaluated LLMs generated responses exceeding a 6th-grade reading level.
• Significant variation was observed in the quality and depth of information provided by the different LLMs.

Conclusions:

• LLMs show promise in supporting postoperative communication for patients undergoing Mohs facial reconstruction.
• The observed variability in AI response quality underscores the critical need for physician review and oversight.
• Further research is needed to optimize LLM use in patient education while ensuring medical accuracy.