Evaluating large language models in patient education on facial plastic surgery: a standardized protocol

Large language models (LLMs) are increasingly used in healthcare settings to provide patient education and answer medical inquiries. However, their reliability in delivering accurate, clear, and unbiased information remains uncertain. This study aims to evaluate the quality of responses generated by LLMs to common patient questions regarding facial plastic surgery.

A total of 60 patient-oriented questions related to facial plastic surgery will be selected from professional bodies, patient support groups, and social media platforms. These questions will be categorized into six main topics: fundamental knowledge, preoperative considerations, surgical procedures, procedural risks and postoperative complications, preparation and recovery, and miscellaneous concerns. Seven LLMs - ChatGPT 4o, Claude, Copilot, DeepSeek, Gemini, Grok, and OpenEvidence - will be tested by inputting each question twice using the "New Chat" feature to assess response consistency. Responses will be evaluated by ten American board-certified plastic surgeons using a structured scoring rubric covering four criteria: accuracy, clarity, completeness, and appropriateness. A standardized scoring system will be employed, and inter-rater reliability will be measured to ensure consistency among evaluators.

By systematically assessing the responses of multiple LLMs to patient inquiries on facial plastic surgery, this study will provide insights into their reliability and clinical applicability. Findings may help refine LLM-based tools for patient education and identify areas requiring improvement to ensure safe and effective AI-assisted communication in plastic surgery.