Accuracy, readability, and bias of GPT-4o mini responses to oculoplastic patient questions

This cross-sectional evaluation of model outputs evaluates the accuracy, readability, transparency, objectivity, completeness, and bias of GPT-4o mini responses to common oculoplastic patient questions.

The study was reviewed and deemed exempt from institutional review board approval. Thirty-three questions were developed across eight clinical domains: floppy eyelid syndrome, cicatricial ectropion, entropion, blepharoplasty, facial aging, brow lift, ptosis, and facial nerve function. Seven domains contained four patient-centered questions, while the facial aging domain contained five questions, for a total of 33 questions. On September 16, 2024, GPT-4o mini was prompted for each question with a standardized prompt requesting responses from the perspective of an expert oculofacial surgeon, using the latest medical guidelines. Verbatim responses were recorded. Six American Society of Ophthalmic Plastic and Reconstructive Surgery fellowship-trained surgeons independently evaluated each response using a 5-point Likert scale assessing clinical accuracy, completeness, transparency, objectivity, and bias. Readability was assessed using established readability indices.

GPT-4o mini demonstrated moderate clinical accuracy and completeness. Clinical accuracy was rated favorably in 51% of ratings (101/198), while completeness was rated favorably in 42% (84/198). Transparency and objectivity demonstrated stronger performance. 78% of ratings (155/198) agreed or strongly agreed that responses were transparent and 76% of ratings (150/198) agreed or strongly agreed that responses were objective. 8% of ratings (16/198) agreed or strongly agreed that Bias was present, while 65% (129/198) disagreed or strongly disagreed. Readability analyses demonstrated that ratings required college- to graduate-level reading ability.

GPT-4o mini generated transparent and objective ratings with low measured bias but demonstrated limitations in clinical accuracy, completeness, and readability. These findings apply specifically to GPT-4o mini and should not be generalized to other large language models. Future investigations should compare contemporary language models, evaluate alternative prompting strategies, and investigate multimodal systems that integrate text, clinical photographs, periocular anatomy, and patient symptoms.