Blinded evaluation of GPT-4 and physician responses to patient inquiries across multiple specialties

To evaluate the quality of responses generated by GPT-4 in comparison to those written by hospital specialists and general practice physicians across multiple medical specialties. The goal was to assess whether large language models (LLMs) can support patient communication by providing accurate, useful, complete, and empathetic responses to real-world patient inquiries.

We collected 100 anonymized patient questions from a public online health forum covering five specialties: cardiology, infectious diseases, neurology, gynecology, and gastroenterology. Responses were generated by GPT-4, 50 hospital-based specialists, and 50 general practice physicians. A group of 50 specialists, blinded to response source, evaluated each answer using four 7-point Likert scales: accuracy, usefulness, completeness, and empathy. The study design and reporting were informed by the CONSORT-AI Extension to promote transparency in AI evaluation.

The model received significantly higher ratings across all four categories compared to both physician groups. It was ranked best in 67% of evaluations, particularly outperforming physicians in completeness and empathy. While response length correlated positively with quality for physicians, model's longer responses were less useful when overly detailed. The model consistently produced longer, more comprehensive replies than human groups.

The model's strong performance in completeness and empathy highlights its potential role in enhancing patient communication. Although it matched or exceeded physicians in accuracy and usefulness, caution is warranted due to risks like hallucinations and lack of true understanding.

This blinded evaluation suggests that AI-generated responses may support clinical practice by delivering accurate, comprehensive, and empathetic information for patient communication.