Navigating digital surgical advice: evaluating the reliability of large language models for pediatric phimosis

Phimosis is an extremely common pediatric surgical condition. Given the convenience and accessibility brought by artificial intelligence, the readability, quality and accuracy evaluation on the information provided by large language models (LLM) is of utmost importance in providing comprehensive clinical care.

Six case questions relating to phimosis on its definition, diagnosis, treatment options and potential complications were inputted into each LLM-Chatbots (ChatGPT-3.5, Gemini-1.5 and Claude-3). All responses were reformatted, shuffled and recombined into three passages for single-blind scoring on accuracy by three pediatric surgeons with reference to European Association of Urology (EAU) pediatric urology guidelines. The responses were also assessed for readability using the Gunning-Fog score, Simple Measure of Gobbledygook (SMOG) index and Dale-Chall score and for quality using the DISCERN score.

Comprehension of the information was considered difficult with a mean Gunning-Fog score, SMOG index score and Dale-Chall score of 14.2 $\pm$ 2.6, 11.3 $\pm$ 2.2 and 9.9 $\pm$ 0.9, respectively. The mean summed DISCERN score was 39.6 $\pm$ 2 out of 80. Claude-3-sonnet was considered of poor quality while others were fair. The mean accuracy score of the LLM-chatbots was 4.4 $\pm$ 0.4 out of 5. The three LLMs performed similarly on readability and accuracy. The LLM-chatbots generally scored well in providing treatment options and their mechanisms, while lacking references and information regarding treatment benefits, risks of alternative treatments and impact on quality of life.

The three LLM-chatbots showed comparable and satisfactory accuracy in providing information on pediatric phimosis; however, their readability and overall quality were suboptimal. These limitations may affect parent counseling, shared decision-making and informed consent. LLM-generated information should, therefore, supplement, not replace, clinician guidance and reliable medical resources.

III.