Responses of ChatGPT-4 on intraocular lenses: an evolving artificial intelligence assessment

Recently, studies have been carried out on the accuracy of large language models, which are increasingly used, but the reproducibility of the answers given by these language models is as important as their accuracy.

To evaluate the accuracy and reproducibility of the responses of ChatGPT-4.0 to queries about intraocular lenses (IOLs) over a six-month interval.

This observational, cross-sectional study was conducted using 45 open-ended questions related to IOLs, categorised into three difficulty levels: easy, moderate, and difficult. Two evaluators independently posed these questions to ChatGPT-4.0 at two separate time points (April and October 2024) using the 'new conversation' feature. Responses were analysed for word count, character count, and plagiarism rates, and evaluated on a 5-point Likert scale for accuracy, relevance, and consistency.

A total of 90 responses were analysed. Accuracy rates were 93.3%, 86.7%, and 80% for easy, moderate, and difficult questions, respectively. Significant increases in word and character counts were observed between the initial and final evaluations (p < 0.0001). Likert scores improved significantly over time (p = 0.046). ICC analysis showed moderate consistency (0.639) overall, with excellent reproducibility (1.00) for easy questions, acceptable (0.788) for moderate, and moderate (0.550) for difficult questions.

ChatGPT-4.0 demonstrated significant improvements in accuracy and response length over time. While the model exhibited high reproducibility for easy questions, its performance declined with increased question difficulty. Despite its limitations, ChatGPT-4.0 shows promise for use in medical information dissemination, highlighting the need for further studies to enhance its reliability and accuracy.