Learning the Randleman Criteria in Refractive Surgery: Utilizing ChatGPT-3.5 Versus Internet Search Engine

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Information Retrieval

Background:

• Large language models (LLMs) show promise for self-directed medical learning.
• Concerns exist regarding the accuracy of LLMs, such as ChatGPT-3.5, in specialized medical fields.
• The Randleman criteria require precise definition for accurate medical application.

Purpose of the Study:

• To compare the accuracy and reliability of ChatGPT-3.5 versus internet search engines.
• To evaluate the efficacy of these tools in defining the Randleman criteria within a self-directed learning context.
• To assess the efficiency and truthfulness of information provided by ChatGPT-3.5 and internet searches.

Main Methods:

• Twenty-three medical students used ChatGPT-3.5 for 10 minutes, then conducted independent internet searches for 10 minutes.
• Information gathered on the Randleman criteria from both sources was analyzed.
• Analyses focused on accuracy, efficiency, and reliability of the definitions and parameters provided.

Main Results:

• Internet searches provided correct Randleman criteria definitions for 100% of students.
• ChatGPT-3.5 correctly defined the criteria for only 26.1% of students.
• ChatGPT-3.5 generated incorrect definitions (17.4%), fabricated information (4.3%), and failed to define criteria for 52.2% of students.

Conclusions:

• Internet search engines are superior to ChatGPT-3.5 for accessing accurate and reliable medical information like the Randleman criteria.
• ChatGPT-3.5 demonstrated significant inaccuracies, including fabricated data and incomplete definitions, hindering self-directed learning.
• Medical learners must exercise caution and critical judgment when utilizing LLMs; further research into prompt engineering and LLM updates is warranted.