Determining the reliability of Artificial Intelligence programs in medical faculty board exams

Area of Science:

• Medical Education
• Artificial Intelligence
• Anatomy

Background:

• Generative AI applications are increasingly used in medical education.
• Evaluating AI reliability in assessing medical students' knowledge is crucial.

Purpose of the Study:

• To assess the accuracy of generative AI programs (Microsoft Copilot, Google Gemini, OpenAI ChatGPT) against first- and second-year medical students' performance on anatomy exams.
• To determine the reliability of AI tools in basic medical science education.

Main Methods:

• 286 anatomy questions from the 2023-2024 academic year were used.
• Questions were categorized by difficulty based on student performance.
• AI tools answered the same questions posed to students.

Main Results:

• Microsoft Copilot (97.7%), OpenAI ChatGPT (94.4%), and Google Gemini (86.5%) demonstrated higher accuracy than students.
• Gemini and ChatGPT showed performance similar to students on very difficult questions.
• Gemini's performance varied, excelling in clinical interpretation over basic knowledge questions.

Conclusions:

• While AI tools exceed student accuracy, they are not suitable for unsupervised use in anatomy education.
• AI lacks clinical reasoning and human experience, necessitating its use as a controlled, supplementary educational tool.