Assessment of Large Language Model Performance on Medical School Essay-Style Concept Appraisal Questions: Exploratory...

Area of Science:

• Medical Education
• Artificial Intelligence

Background:

• The integration of artificial intelligence (AI) into academic settings presents new challenges and opportunities.
• Large language models (LLMs) like ChatGPT 4.0 are increasingly capable of generating human-like text.

Purpose of the Study:

• To evaluate the performance of Microsoft Copilot (based on ChatGPT 4.0) against medical students in essay-style concept appraisals.
• To assess the ability of human evaluators to differentiate AI-generated responses from those written by students.

Main Methods:

• An essay-style concept appraisal task was administered.
• Responses were evaluated by human assessors, who were unaware of the origin of the text (AI vs. human).

Main Results:

• Microsoft Copilot demonstrated performance comparable to that of medical students.
• Assessors experienced significant difficulty in distinguishing between AI-generated and human-written responses.

Conclusions:

• AI tools like Microsoft Copilot show advanced capabilities in academic tasks, rivaling human performance.
• Educational institutions must adapt to the rise of AI by emphasizing critical thinking and reflective learning for both students and educators.