Benchmarking ChatGPT-generated multiple-choice questions against faculty-authored items in dental education
Amber Kiyani1, Fariha Hanif2, Muhammad Muhammad2
1Qatar University, Doha, Qatar. a.kiyani@qu.edu.qa.
Scientific Reports
|December 29, 2025
Summary
This study compared ChatGPT-generated multiple-choice questions (MCQs) with instructor-written ones for dental students. Instructor items performed better, but AI-generated questions show future potential in educational assessments.
Area of Science:
- Educational Technology
- Artificial Intelligence in Education
- Medical Education
Background:
- Large language models (LLMs) are increasingly used in self-directed student learning.
- Assessing the quality of AI-generated educational content is crucial for effective integration.
- Benchmarking AI tools against traditional methods provides insights into their utility.
Purpose of the Study:
- To compare the psychometric properties of assessment items generated by ChatGPT-3.5 against those created by faculty instructors.
- To evaluate the performance of AI-generated multiple-choice questions (MCQs) in an Oral Medicine context for undergraduate dental students.
- To analyze the reliability, difficulty, and discrimination of both AI-generated and instructor-created assessment items.
Main Methods:
- A 40-item Oral Medicine assessment was developed, comprising 20 MCQs by instructors and 20 by ChatGPT-3.5.
- The assessment was administered to 547 undergraduate dental students across four institutions.
- Item Response Theory, specifically the 3-Parameter Logistic model, was used for data analysis.
Main Results:
- Instructor items demonstrated higher person reliability (0.845) compared to ChatGPT items (0.778).
- Instructor items showed a wider difficulty range (-1.29 to 3.22) and higher discrimination (0.50 to 4.64) than ChatGPT items.
- While instructor items had higher average scores and variability, ChatGPT items exhibited promising psychometric properties with a low guessing parameter.
Conclusions:
- Faculty-generated assessment items currently exhibit superior psychometric properties, including better discrimination and difficulty range.
- ChatGPT-generated items show potential for MCQ generation in educational settings, warranting further investigation.
- LLMs like ChatGPT may play a significant role in future educational assessment development, potentially surpassing human capabilities.
More Related Videos
13:44Project-Based Learning Guidelines for Health Sciences Students: An Analysis with Data Mining and Qualitative Techniques
Published on: December 9, 2022
4.1K
07:42Quasistatic Mechanical Testing for Computer-Aided Design and Manufacturing Occlusal Veneers Cemented to Milled Dentin Analog Material
Published on: December 20, 2024
683
Related Concept Videos
01:26Assessment of the Mouth
962
A thorough mouth assessment, including inspection and palpation of the lips, gums, tongue, tonsils, uvula, and pharynx, is crucial in detecting potential health issues. Diseases ranging from oral cancer to systemic conditions like diabetes could be identified early through careful oral examination. This article provides a detailed guide on conducting a comprehensive mouth assessment.
Mouth Inspection
The inspection begins with visually examining the mouth for symmetry, color, and size.
Mouth Inspection
The inspection begins with visually examining the mouth for symmetry, color, and size.
962
02:16Surveys
16.6K
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
16.6K
01:20Biostatistics: Overview
687
Biostatistics plays a crucial role in understanding and analyzing data in healthcare and biology. Biostatisticians conduct experiments, gather evidence, and draw meaningful conclusions using statistical methods and techniques. Different variables form the foundation of biostatistical analysis, allowing researchers to understand and interpret data effectively. These variables are classified into different types, each serving a specific purpose in statistical analysis.
Discrete variables are...
Discrete variables are...
687
01:08The Availability Heuristic
6.9K
A heuristic is a general problem-solving framework (Tversky & Kahneman, 1974). You can think of these as mental shortcuts that are used to solve problems. Different types of heuristics are used in different types of situations, and the impulse to use a heuristic occurs when one of five conditions is met (Pratkanis, 1989):
6.9K
01:09Comparing Experimental Results: Student's t-Test
4.7K
The t-test is a statistical method used to compare the sample mean with a population mean or compare two means from two data sets. The test statistic is calculated from the standard deviation, mean, and number of measurements in the data set at a selected confidence interval and then compared to a table of critical values at this confidence level. If the test statistic is smaller than the critical value, the null hypothesis is accepted. In this case, we state that the difference between the...
4.7K
01:10Pre-Procedural Guidelines for Assessing Blood Pressure
788
Accurate blood pressure assessment is crucial for diagnosing and managing various health conditions. To ensure the reliability of these measurements, healthcare professionals must adhere to standardized pre-procedural guidelines. These guidelines enhance patient safety and improve the overall quality of healthcare. The following steps are essential for obtaining accurate and consistent blood pressure readings, from using the appropriate tools to ensuring effective communication with the...
788