ChatGPT Generated Otorhinolaryngology Multiple-Choice Questions: Quality, Psychometric Properties, and Suitability
Cecilia Lotto1,2,3, Sean C Sheppard1, Wilma Anschuetz4
1Department of Otorhinolaryngology, Head and Neck Surgery Inselspital, Bern University Hospital, University of Bern Bern Switzerland.
OTO Open
|September 27, 2024
Summary
ChatGPT can generate multiple-choice questions for otorhinolaryngology (ORL) education, but quality varies. Expert review is essential for medical exam suitability.
Area of Science:
- Medical Education
- Artificial Intelligence in Healthcare
- Otorhinolaryngology
Background:
- Artificial intelligence tools like ChatGPT are increasingly explored for educational applications.
- Developing high-quality assessment tools is crucial for medical training in specialized fields like otorhinolaryngology (ORL).
Purpose of the Study:
- To evaluate the capability of ChatGPT 3.5 in generating multiple-choice questions (MCQs) for otorhinolaryngology (ORL).
- To assess the quality, difficulty, and internal consistency of AI-generated ORL MCQs.
Main Methods:
- ChatGPT 3.5 was prompted to create 20 clinical case-based MCQs in ORL.
- Generated questions were reviewed for quality criteria by medical students, residents, and consultants.
- Data analysis included question difficulty, relevance, and internal consistency (Cronbach's alpha).
Main Results:
- ChatGPT generated 20 ORL MCQs; 1 was off-topic, 3 had incorrect answers, and 3 had multiple correct answers.
- Question quality varied: focus and relevance were good, but vignette and distractor quality were low.
- The difficulty was appropriate for medical students but too easy for residents and consultants; Cronbach's alpha was 0.69 for 15 selected questions using student data.
Conclusions:
- ChatGPT 3.5 can produce grammatically correct ORL MCQs suitable for medical students.
- The overall quality of AI-generated MCQs is average and requires significant expert revision for use in formal medical examinations.
Related Concept Videos
01:17Cochran's Q Test
242
Cochran's Q Test is a nonparametric statistical test used to determine if there are potential differences in the outcomes of three or more related groups on a binary (yes/no) or dichotomous outcome. It is essentially an extension of the McNemar Test, which is limited to two related samples - Cochran's Q test can handle three or more related samples, making it more versatile in scenarios where subjects are measured under multiple conditions. The test statistic follows a Chi-Square...
242
01:22Self-Report Tests of Personality
325
Self-report inventories are objective personality assessments that use multiple-choice items or numbered scales, typically ranging from 1 (strongly disagree) to 5 (strongly agree). They are often called Likert scales after Rensis Likert. These inventories are widely used due to their ease of administration and cost-effectiveness. One of the most prominent examples is the Minnesota Multiphasic Personality Inventory (MMPI), initially developed in the 1940s to assess abnormal personality traits.
325
01:29Assessment of the Gastrointestinal System II: Health Perception Pattern
81
Assessing the gastrointestinal (GI) system is a complex process that begins with collecting subjective data. This data, collected through patient interviews, provides crucial insights into the patient's health history, perception patterns, and lifestyle habits, all contributing significantly to GI health.
Health Perception Patterns
Health perception patterns offer valuable insights into a patient's lifestyle habits and how they may impact their GI health. These patterns include:
Health Perception Patterns
Health perception patterns offer valuable insights into a patient's lifestyle habits and how they may impact their GI health. These patterns include:
81
01:29Reliability and Validity
12.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
12.7K
01:13Sensitivity, Specificity, and Predicted Value
213
In healthcare diagnostics, laboratory tests play a crucial role in identifying and diagnosing a wide range of medical conditions. However, interpreting test results is not always straightforward. An abnormal test result does not always confirm the presence of a disease, just as a normal result does not guarantee its absence. To assess the reliability of these diagnostic tools, healthcare practitioners rely on two key statistical indicators: sensitivity and specificity.
Sensitivity is the...
Sensitivity is the...
213