Auditing RadExam: Employing Psychometrics to Improve Exam Quality.
Elana B Smith1, Petra Lewis1, Thad Benefield1
1University of Maryland Medical Center, Department of Radiology, R. Adams Cowley Shock Trauma Center, 22 S. Greene St., Baltimore, MD 21201.
Academic Radiology
|July 18, 2020
Summary
Regular audits of the RadExam database significantly improved question and exam performance. This quality improvement initiative reduced the number of poorly performing radiology exam questions by over 49% in the second audit.
Area of Science:
- Medical Education
- Radiology Assessment
- Psychometrics
Background:
- RadExam is a radiology resident assessment database developed by the Association of Program Directors in Radiology and the American College of Radiology.
- It provides performance metrics benchmarked against institutional and national resident performance.
- Despite quality control measures, some exam questions were anticipated to perform poorly.
Purpose of the Study:
- To identify and address poorly performing questions within the RadExam database.
- To assess the impact of an audit system on question and exam performance.
Main Methods:
- Two audits were conducted on the RadExam database, evaluating exam questions and exams using statistical data.
- Data analyzed included question and test administrations, p-values, Discrimination Index (DI), Bloom's taxonomy level, and image-based question counts.
- Poorly performing questions were modified or replaced based on audit findings.
Main Results:
- Audit 1 identified 617 questions with DI <0.1 and 565 with unacceptable P-values.
- Audit 2 identified 229 questions with DI <0.1 and 290 with unacceptable P-values.
- A 49.1% decrease in total flagged questions was observed between Audit 1 and Audit 2, with statistically significant improvements across subspecialties.
Conclusions:
- The audit system demonstrated a positive impact on question and exam performance.
- A significant decrease in flagged questions and improved overall exam performance were noted in Audit 2.
- The findings illustrate the effectiveness of the audit system in enhancing the quality of the RadExam database.
Related Concept Videos
01:16Response Surface Methodology
483
Response Surface Methodology (RSM) is a collection of statistical and mathematical techniques used to develop, improve, and optimize processes. It is particularly valuable when many input variables or factors potentially influence a response variable.
The process of RSM involves several key steps:
The process of RSM involves several key steps:
483
01:52Regression Toward the Mean
6.7K
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.7K
01:25Assessment of apical radial pulse
1.1K
Apical-Radial (A-R) Pulse Assessment
The A-R pulse assessment involves simultaneous evaluation of the apical and radial pulses. When the apical and radial pulse rates vary, this assessment helps identify a pulse deficit.
Pre-Procedural Preparation
The A-R pulse assessment involves simultaneous evaluation of the apical and radial pulses. When the apical and radial pulse rates vary, this assessment helps identify a pulse deficit.
Pre-Procedural Preparation
1.1K