Full Resolution Simulation for Evaluation of Critical Care Imaging Interpretation; Part 2: Random Effects Reveal the...

Area of Science:

• Medical Imaging Analysis
• Radiology Education Research
• Statistical Modeling in Healthcare

Background:

• Accurate interpretation of critical care imaging is vital for patient outcomes.
• Evaluating resident performance in radiology requires robust and scalable methods.
• Previous studies lacked comprehensive models to dissect sources of score variation.

Purpose of the Study:

• To characterize empirical data from simulated critical care imaging interpretations.
• To estimate random effects in a hierarchical regression model for radiology interpretation scores.
• To quantify sources of variation including case difficulty and resident competence.

Main Methods:

• Utilized a full-resolution simulation of critical care imaging.
• Employed a comprehensive mixed (hierarchical) regression model.
• Estimated normally distributed random effects for case, resident, program, and grader.

Main Results:

• The mixed model explained 36% of score variation, a significant increase from fixed effects alone (8.8%).
• Case difficulty was the largest source of variation (28.5% ICC), with 95% reliability.
• Resident competence accounted for substantial remaining variation (5.3% ICC), with 82% reliability.

Conclusions:

• Large-scale, remote simulation of critical care radiology interpretation is feasible and efficient.
• A comprehensive mixed model reliably quantifies "case difficulty" and "resident competence" in radiology assessments.
• The findings support the use of simulation and mixed models for objective evaluation in radiology training.