Calibration Bias and the Interpretation of Clinical Learning Environment Perceptions Surveys

Area of Science:

• Medical Education Research
• Health Professions Education
• Quantitative Analysis

Background:

• Clinical learning environments (CLEs) are often evaluated using perception surveys.
• Existing surveys may capture subjective factors beyond direct trainee experiences.
• This subjectivity is termed 'calibration bias'.

Purpose of the Study:

• To assess calibration bias in health professions trainees' perceptions of the CLE.
• To determine how calibration bias varies across different health professions disciplines.
• To identify program, patient-mix, and trainee factors associated with calibration bias.

Main Methods:

• Utilized 2011-2017 US Department of Veterans Affairs (VA) Learners' Perceptions Survey data.
• Compared perception data from medical students and physician residents/fellows (n=32,830) with nursing (n=29,758) and allied/associated health trainees (n=27,092).
• Measured calibration bias using statistical analysis, including odds ratios (OR) and confidence intervals (CI).

Main Results:

• Nursing (OR 1.31) and allied/associated health trainees (OR 1.18) were more likely than physician trainees to overrate their CLE experiences.
• Overrating CLEs was associated with higher reported psychological safety, more time in CLE, female gender, lower academic level, and specific patient characteristics (lack of social support, low income, addictions, mental illness).
• Specific patient factors associated with overrating included lack of social support (OR 1.16), low income (OR 1.05), co-occurring addictions (OR 1.06), and mental illness (OR 1.06).

Conclusions:

• Calibration bias is a significant factor influencing perception survey scores in health professions education.
• Accounting for calibration bias is crucial for accurately interpreting physician trainee experiences and CLE quality.
• Understanding these biases enhances the assessment of complex clinical learning environments.