The objective structured public health examination: a study of reliability using multi-level analysis.

Area of Science:

• Medical Education
• Public Health Assessment
• Psychometrics

Background:

• The Objective Structured Public Health Examination (OSPHE) was introduced in 2006 as part of the UK Membership of the Faculty of Public Health examination.
• The OSPHE simulates real-world public health practice, assessing five generic competencies across six stations within a 1 hour 48 minute timeframe.
• This study aimed to estimate the examination reliability of the OSPHE.

Purpose of the Study:

• To estimate the examination reliability of the Objective Structured Public Health Examination (OSPHE).
• To identify sources of error within the OSPHE using advanced statistical methods.
• To compare the OSPHE's reliability with existing literature and assessment methodologies.

Main Methods:

• Analysis of data from 198 candidates, 54 questions, and 39 examiners over 9 test days in 2006.
• Utilized multilevel cross-classified analysis and generalisability theory to assess reliability and error sources.
• Employed multilevel modeling for a more rigorous reliability estimation compared to traditional methods like Cronbach's alpha.

Main Results:

• An estimated reliability coefficient of 0.85 (95% CI: 0.82-0.88) was achieved for the OSPHE.
• Minimal variance was attributed to specific questions or examiners, indicating consistent assessment.
• Increasing the number of questions to twelve could enhance reliability to 0.92; multiple examiners per question had negligible impact.

Conclusions:

• Multilevel modeling provides a more rigorous estimation of examination reliability than Cronbach's alpha or traditional ANOVA-based generalisability methods.
• The OSPHE exhibits favorable reliability compared to other assessments, despite its concise format (six stations, under 2 hours).
• The OSPHE's parallel testing approach, assessing key competencies across all stations, could be adopted by other medical specialties.