Expert panel consensus on assessment checklists for a rheumatology objective structured clinical examination

While several regional fellowship groups conduct rheumatology objective structured clinical examinations (ROSCEs), none have been validated for use across programs. We aimed to establish agreement among subspecialty experts regarding checklist items for several ROSCE stations.

We administered a 1-round survey to assess the importance of 173 assessment checklist items for 11 possible ROSCE stations. We e-mailed the survey to 127 rheumatology educators from across the US. Participants rated each item's importance on a 5-point Likert scale (1 = not important to 5 = very important). Consensus for high importance was predefined as a lower bound of the 95% confidence interval ≥4.0.

Twenty-five individuals (20%) completed the expert panel survey. A total of 133 of the 173 items (77%) met statistical cutoff for consensus to retain. Several items that had population means of ≥4.0 but did not meet the predetermined definition for consensus were rejected. The percentage of retained items for individual stations ranged from 24% to 100%; all items were retained for core elements of patient counseling and radiograph interpretation tasks. Only 24% of items were retained for a rehabilitation medicine station and 60% for a microscope use/synovial fluid analysis station.

This single-round expert panel survey established consensus on 133 items to assess on 11 proposed ROSCE stations. The method used in this study, which can engage a diverse geographic representation and employs rigorous statistical methods to establish checklist content agreement, can be used in any medical field.