Checking the checklist: a content analysis of expert- and evidence-based case-specific checklist items

Area of Science:

• Medical Education
• Assessment in Healthcare
• Clinical Skills Evaluation

Background:

• Objective Structured Clinical Examinations (OSCEs) are widely researched, yet development of case-specific checklists for history taking and physical examination lacks detailed literature.
• Content validity of OSCE checklists relies on background information, often derived from expert panels.
• This study addresses the gap by comparing expert-generated checklist items with literature-derived, evidence-based items.

Purpose of the Study:

• To compare items on expert-based OSCE checklists with evidence-based items identified in medical literature.
• To evaluate the alignment between expert consensus and empirical evidence in OSCE checklist development.
• To identify discrepancies in content validity for history taking and physical examination components of OSCEs.

Main Methods:

• Literature search to identify evidence-based items for history taking and physical examination across various clinical problems.
• Evaluation of nine expert-based OSCE checklists by comparing their items against the identified evidence-based items.
• Categorization of items into expert-based, evidence-based, and evidence-based with relevance measures.

Main Results:

• A significant portion of expert-based items (26%) were not found in the literature.
• Over half (56%) of evidence-based items identified in the literature were absent from expert-based checklists.
• A substantial number of these missing evidence-based items (37%) included specific measures of diagnostic importance.

Conclusions:

• Expert-derived checklist items for OSCEs do not consistently match evidence-based literature findings.
• The observed inconsistency raises questions about the test validity of current OSCE checklists.
• Further research is recommended to understand the implications of this divergence for clinical assessment accuracy.