Some assembly required: tracing the interpretative work of Clinical Competency Committees

Area of Science:

• Medical Education
• Qualitative Research
• Assessment and Evaluation

Background:

• Clinical Competency Committees (CCCs) are crucial for evaluating postgraduate trainees.
• These committees rely on assessment data to make informed decisions about trainee progression.
• The interpretation of assessment data can be complex and subjective.

Purpose of the Study:

• To describe the social processes of evidence interpretation used by CCCs.
• To understand how CCCs grapple with and weigh assessment data.
• To explicate the challenges and strategies in interpreting learner data.

Main Methods:

• Qualitative study using constructivist grounded theory methodology.
• Observation of 10 CCC meetings (over 25 hours) across four postgraduate programs.
• Semi-structured interviews with CCC members post-meeting, with iterative data collection and analysis.

Main Results:

• CCCs assumed high-quality data but frequently encountered 'problematic evidence' (difficult to interpret).
• Confronted with problematic evidence, CCCs engaged in lengthy, effortful discussions.
• Contextual data aided interpretation, enabling decisions informed by, rather than ignoring, problematic evidence.

Conclusions:

• Groups reviewing trainee data must anticipate and discuss strategies for interpreting uncertain or incomplete evidence.
• Addressing challenges in data interpretation requires grappling with subjective assessment data and interpretive frameworks.
• The solution is not solely generating more psychometrically sound data, but understanding the interpretive process.