Poorly performing physicians: does the Script Concordance Test detect bad clinical reasoning?

Area of Science:

• Medical Education
• Physician Assessment
• Clinical Reasoning

Background:

• Physician performance evaluation is a global concern for medical licensing bodies.
• The Collège des Médecins du Québec uses the Structured Oral Interview (SOI) to assess physician competence.
• There is a need for objective measures of clinical reasoning process (CRP) for remediation and legal purposes.

Purpose of the Study:

• To evaluate the Script Concordance Test (SCT) as a tool for measuring clinical data interpretation (CDI) in physicians.
• To determine if SCT can provide standardized and objective data on CRP.
• To assess the utility of SCT in the evaluation of physicians with potential clinical competence difficulties.

Main Methods:

• A 2-year study involving 20 family physicians who completed both the SOI and a 1-hour SCT.
• Three blinded evaluators retrospectively assessed SOI reports for clinical reasoning quality.
• Physicians were grouped based on median scores for both assessment methods, with agreement analyzed using the Kappa coefficient.

Main Results:

• High reliability was found for both SOI (Intraclass correlation: 0.89) and SCT (Cronbach alpha: 0.90).
• Agreement between SOI and SCT classifications was moderate (Kappa: 0.30), with 7 of 20 physicians classified differently.
• Most participants (19/20) scored below the panel mean on SCT, indicating significant deficiencies in CDI.

Conclusions:

• The majority of referred physicians demonstrated significant weaknesses in clinical data interpretation (CDI).
• The SCT is a useful adjunct to the SOI for assessing cognitive reasoning in underperforming physicians.
• Improvements in SOI reporting and cautious interpretation of SCT scores are recommended.