The script concordance test for clinical reasoning: re-examining its utility and potential weakness

Area of Science:

• Medical Education
• Clinical Reasoning Assessment

Background:

• The Script Concordance Test (SCT) assesses clinical reasoning skills in uncertain situations.
• Limited data exists on Z-score cut-offs for differentiating trainee experience levels and the influence of factual knowledge on SCT scores.
• A known weakness of the SCT is that avoiding extreme responses can artificially inflate scores.

Purpose of the Study:

• Determine optimal Z-score cut-offs for distinguishing between residents, fellows, and attending physicians.
• Correlate SCT scores with medical knowledge examination (MKE) scores.
• Investigate methods to address the SCT's vulnerability to extreme response avoidance.

Main Methods:

• Analysis of SCT and MKE scores from pulmonary and critical care medicine trainees (residents and fellows) and attending physicians.
• SCTs (Question Sets 1 or 2) assessed clinical reasoning; a computer-adapted MKE assessed medical knowledge.
• Statistical analysis to identify Z-score cut-offs and correlations, and to evaluate strategies for mitigating response bias.

Main Results:

• SCT reliability was acceptable (Cronbach's alphas 0.79-0.89).
• Z-scores of -2.91 and -1.76 effectively separated residents from fellows and fellows from attendings, respectively.
• SCT and MKE scores showed poor correlation; avoiding extreme answers increased lowest scorers' Z-scores by at least 1 SD.
• Increasing questions with extreme modal answers to 50% and using hypothetical sets mitigated the bias, but consensus scoring did not.

Conclusions:

• The SCT successfully differentiates trainees by experience level, independent of medical knowledge.
• The test is susceptible to candidates exploiting response strategies to avoid extreme answers.
• Modifying the SCT by increasing the proportion of questions with extreme modal answers can reduce the impact of this test weakness.