Experts' responses in script concordance tests: a response process validity investigation

Area of Science:

• Medical Education
• Clinical Reasoning Assessment

Background:

• The script concordance test (SCT) is used to assess clinical reasoning but lacks response process validity evidence.
• Divergent expert beliefs on how clinical data influence proposed actions in simulated cases require investigation.

Purpose of the Study:

• To explore the response processes of experts on an SCT scoring panel.
• To understand experts' reasoning regarding new clinical data and proposed actions in simulated patient cases.

Main Methods:

• 10 Argentine gastroenterologists re-answered 15 SCT cases after 9 months.
• Experts provided reasoning and reacted to other experts' perspectives.

Main Results:

• Experts often disregarded proposed actions or sought more data (20% and 36% respectively).
• Significant disagreement existed on action suitability before new data was introduced (89%).
• Expert responses showed limited consistency over time (kappa = 0.33) and were influenced by local practice environments.

Conclusions:

• Expert response processes challenge the classical interpretation of SCT scores.
• Alternative applications for SCT-like methods, such as formative learning assessments, are proposed.