Related Experiment Video
Updated: Jun 18, 2026
Development of a Virtual Reality Assessment of Everyday Living Skills
Published on: April 23, 2014
Construct Validity of a Multimodal Assessment Framework Combining Script Concordance Testing and High-Fidelity
Ming Zhou1, Gang Zhao1, Yunxia Zhou2
1Department of Orthopaedics, Wuxi Ninth People's Hospital Affiliated to Soochow University, Wuxi, China.
Objective:
To develop and evaluate a multimodal assessment framework integrating the script concordance test (SCT) and high-fidelity simulation to provide a holistic evaluation of orthopedic resident competence.
Design:
A prospective, cross-sectional validation study.
Setting:
The study was conducted in a simulation center at an academic teaching hospital. Residents underwent a multimodal evaluation consisting of a 28-item SCT focusing on clinical decision-making and 3 high-fidelity simulation scenarios (acute hip fracture, degenerative spine disease, and knee osteoarthritis). The SCT was administered 1 week before simulation in a closed-book electronic format to establish baseline cognitive performance. Simulation performance was evaluated using decision time, error rates based on missed critical actions, and communication/teamwork scores using a modified Global Rating Scale (GRS).
Participants:
Thirty orthopedic surgery residents (15 PGY-1, 15 PGY-2) and a reference panel of 12 board-certified expert surgeons.
Results:
The SCT demonstrated acceptable internal consistency (Cronbach's α = 0.662) and strong discriminative validity, with experts significantly outperforming residents (mean Z-score: +1.05 vs -0.42; p < 0.001; Cohen's d = 2.32). Within the resident cohort, PGY-2 residents achieved significantly higher SCT scores than PGY-1 residents (mean Z: 0.13 vs -0.97; p < 0.001), indicating a marked developmental difference in clinical reasoning between PGY-1 and PGY-2 residents (d = 1.61). In high-fidelity simulation, PGY-2 residents demonstrated significantly faster decision times (12.61 vs 13.58 min; p = 0.027) and lower error rates (4.53% vs 6.27%; p = 0.025) compared to PGY-1 residents. Spearman correlation analysis revealed a significant moderate-to-strong predictive relationship between SCT scores and simulation metrics (all ∣ρ∣ ≥ 0.533, p < 0.01), indicating that cognitive script organization is a key determinant of behavioral performance. Overall participant satisfaction was high (8.57/10).
Conclusion:
This multimodal framework is a feasible and valid approach for assessing orthopedic competence. The significant correlation between cognitive reasoning and simulation performance suggests that the SCT may serve as a useful predictor of performance in simulated orthopedic scenarios, offering educators a comprehensive tool to evaluate surgical readiness and guide targeted training interventions.