Bridging the experience gap: reasoning-oriented LLMs as cognitive scaffolds for junior anesthesiologists-a multi-arm controlled study
Related Experiment Videos
Huiju Yang1, Guangyan Zhang1, Zhipeng Wang2
1Department of Anesthesiology, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China.
Background:
The gap between junior and senior physicians in anesthesia is exacerbated by the reliance on experience. Large language models like DeepSeek-R1 offer a potential solution as "cognitive scaffolding."
Methods:
A simulation-based, multi-arm controlled study with randomized junior groups and a non-randomized senior expert comparator was conducted. Forty-eight (48) anesthesiologists, including 16 senior experts and 32 junior residents, were recruited and randomly assigned (using a computer-generated randomization sequence) to two groups: Junior-alone and Junior + AI. Participants evaluated 20 standard clinical scenarios across four clinical domains: airway management, hemodynamic regulation, anesthetic pharmacology, and crisis management. The AI-assisted group followed a triple-step method: independent initial decision, subsequent AI consultation, and decision revision. Their final performance was compared against sixteen (16) senior expert anesthesiologists who served as the gold standard.
Results:
DeepSeek-R1 successfully improved performance in all four domains for junior doctors. Compared to the Junior-alone group, the Junior + AI group achieved significantly higher scores in accuracy (86.00 ± 16.61 vs 61.67 ± 19.45, P < 0.0001) and comprehensiveness (84.33 ± 17.18 vs 53.67 ± 23.27, P < 0.0001). However, further analysis revealed that while significantly improved, both accuracy and comprehensiveness remained lower than the senior expert scores (differences not statistically significant in mixed-effects analysis: P = 0.349 for Accuracy; P = 0.335 for Comprehensiveness). Performance in safety showed no significant difference to expert standards (P = 0.644). The AI-assisted group required slightly less total time (33.3 vs 33.8 min), showing no significant delay. 65% of AI suggestions were adopted overall.
Conclusion:
DeepSeek-R1 shows promise as "cognitive scaffolding", supporting knowledge externalization for junior anesthesiologists. It substantially improves decision-making, though a numerical gap with experts persists in certain dimensions. The high risk of automation bias in critical scenarios requires targeted training on critical evaluation of AI outputs.
