Enabling micro-assessments of skills in the simulated setting using temporal artificial intelligence-models.

Area of Science:

• Medical Simulation
• Artificial Intelligence in Medical Education
• Ultrasound Training

Background:

• Skill assessment in simulated environments is costly and lacks validated metrics.
• Artificial intelligence (AI) presents an opportunity for automated competence evaluation.
• Current limitations necessitate novel approaches for objective skill assessment in medical training.

Purpose of the Study:

• To develop and validate a machine learning AI model for automated competence assessment in simulation-based thyroid ultrasound (US) training.
• To evaluate the AI model's ability to differentiate between expert and novice performance.
• To assess the real-time performance and interpretability of AI-driven skill evaluation.

Main Methods:

• Video analysis of thyroid US procedures performed by experts and novices on a simulator.
• Utilizing a convolutional neural network (CNN) with ResNet-50 and a long short-term memory (LSTM) layer for sequence analysis.
• Employing fourfold cross-validation and performance metrics (precision, recall, F1, accuracy) for model validation.

Main Results:

• The AI model successfully distinguished between expert and novice US performance.
• Optimal performance was achieved using 50-second video sequences, yielding 70% accuracy and a 0.76 F1 score.
• Experts demonstrated significantly longer durations above the competence threshold (15.71s) compared to novices (9.31s).

Conclusions:

• An LSTM-based AI model offers near real-time, automated competence assessment for US training.
• The use of temporal video data allows for detailed micro-assessments of complex procedural skills.
• This AI approach holds potential for enhancing interpretability and applicability across diverse procedural training domains.