An Auxiliary Tasks Based Framework for Automated Medical Skill Assessment with Limited Data.

Area of Science:

• Medical Education
• Machine Learning
• Computer-Assisted Assessment

Background:

• Automated medical skill assessment standardizes training by integrating diverse clinical experiences.
• Limited medical dataset sizes hinder the development of robust automated assessment systems due to data collection costs, safety, and privacy concerns.
• Current training relies on evaluation rubrics with auxiliary labels for comprehensive assessment.

Purpose of the Study:

• To develop a generalizable auxiliary task-based framework for medical skill assessment using machine learning.
• To address the challenge of training automated systems with limited medical data.
• To leverage auxiliary information from evaluation rubrics for effective pre-training.

Main Methods:

• Exploration of machine learning algorithms to create an auxiliary task-based framework.
• Exhaustive mining of valid auxiliary information within evaluation rubrics for pre-training feature extractors.
• Implementation of a novel regression-based multitask weighting method for comprehensive feature representation pre-training.
• Fine-tuning the overall skill assessment classifier using the pre-trained feature representation.

Main Results:

• The framework significantly improves performance on medical skill datasets.
• Achieved 85.9% accuracy on an intubation dataset.
• Achieved 97.4% accuracy on a surgical skill dataset.

Conclusions:

• The proposed auxiliary task-based framework effectively addresses the challenge of limited data in automated medical skill assessment.
• Pre-training feature extractors using rubric-based auxiliary information enhances model generalizability and performance.
• The regression-based multitask weighting method is crucial for creating meaningful feature representations that accurately imitate evaluation rubrics.