External validation of an artificial intelligence multi-label deep learning model capable of ankle fracture...

Area of Science:

• Orthopedic surgery
• Radiology
• Artificial Intelligence in Medicine

Background:

• Medical imaging and AI enable ankle fracture classification.
• Machine learning models show good internal validity for AO/OTA classification.
• External validation of AI models is crucial for clinical adoption.

Purpose of the Study:

• To externally validate an AI model for ankle fracture classification using the AO/OTA system.
• To assess the model's performance on a dataset from a different institution and time period.
• To identify methods for improving the external validity of AI models in fracture classification.

Main Methods:

• A deep-learning neural network was trained on 7,500 ankle studies for AO/OTA fracture classification.
• Internal validation used 409 studies from Sweden (2002-2016).
• External validation utilized 399 studies from Australia (2016-2020).
• Primary outcomes included area under the receiver operating characteristic (AUC) and area under the precision-recall curve (AUPR).

Main Results:

• The AI model achieved a weighted mean AUC of 0.95 (internal) and 0.86 (external).
• Area under the precision-recall curve was 0.96 (internal) and 0.93 (external).
• Performance metrics for specific fracture subtypes (44A-C) were 0.93 (internal) and 0.82 (external).

Conclusions:

• The AI model demonstrated good transferability to an external dataset with different characteristics.
• Local validation of AI algorithms is essential before clinical implementation.
• Targeted retraining can enhance AI model performance and support objective fracture treatment recommendations.