Comparative analysis of machine learning algorithms for computer-assisted reporting based on fully automated...

Area of Science:

• Radiology and Medical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing in Medicine

Background:

• Computer-assisted reporting (CAR) tools aim to enhance radiology report quality by recommending imaging biomarkers.
• Existing CAR tools often lack robust evaluation of machine learning (ML) algorithms using cross-lingual ontological mappings.
• Development of embedded CAR algorithms requires efficient methods for feature extraction and model training.

Purpose of the Study:

• To evaluate ML algorithms for CAR tools using automated, cross-lingual RadLex mappings compared to human-annotated features.
• To assess the feasibility of ML-based contextual assistance for recommending specific clinical scores like the Alberta Stroke Programme Early CT Score (ASPECTS).
• To provide guidance on selecting appropriate ML classifiers for CAR development in a language-agnostic manner.

Main Methods:

• Comparison of ML algorithms (tree-based methods, elastic net, SVMs, fastText) trained on human-annotated vs. automated German-to-English RadLex mappings.
• Utilized 206 CT reports for suspected stroke, with ASPECTS recommendation as the target label.
• Employed a 5x5-fold nested cross-validation framework for model evaluation using calibration metrics (AUC, Brier score, log loss) and plots.

Main Results:

• ML-based contextual assistance for recommending ASPECTS is feasible.
• Support Vector Machines (SVMs) achieved high accuracies (87% on human features, 85.4% on RadLex impressions).
• FastText demonstrated top performance (89.3% accuracy, 92% AUC) on impressions, while boosted trees offered the best calibration on findings.

Conclusions:

• Automated, cross-lingual RadLex mappings are effective for developing CAR tools, reducing reliance on extensive expert annotation.
• The study offers a practical framework for selecting ML classifiers for CAR, supporting language-agnostic development.
• This approach facilitates the creation of efficient CAR tools, particularly beneficial when expert-labeled training data is limited.