Accuracies of Training Labels and Machine Learning Models: Experiments on Delirium and Simulated Data

Area of Science:

• Machine Learning
• Clinical Informatics
• Data Science

Background:

• Supervised predictive models rely on labeled data for training.
• Incomplete or inaccurate labeled data is a common challenge in real-world applications.
• The impact of imperfect data accuracy on model performance is a critical research question.

Purpose of the Study:

• To investigate if imperfectly labeled data imposes a performance ceiling on trained models.
• To evaluate the performance of models trained on clinical data with inaccurate annotations.
• To determine if models trained on imperfect data can outperform the accuracy of the training data.

Main Methods:

• Trained multiple supervised models to detect delirium in clinical documents using imperfectly labeled data.
• Utilized a support vector machine with a linear kernel as a primary model.
• Generated simulated data and conducted experiments to validate findings on imperfect data.

Main Results:

• The support vector machine model achieved an area under the curve of 89.3% and 88% accuracy in external evaluation.
• Model performance (88% accuracy) surpassed the training sample's accuracy (80%).
• Experiments demonstrated that models trained on imperfect data can, but do not always, exceed training data accuracy.

Conclusions:

• Imperfectly labeled data does not necessarily create a performance ceiling for supervised models.
• High-performing predictive models can be developed using clinical data with annotation inaccuracies.
• The study highlights the potential of machine learning in clinical settings despite data quality challenges.