Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Models:...

Area of Science:

• Health Informatics
• Machine Learning
• Natural Language Processing

Background:

• Real-world data collection often compromises data quality, impacting machine learning (ML) model performance.
• Textual data in healthcare, such as progress notes, requires high accuracy due to potential life-threatening consequences of incorrect ML predictions.
• Assessing the impact of textual data quality is essential for reliable healthcare ML applications.

Purpose of the Study:

• To quantify textual dataset quality and evaluate the impact of errors on ML models.
• To determine the tolerance of feature representations and ML models to data errors.
• To assess the justification for investing resources in improving data quality for healthcare ML.

Main Methods:

• Developed a token-level error rate metric for textual dataset quality assessment.
• Utilized the Mixtral large language model (LLM) to quantify and correct errors in datasets.
• Analyzed the MIMIC-III (high-quality) and Australian Aged Care Homes (AACHs; lower-quality) datasets, introducing errors into MIMIC-III and correcting AACHs.
• Evaluated feature representations and ML models using the area under the receiver operating curve (AUC).

Main Results:

• Mixtral detected errors in 63% of progress notes, with 17% having single token misclassifications.
• Feature representation performance tolerated error rates below 10% but declined significantly above this threshold.
• The AACH dataset, with an 8% error rate, showed no major performance degradation.
• Term frequency-inverted document frequency (TF-IDF) outperformed embedding features; ML model effectiveness varied by task.

Conclusions:

• ML models are sensitive to data quality, with performance degrading significantly at error rates of 10% or higher.
• Dataset quality evaluation is critical before ML implementation in healthcare.
• Corrective measures are essential for datasets with high error rates to ensure ML model reliability and effectiveness.