The incremental value of unstructured data via natural language processing in machine learning-based COVID-19...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Prediction Models

Background:

• Machine learning models are increasingly used for clinical prediction.
• The value of unstructured clinical data for enhancing these models is debated.
• Few studies have rigorously evaluated the impact of unstructured data on model performance.

Purpose of the Study:

• To evaluate the performance improvement of machine learning models for in-hospital mortality prediction by including unstructured data.
• To compare models using only structured data versus hybrid models incorporating unstructured data.
• To quantify the impact of unstructured data on predictive accuracy.

Main Methods:

• A retrospective study compared machine learning models using structured data versus hybrid models with added unstructured data.
• Models were developed for patients diagnosed with COVID-19 at a tertiary emergency care hospital.
• Performance was assessed using metrics like Area Under the Receiver Operating Characteristic Curve (AUC ROC), sensitivity, and specificity.

Main Results:

• The best performing model, a random forest, achieved an AUC ROC of 0.9260 with unstructured data, a slight increase from 0.9170 with structured data alone.
• Sensitivity improved from 0.8108 to 0.8378, while specificity remained stable at 0.8667.
• These performance gains were not statistically significant compared to models using only structured data.

Conclusions:

• Inclusion of unstructured data did not significantly enhance the predictive power of machine learning models for COVID-19 mortality.
• Human involvement is critical for validating natural language processing outputs and selecting relevant unstructured features.
• Challenges in processing unstructured data necessitate expert human oversight for effective implementation.