Selective prediction for extracting unstructured clinical data | JoVE Visualize

Area of Science:

Clinical Informatics
Machine Learning in Healthcare
Natural Language Processing

Background:

Current methods for extracting information from unstructured clinical data, like manual abstraction and structured proxy variables, are often inefficient and imprecise.
There is a need for scalable and accurate solutions to handle the growing volume of clinical text data.

Purpose of the Study:

To evaluate the effectiveness of selective prediction models in improving the accuracy and efficiency of unstructured clinical data abstraction.
To compare the performance of selective classifiers against non-selective models and structured proxy variables.

Main Methods:

Trained selective classifiers (logistic regression, random forest, support vector machine) to extract five variables: depression, glioblastoma (GBM), rectal adenocarcinoma (DRA), abdominoperineal resection (APR), and low anterior resection (LAR).
Varied the costs associated with false positives, false negatives, and abstained predictions to measure total misclassification cost.
Assessed performance metrics including sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).

Main Results:

Selective classifiers demonstrated significant abstention rates, ranging from 0% to 97% for depression and 5% to 43% for GBM and colorectal cancer models.
For glioblastoma (GBM) extraction, a selective classifier abstained on 43% of notes, leading to improved sensitivity (0.94 to 0.96), specificity (0.79 to 0.96), PPV (0.89 to 0.98), and NPV (0.88 to 0.91) compared to non-selective classifiers and structured proxy variables.
Selective prediction models reduced total misclassification costs by up to 58% in some cases.

Conclusions:

Selective classifiers outperformed both non-selective classifiers and structured proxy variables in extracting data from unstructured clinical notes.
Selective prediction is a valuable strategy when avoiding incorrect predictions is more critical than making a prediction for every instance.
This approach offers a promising avenue for more accurate and efficient clinical data abstraction.