Selective prediction for extracting unstructured clinical data

Area of Science:

• Clinical Informatics
• Machine Learning in Healthcare
• Natural Language Processing

Background:

• Current methods for extracting information from unstructured clinical data, like manual abstraction and structured proxy variables, are often inefficient and imprecise.
• There is a need for scalable and accurate solutions to handle the growing volume of clinical text data.

Purpose of the Study:

• To evaluate the effectiveness of selective prediction models in improving the accuracy and efficiency of unstructured clinical data abstraction.
• To compare the performance of selective classifiers against non-selective models and structured proxy variables.

Main Methods:

• Trained selective classifiers (logistic regression, random forest, support vector machine) to extract five variables: depression, glioblastoma (GBM), rectal adenocarcinoma (DRA), abdominoperineal resection (APR), and low anterior resection (LAR).
• Varied the costs associated with false positives, false negatives, and abstained predictions to measure total misclassification cost.
• Assessed performance metrics including sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).

Main Results:

• Selective classifiers demonstrated significant abstention rates, ranging from 0% to 97% for depression and 5% to 43% for GBM and colorectal cancer models.
• For glioblastoma (GBM) extraction, a selective classifier abstained on 43% of notes, leading to improved sensitivity (0.94 to 0.96), specificity (0.79 to 0.96), PPV (0.89 to 0.98), and NPV (0.88 to 0.91) compared to non-selective classifiers and structured proxy variables.
• Selective prediction models reduced total misclassification costs by up to 58% in some cases.

Conclusions:

• Selective classifiers outperformed both non-selective classifiers and structured proxy variables in extracting data from unstructured clinical notes.
• Selective prediction is a valuable strategy when avoiding incorrect predictions is more critical than making a prediction for every instance.
• This approach offers a promising avenue for more accurate and efficient clinical data abstraction.