Deep learning uncertainty quantification for clinical text classification.

Area of Science:

• Computational pathology
• Machine learning in healthcare
• Cancer registry informatics

Background:

• Deep neural networks (DNNs) are state-of-the-art for classification tasks.
• Reliability and calibration of DNNs are crucial for human-AI collaboration in decision-making.
• Automating information extraction from pathology reports is vital for cancer registries.

Purpose of the Study:

• To demonstrate DNN-based classification for automated extraction of cancer diagnosis and surgery information from pathology reports.
• To introduce selective classification methods to achieve target accuracy while minimizing unreliable predictions.
• To compare proposed methods against the current deep learning-based abstaining classifier (DAC).

Main Methods:

• Development and application of multiple selective classification methods using DNNs.
• Automated information extraction from electronic pathology reports for US National Cancer Institute (NCI) Surveillance, Epidemiology, and End Results (SEER) registries.
• Comparative analysis of proposed methods against a deep learning-based abstaining classifier (DAC) on in-distribution and out-of-distribution data.

Main Results:

• All proposed selective classification methods achieved target accuracy with minimized rejection rates.
• The proposed methods demonstrated lower rejection rates compared to the DAC on both in-distribution and out-of-distribution test data.
• The best proposed methods achieved high accuracy (≥97%) with a reduced rejection rate compared to DAC.

Conclusions:

• Selective classification methods effectively balance accuracy and rejection rates for reliable DNN predictions.
• Proposed methods retain a larger fraction of reliable predictions without retraining, offering reduced computational cost.
• These advancements enhance the automation of information extraction in cancer registries, supporting human annotators.