Enhancing chest X-ray datasets with privacy-preserving large language models and multi-type annotations: a...

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Current chest X-ray (CXR) report labeling relies on rule-based systems or supervised deep learning, often yielding limited, presence-only labels.
• Existing methods lack adaptability and struggle with nuanced information like location, severity, and uncertainty.
• The quality of labels directly impacts the utility of datasets for developing and evaluating AI models.

Purpose of the Study:

• To introduce MAPLEZ (Medical report Annotations with Privacy-preserving Large language model using Expeditious Zero shot answers), a novel approach for extracting and enhancing CXR report findings labels.
• To demonstrate MAPLEZ's capability to extract comprehensive annotations, including presence, location, severity, and uncertainty.
• To evaluate the impact of MAPLEZ-generated annotations on the performance of downstream classification models.

Main Methods:

• Leveraging a locally executable Large Language Model (LLM) for zero-shot extraction of medical report annotations.
• Developing MAPLEZ to extract multi-type findings labels (presence, location, severity, uncertainty) from CXR reports.
• Comparing MAPLEZ's annotation performance against existing labelers across multiple datasets and abnormalities.

Main Results:

• MAPLEZ achieved a 3.6 percentage point (pp) increase in macro F1 score for categorical presence annotations and over 20 pp increase for location annotations compared to competing methods.
• Models trained with MAPLEZ annotations showed a 1.1 pp increase in AUROC, outperforming those trained with the best alternative annotations.
• Significant improvements in both label quality and downstream model performance were observed.

Conclusions:

• MAPLEZ offers a significant advancement in automated chest X-ray report labeling, providing richer and more accurate annotations.
• The enhanced labels generated by MAPLEZ lead to substantial improvements in the performance of AI models for medical image analysis.
• This approach demonstrates the potential of LLMs for improving the quality and utility of medical datasets.