LCD Benchmark: Long Clinical Document Benchmark on Mortality Prediction for Language Models.

Area of Science:

• Clinical Informatics
• Natural Language Processing
• Machine Learning

Background:

• Clinical documents contain valuable unstructured data crucial for healthcare insights.
• Natural Language Processing (NLP) methods are essential for extracting information from clinical text.
• Existing benchmark datasets are insufficient for evaluating NLP models on long clinical documents.

Purpose of the Study:

• Introduce the LCD benchmark, a novel dataset for long clinical document classification.
• Facilitate the prediction of 30-day out-of-hospital mortality using discharge notes.
• Provide a standardized resource for developing and assessing NLP models in the clinical domain.

Main Methods:

• Developed the LCD benchmark using MIMIC-IV discharge notes and statewide death data.
• Evaluated the benchmark with diverse models, including bag-of-words, CNN, and large language models (LLMs).
• Conducted a comprehensive analysis of model outputs, including manual review and weight visualization.

Main Results:

• The dataset features clinical notes with a median word count of 1687.
• Best-performing supervised models achieved 28.9% F1-score, while GPT-4 reached 32.2%.
• The benchmark presents a challenge for both models and human experts, yet models demonstrate an ability to identify relevant predictive signals.

Conclusions:

• The LCD benchmark serves as a valuable resource for advancing NLP in clinical text analysis.
• It supports the development of sophisticated supervised models and prompting techniques for long clinical documents.
• The dataset is publicly available to foster research in clinical NLP.