LCD benchmark: long clinical document benchmark on mortality prediction for language models.

Area of Science:

• Clinical informatics
• Natural Language Processing (NLP)
• Machine Learning

Background:

• Clinical documents contain rich unstructured data crucial for healthcare insights.
• Existing Natural Language Processing (NLP) benchmark datasets are insufficient for long clinical documents.
• Lack of benchmarks hinders development and evaluation of models for clinical text analysis.

Purpose of the Study:

• Introduce the Long Clinical Document (LCD) benchmark for classifying long clinical texts.
• Enable prediction of 30-day out-of-hospital mortality using discharge notes.
• Facilitate the development and assessment of NLP models for clinical document understanding.

Main Methods:

• Developed the LCD benchmark using MIMIC-IV discharge notes and statewide death data.
• Evaluated the benchmark with diverse models, including bag-of-words, CNNs, and large language models.
• Conducted comprehensive analysis of model outputs, including manual review and weight visualization.

Main Results:

• The LCD benchmark features clinical notes with a median word count of 1687.
• Best supervised models achieved 28.9% F1 score, while GPT-4 reached 32.2% F1 score.
• Model performance indicates the dataset is challenging but contains valuable predictive signals.

Conclusions:

• The LCD benchmark presents a significant challenge for current NLP models and human experts.
• Developed models demonstrate the ability to extract meaningful predictive information from long clinical documents.
• The LCD benchmark is expected to drive advancements in supervised and prompting methods for clinical NLP.