The DRAGON benchmark for clinical NLP.

Area of Science:

• Medical Informatics
• Computational Linguistics
• Artificial Intelligence

Background:

• Artificial Intelligence (AI) can address the shortage of medical diagnostic personnel.
• Large-scale annotated datasets are crucial for training clinical AI algorithms.
• Natural Language Processing (NLP), particularly Large Language Models (LLMs), shows promise for clinical data annotation but lacks public benchmarks.

Purpose of the Study:

• Introduce the DRAGON challenge, a benchmark for clinical NLP.
• Facilitate automated, large-scale, and cost-effective annotation of clinical data.
• Evaluate the impact of different pretraining strategies on LLM performance for clinical NLP tasks.

Main Methods:

• Developed the DRAGON challenge with 28 tasks and 28,824 annotated medical reports from five Dutch care centers.
• Pretrained foundational LLMs on four million clinical reports from a sixth Dutch care center.
• Evaluated LLM performance using domain-specific, mixed-domain, and general-domain pretraining strategies.

Main Results:

• Domain-specific pretraining achieved the highest performance (DRAGON 2025 test score of 0.770), outperforming mixed-domain (0.756) and general-domain pretraining (0.734, p < 0.005).
• Strong performance was observed on 18 out of 28 tasks.
• Subpar performance on 10 tasks highlights areas requiring further innovation in clinical NLP.

Conclusions:

• Domain-specific pretraining is superior for clinical NLP tasks.
• The DRAGON challenge benchmark provides a valuable resource for advancing clinical NLP.
• Publicly available benchmark, code, and foundational LLMs will accelerate research and development in automated clinical data annotation.