CaseReportBench: An LLM Benchmark Dataset for Dense Information Extraction in Clinical Case Reports.

Area of Science:

• Medical Informatics
• Natural Language Processing
• Artificial Intelligence in Medicine

Background:

• Rare diseases, such as Inborn Errors of Metabolism (IEM), present significant diagnostic hurdles.
• Case reports are valuable but underutilized resources for diagnosing rare diseases.
• Computational methods for extracting dense clinical information from case reports are needed.

Purpose of the Study:

• To introduce CaseReportBench, a novel dataset for dense information extraction from IEM case reports.
• To evaluate Large Language Models (LLMs) for clinical information extraction from case reports.
• To develop and assess advanced prompting strategies for improved LLM performance.

Main Methods:

• Development of CaseReportBench, an expert-annotated dataset for dense information extraction.
• Assessment of various LLMs and prompting techniques, including category-specific prompting and subheading-filtered data integration.
• Clinician evaluation of LLM-extracted information for clinical relevance.

Main Results:

• Category-specific prompting significantly improves LLM alignment with the benchmark dataset.
• Open-source Qwen2.5:7B demonstrated superior performance compared to GPT-4o for this task.
• LLMs successfully extracted clinically relevant details, supporting rare disease diagnosis and management.

Conclusions:

• LLMs show promise in extracting critical information from case reports to aid rare disease diagnosis.
• Novel prompting strategies enhance LLM capabilities for clinical NLP tasks.
• Further improvements are needed, particularly in recognizing negative findings for differential diagnosis.