LongHealth: A Question Answering Benchmark with Long Clinical Documents.

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing for Clinical Data
• Medical Informatics

Background:

• Large language models (LLMs) offer potential for analyzing extensive patient records.
• Existing benchmarks inadequately assess LLM performance on real-world, lengthy clinical data.

Purpose of the Study:

• To introduce the LongHealth benchmark for evaluating LLMs on long clinical documents.
• To assess the capabilities of various LLMs in processing and interpreting complex patient case files.

Main Methods:

• Developed the LongHealth benchmark with 20 detailed fictional patient cases (5090-6754 words each).
• Created 400 multiple-choice questions across information extraction, negation, and sorting tasks.
• Evaluated eleven open-source LLMs and GPT-3.5 Turbo on the benchmark.

Main Results:

• Mistral-Small-24B-Instruct-2501 and Llama-4-Scout-17B-16E-Instruct achieved the highest accuracy, particularly in information retrieval.
• All evaluated models demonstrated significant difficulties in identifying missing information within clinical documents.
• Current LLM accuracy is insufficient for safe and reliable clinical application.

Conclusions:

• LLMs have potential for processing long clinical documents but require further refinement for healthcare.
• The LongHealth benchmark offers a realistic assessment of LLM performance in clinical settings.
• Future research should focus on improving LLM accuracy in identifying missing clinical data.