MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing for Healthcare
• Clinical Informatics

Background:

• Large language models (LLMs) show promise for reducing healthcare administrative burden and improving care quality.
• Evaluating LLMs on realistic healthcare text generation tasks, particularly with electronic health record (EHR) data, is challenging due to limitations in existing datasets.
• Current EHR question-answering datasets do not fully represent clinicians' complex information needs or documentation burdens.

Purpose of the Study:

• To introduce MedAlign, a novel benchmark dataset designed for evaluating LLMs on EHR data.
• To assess the performance of general-domain LLMs on realistic clinical instruction-following tasks using the MedAlign dataset.
• To explore correlations between human clinician rankings and automated metrics for LLM evaluation.

Main Methods:

• Developed MedAlign, a dataset comprising 983 natural language instructions for EHR data, curated by 15 clinicians across 7 specialties.
• Included clinician-written reference responses for 303 instructions and 276 longitudinal EHRs for grounding.
• Evaluated 6 general-domain LLMs using MedAlign, with clinicians ranking the accuracy and quality of generated responses.

Main Results:

• LLM error rates were substantial, ranging from 35% for GPT-4 to 68% for MPT-7B-Instruct.
• GPT-4 experienced an 8.3% decrease in accuracy when reducing context length from 32k to 2k tokens.
• Correlations were found between clinician rankings and automated natural language generation metrics, suggesting potential for automated LLM assessment.

Conclusions:

• Existing general-domain LLMs exhibit significant limitations in accurately processing and generating EHR data based on clinical instructions.
• The MedAlign dataset provides a crucial resource for developing and evaluating LLMs tailored to specific clinician needs and preferences in healthcare.
• Further research is needed to improve LLM performance and develop reliable automated evaluation methods for clinical applications.