Generalizable clinical note section identification with large language models.

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Artificial Intelligence in Healthcare

Background:

• Clinical note section identification is crucial for information retrieval and downstream NLP tasks.
• Traditional supervised methods face challenges with transferability across different clinical datasets.
• Large language models (LLMs) offer a potential solution to overcome these limitations.

Purpose of the Study:

• To evaluate the effectiveness of LLMs for clinical note section identification.
• To compare the performance of various LLMs, including GPT-4, GPT-3.5, and open-source models.
• To investigate the impact of fine-tuning dataset size and specificity on LLM performance.

Main Methods:

• Framed section identification as a question-answering task using free-text section definitions.
• Evaluated multiple LLMs off-the-shelf without prior training.
• Fine-tuned selected LLMs using datasets of varying sizes and specificities.

Main Results:

• GPT-4 achieved the highest F1 score (0.77), outperforming other models.
• GPT-4 demonstrated high accuracy for specific section types (F1 > 0.9 for 33%, F1 > 0.8 for 56%).
• Fine-tuned models showed diminishing returns with larger general datasets but improved with specific section identification examples.

Conclusions:

• LLMs, particularly GPT-4, are highly promising for generalizable clinical note section identification and are nearing production readiness.
• Open-source LLMs are rapidly improving and approaching the performance of leading proprietary models.
• Further improvements can be achieved by incorporating section identification examples into LLM fine-tuning datasets.