MedSlice: fine-tuned large language models for secure clinical note sectioning.

Area of Science:

• Natural Language Processing in Healthcare
• Machine Learning for Clinical Data Analysis

Background:

• Automated extraction of clinical note sections is vital for research but hindered by manual effort and data variability.
• Proprietary large language models (LLMs) show potential but raise privacy concerns in medical applications.

Purpose of the Study:

• To develop an automated pipeline for clinical note sectioning using open-source LLMs.
• To compare the performance of fine-tuned open-source LLMs against proprietary models for section extraction.

Main Methods:

• Fine-tuning three open-source LLMs on a dataset of 487 clinical progress notes.
• Evaluating performance using precision, recall, and F1 scores for internal and external validity.
• Benchmarking against proprietary models like GPT-4o and GPT-4o mini.

Main Results:

• A fine-tuned Llama 3.1 8B model achieved an F1 score of 0.92, outperforming GPT-4o.
• High performance was maintained on an external validity test set with an F1 score of 0.85.

Conclusions:

• Fine-tuned, open-source LLMs demonstrate superior performance in clinical note sectioning compared to proprietary models.
• Open-source LLMs provide a cost-effective, privacy-preserving, and accessible alternative for clinical text analysis.