Application of large language models in clinical record correction: a comprehensive study on various retraining...

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing in Healthcare
• Medical Education Technology

Background:

• Clinical record (CR) assessment is vital for medical training and diagnostics.
• Evaluating the efficacy of Large Language Models (LLMs) in autonomously assessing CRs is an emerging area.
• Proprietary models like GPT-3.5 and GPT-4 offer advanced capabilities, but open-source alternatives require optimization.

Purpose of the Study:

• To assess the effectiveness of various LLMs (GPT-3.5, GPT-4, Llama-2 7B, 13B) in autonomously evaluating clinical records.
• To compare different adaptation techniques (prompt engineering, fine-tuning, LoRA) on open-source models.
• To determine the potential of LLMs in enhancing medical education and diagnostic skills through CR analysis.

Main Methods:

• Evaluated GPT-3.5, GPT-4, Llama-2 13B, and Llama-2 7B models.
• Implemented and compared prompt engineering, fine-tuning (FT), and low-rank adaptation (LoRA) on Llama-2 7B.
• Assessed model performance using prompts in English and Spanish, benchmarking against specialist evaluations.

Main Results:

• GPT-4 demonstrated performance comparable to specialist evaluations.
• Fine-tuning (FT) on Llama-2 7B improved Spanish text comprehension to Llama-2 13B English prompt levels.
• Low-rank adaptation (LoRA), especially with FT, significantly boosted performance, outperforming GPT-3.5.

Conclusions:

• GPT-4 exhibits strong performance, but FT and LoRA are critical for optimizing open-source LLMs like Llama-2 7B.
• Low-rank adaptation is a highly effective technique for enhancing open-source model capabilities.
• LLMs offer a promising, innovative approach to clinical record correction and medical education.