Performance of Open-Source Large Language Models to Extract Symptoms from Clinical Notes

Area of Science:

• Natural Language Processing (NLP) in Healthcare
• Clinical Informatics
• Artificial Intelligence (AI) in Medicine

Background:

• Automated extraction of clinical information from electronic health records (EHRs) is crucial for improving healthcare efficiency.
• Large language models (LLMs) offer potential for analyzing unstructured clinical text.
• Evaluating LLM performance on specific clinical tasks like symptom extraction and coding is essential.

Purpose of the Study:

• To assess the capability of open-source foundational large language models (LLMs) in extracting symptoms and signs (S&S) and their corresponding ICD-10 codes from clinical notes.
• To compare the performance of different Llama model versions (Llama 3.1-13B, Llama 3.3-70B, Me-Llama-13B) on S&S extraction and ICD-10 code generation.

Main Methods:

• Utilized the public MTSamples dataset, focusing on genitourinary conditions.
• Manually annotated a subset of the dataset for ground truth comparison.
• Evaluated three Llama model versions on S&S extraction and ICD-10 code generation tasks, measuring consistency, runtime, and performance metrics (recall, precision).

Main Results:

• Llama 3.3-70B exhibited the best overall performance among the tested models.
• This model achieved a fast runtime and high consistency.
• For S&S extraction, Llama 3.3-70B reported an average recall of 0.87 and precision of 0.71.
• For ICD-10 code generation, it achieved an average recall of 0.71 and precision of 0.54.

Conclusions:

• Open-source LLMs, particularly Llama 3.3-70B, are effective tools for automated extraction of symptoms, signs, and ICD-10 codes from clinical notes.
• The findings suggest significant potential for LLMs to streamline clinical data processing and coding in healthcare settings.
• Further research can explore optimizing LLM performance for diverse clinical datasets and complex coding requirements.