Extracting Cardiorespiratory Symptoms From Clinical Notes Using Open-Weight Large Language Models: Method Development and Validation Study | JoVE Visualize

Area of Science:

Biomedical Informatics
Natural Language Processing (NLP)
Artificial Intelligence in Healthcare

Background:

Accurate identification of cardiorespiratory symptoms and signs (S&S) is crucial for early detection of high-burden conditions like lung cancer, COPD, and heart failure.
S&S data are vital for diagnostics and predictive modeling but are often trapped in unstructured electronic health record (EHR) notes, limiting automated use.
Traditional NLP methods struggle with clinical text's complexity, while large language models (LLMs) offer promise but face challenges like hallucinations and safe deployment.

Purpose of the Study:

To evaluate the accuracy of locally deployed, open-source LLMs in extracting cardiorespiratory S&S from clinical notes.
To assess the LLMs' ability to map extracted S&S to International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) codes.
To compare the performance of four distinct prompt-engineering strategies, including a multimodule LLM framework.

Main Methods:

A total of 593 clinical notes were reviewed; 93 were used for prompt development and 500 for testing.
Four prompting strategies were evaluated: instruction-only, ICD-10-CM definition-based, assumption-free, and a multimodule LLM framework with postprocessing.
Performance was measured using precision, recall, and F1-score for both S&S extraction and ICD-10-CM code generation, utilizing Llama 3.3-70B and gpt-oss-120B models.

Main Results:

Model performance improved with increased structure and constraints in prompts; the multimodule approach yielded the best results during development.
On the test corpus, gpt-oss-120B outperformed Llama 3.3-70B: gpt-oss-120B achieved 0.88 F1-score for S&S extraction and 0.87 F1-score for ICD-10-CM coding.
Llama 3.3-70B achieved an F1-score of 0.73 for S&S extraction and 0.69 for ICD-10-CM coding, highlighting the impact of model choice and prompting strategy.

Conclusions:

Locally deployed LLMs, enhanced by optimized prompting and multimodule orchestration, can accurately extract cardiorespiratory S&S and generate ICD-10-CM codes from unstructured EHR data.
This on-premises processing approach enhances data safety by preventing external transmission, offering a scalable solution for symptom extraction in biomedical informatics.
Future research should focus on expanding datasets and validating generalizability across diverse clinical domains to further refine these LLM applications.