Related Experiment Video
Updated: Aug 6, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Enhancing Large Language Models for Identifying and Prioritizing Important Medical Jargons From Electronic Health
Won Seok Jang1, Sharmin Sultana1, Zonghai Yao2
1Miner School of Computer and Information Sciences, University of Massachusetts Lowell, Lowell, MA, United States.
JMIR AI
|July 17, 2026
Summary
Open-source large language models (LLMs) show promise for extracting medical jargon from electronic health record (EHR) notes. Fine-tuning and data augmentation significantly improve their performance in low-resource settings.
Area of Science:
- Natural Language Processing
- Artificial Intelligence in Healthcare
- Medical Informatics
Background:
- Patients access electronic health record (EHR) notes via patient portals.
- EHR notes contain complex medical jargon, hindering patient comprehension.
- Simplifying EHR notes by prioritizing key medical terms can improve understanding.
Purpose of the Study:
- Evaluate closed-source and open-source large language models (LLMs) for medical jargon extraction and prioritization from EHR notes.
- Investigate the impact of prompting techniques, fine-tuning, and data augmentation on LLM performance.
- Identify optimal strategies for enhancing patient comprehension of EHR notes.
Main Methods:
- Performance evaluation of LLMs on 90 expert-annotated EHR notes.
- Testing various settings: general/structured prompts, zero-shot/few-shot prompting, fine-tuning, and data augmentation.
- Utilizing GPT-4o for data augmentation and a ranking technique for open-source models.
- Assessing performance using 10-fold cross-validation, F1-score, and mean reciprocal rank with two string matching algorithms.
- Conducting error analysis to classify model output inaccuracies.
Main Results:
- Open-source models excelled with fine-tuning on gold-standard data, achieving high F1-scores and mean reciprocal ranks.
- Performance varied significantly based on prompting styles (general vs. structured).
- Fine-tuning with small datasets and data augmentation improved performance, with augmentation yielding comparable or superior results.
- Data augmentation effectiveness depended on the quality of the augmented dataset.
Conclusions:
- Prompting strategies, fine-tuning, and data augmentation are effective for enhancing LLM performance in low-resource scenarios for EHR note analysis.
- Open-source LLMs demonstrate potential for improving patient understanding of medical jargon.
- Further research into data augmentation quality is crucial for maximizing LLM utility in healthcare.
