Zero- and few-shot Named Entity Recognition and Text Expansion in medication prescriptions using large language...

Area of Science:

• Computational Linguistics
• Health Informatics
• Artificial Intelligence in Medicine

Background:

• Medication information in electronic health records (EHR) is often unstructured free-text.
• This free-text format includes diverse languages, brand names, and abbreviations, hindering interpretation.
• Large language models (LLMs) offer potential for processing and structuring complex text data.

Purpose of the Study:

• To evaluate the efficacy of ChatGPT3.5 in automatically structuring and expanding medication statements from EHR discharge summaries.
• To assess the performance of Named Entity Recognition (NER) and Text Expansion (EX) using zero- and few-shot prompt strategies.
• To compare the performance of ChatGPT3.5 with other advanced LLMs for medication statement processing.

Main Methods:

• Utilized ChatGPT3.5 with zero- and few-shot prompt strategies for NER and EX tasks on free-text medication statements.
• Manually annotated and curated 100 medication statements for evaluation.
• Measured NER performance using strict and partial matching, and EX performance via semantic equivalence assessed by experts; F1 scores were calculated.

Main Results:

• The best-performing prompt for NER achieved an average F1 score of 0.94.
• The few-shot prompt for EX demonstrated superior performance with an average F1 score of 0.87.
• Most tested LLMs (ChatGPT4o, Gemini 2.0 Flash, MedLM-1.5-Large, DeepSeekV3) outperformed ChatGPT3.5 in both NER and EX tasks.

Conclusions:

• ChatGPT3.5 shows significant potential for accurate NER and EX of free-text medication statements.
• A few-shot learning approach is crucial for preventing hallucinations in safety-critical medication data processing.
• Advanced LLMs generally offer improved performance over ChatGPT3.5 for these clinical NLP tasks.