Investigating the Impact of Prompt Engineering on the Performance of Large Language Models for Standardizing...

Area of Science:

• Medical Informatics
• Natural Language Processing in Healthcare
• Artificial Intelligence in Medicine

Background:

• Electronic medical records (EMRs) offer significant research value, especially in obstetrics.
• Standardizing diagnostic terminology across institutions is crucial for accurate medical data analysis.
• Large language models (LLMs) are increasingly utilized for diverse medical applications, with prompt engineering being critical for their effectiveness.

Purpose of the Study:

• To evaluate and compare the performance of LLMs using various prompt engineering techniques.
• To standardize obstetric diagnostic terminology using real-world obstetric data.
• To assess the efficiency and effectiveness of LLMs against traditional models in a clinical context.

Main Methods:

• A four-step approach was employed, beginning with similarity measures for mapping diagnoses to ICD-10.
• Candidate terms were collected based on similarity scores for training data.
• Two LLMs (ChatGLM2, Qwen-14B-Chat [QWEN]) were used for zero-shot learning, and three BERT variants (BERT, whole word masking BERT, MC-BERT) were used for unsupervised optimal mapping term generation.

Main Results:

• LLMs and BERT demonstrated comparable performance, with LLMs showing advantages in efficiency in unsupervised scenarios.
• Prompt engineering significantly impacted LLM performance; the self-consistency approach in QWEN improved F1-score by 5% and precision by 7.9%.
• Momentum contrastive learning with BERT (MC-BERT) achieved the highest performance among BERT variants, though differences were minor.

Conclusions:

• LLMs, particularly with optimized prompts like those used with QWEN, can effectively standardize obstetric diagnostic terms.
• The precision achieved by QWEN prompts was comparable to that of BERT models.
• Unsupervised LLM approaches show potential for enhancing diagnostic term alignment in research and unlocking insights from patient data.