Fine-Tuning Large Language Models to Enhance Programmatic Assessment in Graduate Medical Education.

Area of Science:

• Natural Language Processing
• Machine Learning in Medical Education

Background:

• Narrative feedback in medical education is crucial for trainee development.
• Classifying feedback into Accreditation Council for Graduate Medical Education (ACGME) subcompetencies is challenging.
• Large Language Models (LLMs) offer potential for automated feedback analysis.

Purpose of the Study:

• To explore the trade-off between LLM complexity and performance in classifying ACGME subcompetencies.
• To compare the performance of various transformer-based LLMs against a FastText model.

Main Methods:

• Fine-tuned several transformer-based LLMs (BERT-base, BERT-medium, BERT-small, BERT-mini, BERT-tiny, SciBERT) on 10,218 feedback comments.
• Compared LLM performance using F1 score and area under the receiver operating characteristic curve (AUC).
• Evaluated models against a previously trained FastText model.

Main Results:

• No transformer-based LLMs outperformed the FastText model.
• BERT-tiny performed worse than FastText.
• BERT-mini achieved comparable performance to FastText but was 94% smaller.
• AUC scores were similar for BERT-mini and FastText, with minor exceptions.

Conclusions:

• Complex LLMs did not offer superior performance over simpler models for this task.
• Smaller, efficient LLMs like BERT-mini can achieve comparable results, enabling deployment on personal devices.
• This research informs best practices for integrating LLMs in graduate medical education.