Automating the Identification of Feedback Quality Criteria and the CanMEDS Roles in Written Feedback Comments Using...

Area of Science:

• Medical Education
• Natural Language Processing
• Artificial Intelligence

Background:

• Manual analysis of written feedback in healthcare education is resource-intensive.
• Identifying feedback quality criteria and CanMEDS roles is crucial for effective training.
• Existing methods for feedback analysis are time-consuming and require extensive human effort.

Purpose of the Study:

• To investigate the efficacy of fine-tuning a state-of-the-art large language model (LLM) for automated feedback quality assessment.
• To determine if an LLM can accurately identify predefined feedback quality criteria (performance, judgment, elaboration, improvement) and CanMEDS roles (Medical Expert, Communicator, Collaborator, Leader, Health Advocate, Scholar, Professional) in written comments.

Main Methods:

• Utilized a dataset of 2,349 labelled feedback comments from five Belgian healthcare educational programs.
• Split the data into 12,452 sentences for machine learning analysis.
• Trained four multiclass-multilabel classification models using Dutch BERT models (BERTje and RobBERT) to identify feedback quality criteria and CanMEDS roles.

Main Results:

• Classification models achieved macro average F1-scores of 0.73 (BERTje) and 0.76 (RobBERT) for feedback quality criteria.
• Models achieved F1-scores of 0.71 (BERTje) and 0.72 (RobBERT) for identifying CanMEDS roles.
• Demonstrated the LLM's capability in accurately detecting both feedback quality and CanMEDS roles.

Conclusions:

• State-of-the-art LLMs can be effectively fine-tuned to automate the quality analysis of written feedback comments in healthcare education.
• Automated analysis using LLMs offers significant savings in time and resources compared to manual methods.
• This approach has the potential to enhance the efficiency and scalability of feedback assessment in medical training programs.