Leveraging Narrative Feedback in Programmatic Assessment: The Potential of Automated Text Analysis to Support...

Area of Science:

• Medical Education
• Computational Linguistics
• Psychometrics

Background:

• Narrative feedback is crucial for learning, coaching, and decision-making in assessments.
• Interpreting narrative feedback can be challenging and time-consuming, hindering effective use of assessment data.
• Support is needed for learners, coaches, and decision-makers to effectively use and interpret narrative feedback.

Purpose of the Study:

• To explore the utility of automated text analysis for interpreting narrative assessment data.
• To identify predominant feedback themes and sentiment polarity in narrative assessments.
• To examine associations between feedback polarity, performance scores, and task judgments.

Main Methods:

• Applied topic modelling and sentiment analysis to 926 clinical assessments of 80 trainees.
• Utilized automated techniques to identify feedback themes (Medical Skills, Knowledge, Communication & Professionalism) and sentiment polarity.
• Examined correlations between feedback sentiment, numerical performance scores, and overall judgments.

Main Results:

• Topic modelling identified three key feedback themes.
• Assessors provided more detailed feedback to trainees not meeting competence standards.
• A strong positive correlation was found between average performance scores and average sentiment polarity, though not at the single assessment level.

Conclusions:

• Automated text analysis can lead to more efficient, structured, and meaningful assessment experiences.
• These techniques can facilitate deeper conversations about assessment data by aiding interpretation.
• Further research is essential to integrate automated text analysis into educational practices for maximum benefit.