Bridging the gap: use of automated text analysis to support interpretation of narrative feedback

Interpretation of large volumes of narrative feedback, collected in a programmatic assessment setting, poses significant challenges for students, mentors, and assessors. This report introduces an innovative tool designed to provide personalized, real-time interpretations of narrative feedback aligned with predefined learning outcomes.

Through collaborative design sessions with key stakeholders, the authors developed and piloted the tool within the master's in medicine program at Maastricht University from February 2022 to May 2024. The tool's algorithm combines topic modeling and sentiment analysis to provide visual representations of feedback data mapped onto predefined competencies and professional activities.

The innovation was piloted in the master's program in medicine at Maastricht University, the Netherlands, between December 2023 and April 2024. A total of 33 students, approximately 2 years into the 3-year program, and their 20 mentors participated in the pilot. Both mentors and students acknowledged the tool's potential to enhance effective use of narrative feedback data. Students predominantly used the tool to reflect on and self-assess their competency development and set new learning goals, whereas mentors leveraged it to guide and monitor students' learning and provide informed advice to the clinical competency committee on students' competence development. Key benefits included the visualization of feedback data aligned with intended learning outcomes and comprehensive identification of strengths, areas for improvement, and learning gaps.

Informed by pilot insights, the tool will be refined and tested in an expanded pilot. The tool's adaptable structure allows integration into diverse educational frameworks, incorporation of domain-specific terms, and deployment within e-portfolios or as a standalone tool. This scalable innovation offers a robust solution for enhancing the interpretation and use of narrative feedback and can be applied globally across a wide range of educational programs that use overarching (eg, competency-based) assessment frameworks.