Can Artificial Intelligence Coach Faculty to Utilize Growth Mindset Language? A Qualitative Analysis of Feedback...

Feedback is at the core of competency-based medical education. Learner perceptions of the evaluation process influence how feedback is utilized. Systems emphasize a fixed mindset, prioritizing evaluation over growth. Embracing growth mindset culture, the belief that ability is acquired through effort and human capabilities can be developed over time, will allow learners to gain greater benefits from feedback. Transitioning from fixed mindset language (FML) to growth mindset language (GML) will require faculty training. Artificial intelligence (AI) can assist faculty with incorporating GML concepts in written feedback. The aim of this study was to assess the ability of AI to assist in changing FML feedback statements into statements with GML.

A qualitative study was performed utilizing a sample of 83 summative and formative feedback statements provided to students (37) and residents (46) from surgery clerkship and national SIMPL-inguinal hernia evaluations. Of these 83 statements, a reviewer coded 41 statements as using GML and 42 using FML. Original statements identified as using FML were entered into the Google Chrome "Help me write" tool, a writing aid using Generative AI. The AI tool was prompted with the statement "rewrite using growth mindset language:," followed by an original FML statement. A dataset containing a combination of AI-altered and original statements, 99 statements in all, was provided to two additional blinded reviewers trained in GML concepts. Reviewers evaluated statements as predominantly GML or FML and commented on their perception of AI use in statements. Reviewer agreement was adjudicated by the original coder.

Of the 41 original GML statements, coders correctly identified 37 (90.2%) as using GML. Of the 26 original FML statements, coders correctly identified all 26 (100%) as using FML. Of the AI-modified FML to GML statements, coders correctly identified 17 of 18 (94.4%) as using GML. They correctly identified 56.3% as AI-modified and 44.8% as not AI-modified statements. They disagreed on AI use in 39.4% of statements. AI-assistance was unrecognized in 16 (8.1%) statements and mistaken for use in 47 (23.7%) statements.

AI was successful at modifying FML statements into feedback containing GML, and in a way that was not obviously AI-generated. This proof-of-concept study demonstrates that AI can be a helpful tool for faculty to increase the use of GML in written feedback. While AI cannot perfectly create GML feedback without initial input and understanding from faculty, it does serve as a promising educational aid. As the body of work on using GML in surgical education grows, the better AI can assist in the generation of quality feedback.