Related Experiment Video
Updated: Oct 13, 2025
Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention
Published on: December 15, 2023
Exploring how feedback reflects entrustment decisions using artificial intelligence
Brian C Gin1, Olle Ten Cate2,3, Patricia S O'Sullivan3,4
1Department of Pediatrics, University of California San Francisco, San Francisco, CA, USA.
Context:
Clinical supervisors make judgements about how much to trust learners with critical activities in patient care. Such decisions mediate trainees' opportunities for learning and competency development and thus are a critical component of education. As educators apply entrustment frameworks to assessment, it is important to determine how narrative feedback reflecting entrustment may also address learners' educational needs.
Methods:
In this study, we used artificial intelligence (AI) and natural language processing (NLP) to identify characteristics of feedback tied to supervisors' entrustment decisions during direct observation encounters of clerkship medical students (3328 unique observations). Supervisors conducted observations of students and collaborated with them to complete an entrustment-based assessment in which they documented narrative feedback and assigned an entrustment rating. We trained a deep neural network (DNN) to predict entrustment levels from the narrative data and developed an explainable AI protocol to uncover the latent thematic features the DNN used to make its prediction.
Results:
We found that entrustment levels were associated with level of detail (specific steps for performing clinical tasks), feedback type (constructive versus reinforcing) and task type (procedural versus cognitive). In justifying both high and low levels of entrustment, supervisors detailed concrete steps that trainees performed (or did not yet perform) competently.
Conclusions:
Framing our results in the factors previously identified as influencing entrustment, we find a focus on performance details related to trainees' clinical competency as opposed to nonspecific feedback on trainee qualities. The entrustment framework reflected in feedback appeared to guide specific goal-setting, combined with details necessary to reach those goals. Our NLP methodology can also serve as a starting point for future work on entrustment and feedback as similar assessment datasets accumulate.
Related Concept Videos
01:29Introspection
01:14Decision Making: Traditional Method
First, a specific claim about the population parameter is decided based on the research question and is stated in a simple form. Further, an opposing statement to this claim is also stated. These statements can act as null and alternative hypotheses, out of which a null hypothesis would be a...
01:24Effects of feedback
Feedback significantly modifies the gain of a control system. The gain of a system without feedback is altered by a factor of one plus GH, where G represents...
01:20Decision Making
Automatic decision-making is fast, intuitive, and relies on gut feelings...
01:26Feedback control systems
Linear feedback systems are theoretical models that simplify analysis and design. These systems operate under the principle that their output is directly proportional to their input within certain ranges. For instance, an amplifier in a control system behaves linearly as long as the input signal remains within a specific range. However, most physical systems exhibit inherent nonlinearity...
01:26Equity Theory