Related Experiment Video
Updated: Nov 6, 2025
08:27
Three-Dimensional Finger Motion Tracking during Needling: A Solution for the Kinematic Analysis of Acupuncture Manipulation
Published on: October 28, 2021
3.0K
Using Natural Language Processing to Automatically Assess Feedback Quality: Findings From 3 Surgical Residencies
Erkin Ötleş1, Daniel E Kendrick2, Quintin P Solano3
1E. Ötleş is Medical Scientist Training Program fellow, Department of Industrial and Operations Engineering, University of Michigan Medical School, Ann Arbor, Michigan.
Summary
Natural language processing (NLP) can classify surgical trainee feedback quality. Support vector machine (SVM) models achieved 83% accuracy distinguishing high-quality from low-quality feedback, showing promise for automated assessment.
Area of Science:
- Medical Education
- Natural Language Processing
- Surgical Training
Background:
- High-quality feedback is crucial for learning but difficult to scale.
- Natural language processing (NLP) offers potential for automated analysis of narrative feedback.
- The efficacy of NLP in evaluating surgical trainee feedback quality remains unexplored.
Purpose of the Study:
- To evaluate NLP techniques for classifying the quality of surgical trainee formative feedback.
- To identify the NLP models that best categorize feedback quality in a workplace assessment context.
Main Methods:
- Collected and manually coded 600 surgical trainee feedback comments for quality.
- Trained NLP models, including SVM, logistic regression, and random forests, to classify feedback into four categories.
- Assessed model performance based on mean classification accuracy.
Main Results:
- The Support Vector Machine (SVM) NLP model achieved a maximum mean accuracy of 0.64 for four-category classification.
- When simplified to high-quality vs. low-quality feedback, the SVM model reached a maximum mean accuracy of 0.83.
- SVM demonstrated superior performance in classifying feedback quality.
Conclusions:
- NLP models, particularly SVM, can automatically classify the quality of surgical trainee evaluations.
- This is the first study to demonstrate NLP's utility in this domain.
- Larger datasets are expected to further enhance NLP model accuracy for feedback quality assessment.
Related Concept Videos
01:01Feedback Loops
61.2K
In most cases, excessive hormone production is prevented by negative feedback—a loop that starts with a stimulus inducing the release of a particular substance, like a hormone, to maintain a certain level before triggering a signal that results in a decrease in further release of the hormone.
61.2K
01:15Nursing Evaluation
3.9K
The evaluation stage signals the end of the nursing process. The nurse gathers evaluative data to assess whether or not the patient has attained the expected results. Whereas the nurse collects data in the nursing assessment to identify the patient's health concerns, the evaluation stage data determines if the indicated health issues are resolved. Evaluative data collection includes two sections: the data acquired to evaluate patient outcomes and the time criteria for data collection.
3.9K
01:20Nursing Process for Patient and Caregiver Teaching III: Evaluation and Documentation
2.2K
Evaluation of the teaching process enables the nurse to determine if the patient's learning needs were met and if training was effective. If the expected outcomes are not met, the care plan is revised, and additional education or reinforcement is provided. Nurses can ask questions after the session or obtain feedback to assess the patient's understanding of the topic.
Nurses can use several methods to evaluate patient outcomes. For example, oral questions can assess cognitive learning,...
Nurses can use several methods to evaluate patient outcomes. For example, oral questions can assess cognitive learning,...
2.2K
00:46Feedback Inhibition
56.0K
Biochemical reactions are occurring constantly in cells, converting starting substances to different products, usually with the help of enzymes that speed the reactions. Without enzymes, it would take far too long for most reactions to occur to be useful to the cell!
56.0K
01:30Assessment of Airway, Skin Color, and Use of Accessory Muscles
1.3K
A thorough assessment of respiratory health is paramount in clinical settings to identify and manage respiratory distress and ensure adequate oxygenation. This article elaborates on the critical aspects of respiratory evaluation, including airway assessment, skin color examination, and the observation of accessory muscle use, which are integral to effectively diagnosing and managing patients with respiratory conditions.
Introduction
The initial evaluation of a patient's respiratory system...
Introduction
The initial evaluation of a patient's respiratory system...
1.3K
01:08Role of Communication in the Nursing Process III: Evaluation and Documentation
1.7K
A successful patient outcome depends mainly on the evaluation stage of the nursing process. Evaluation determines effectiveness by reviewing what was done previously after the completion of nursing interventions. Every time a healthcare professional steps in or administers treatment, they must reassess or evaluate the action to ensure the intended result. During the evaluation phase, there are three probable patient outcomes:
1.7K