Construction of a Feedback Comment Analysis Model for Evaluation of Endoscopic Surgical Skill

Surgical education and skill assessments are important in improving surgical skills. However, instructors' comments tend to be complex and unorganized, with varying content and categories. This study aimed to develop a natural language processing (NLP) model to automatically classify feedback comments on surgical procedures and evaluate the performance of the developed model. An NLP model was used to automatically analyze the tendency of differences in feedback comments for each surgeon's skill.

Evaluation comments provided to 1813 applicants seeking endoscopic surgical skill qualification for colorectal cases were used. As a ground truth for model training, comments were manually categorized into five competencies-tissue handling, psychomotor skills, dissection quality, exposure quality, and efficiency-and two emotional polarities-positive or negative. The competency and emotional polarity percentages were compared between applicants who did and did not qualify. An automatic feedback comment classification model based on NLP was constructed and verified. Feedback comments on surgical videos were analyzed using an NLP model.

Training data comprised 5846 sentences from 175 cases. The overall accuracies of the comment analysis task were 0.75 and 0.94 for competency and emotional polarity classifications, respectively. The remaining 1526 cases were analyzed using the NLP model. The proportion of positive polarity was significantly higher in the qualified surgeon group than in the unqualified surgeon group.

The NLP model developed to evaluate feedback comments related to surgical skill efficiently organized feedback comments on surgical procedures and may contribute to enhanced skill evaluation and surgical education.