Leveraging Natural Language Processing and Structured Scoring to Evaluate Operative Note Quality in Surgical Training

Area of Science:

• Medical Informatics
• Surgical Education
• Natural Language Processing

Background:

• Operative notes are crucial for procedural documentation, care continuity, and surgical training.
• Previous research indicates resident-authored notes may have omissions or errors.
• Analyzing free-text operative notes at scale has been challenging.

Purpose of the Study:

• To evaluate the quality of resident versus attending operative notes using a novel two-phase approach.
• To leverage natural language processing (NLP) and structured assessment for objective quality measurement.
• To explore the potential of operative notes as a competency assessment tool for surgical trainees.

Main Methods:

• A two-phase approach combining NLP diffing and structured assessment (SAFE-OR) was employed.
• Operative notes from general surgery procedures with both resident and attending versions were analyzed.
• Grader severity was adjusted using z-normalization and rescaling for accurate comparisons.

Main Results:

• NLP diffing revealed significant variations in edit burden across different surgical procedures.
• Resident-authored notes scored significantly lower than attending notes in preoperative (Section I) and postoperative (Section III) elements.
• Overall quality scores were significantly lower for resident notes compared to attending notes, with no significant difference in intraoperative elements (Section II).

Conclusions:

• Resident operative notes demonstrate lower quality and fewer key elements compared to attending notes, especially in preoperative and postoperative sections.
• The combined NLP and structured grading approach provides a high-throughput, objective method for assessing operative note quality.
• This methodology offers a new pathway for evaluating surgical trainees' understanding and documentation skills.