Natural Language Processing and soft data for motor skill assessment: A case study in surgical training simulations.

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine
• Natural Language Processing

Background:

• Automated surgical skill assessment often relies on kinematic and video data.
• Expert opinions in free-text (soft data) offer rich semantic information for skill evaluation.
• Existing datasets lack synchronized soft and hard data for comprehensive analysis.

Purpose of the Study:

• To analyze the feasibility of using free-text expert opinions (soft data) as a sole source for surgical skill assessment.
• To evaluate various Natural Language Processing (NLP) algorithms for their effectiveness in analyzing soft data.
• To compare traditional machine learning with large language models (LLMs) for skill assessment.

Main Methods:

• An experiment, "Vertex Pursuit," was designed to collect synchronized hard and soft data.
• Participants' hand-eye coordination, dexterity, and precision were tracked.
• Traditional machine learning and advanced LLMs (encoder-only, decoder-only) with prompt engineering were applied to free-text feedback.

Main Results:

• Surgical skill assessment using soft data is a complex NLP task.
• Increasing method complexity generally improved assessment results.
• Decoder-only LLMs with rule-based prompting achieved the highest performance.

Conclusions:

• Free-text expert feedback shows feasibility as a standalone surgical skill assessment tool.
• Proposed NLP methods can reduce subjectivity and expert burden in skill evaluation.
• This approach enables more scalable and widespread surgical training assessment.