Predicting Procedure Step Performance From Operator and Text Features: A Critical First Step Toward Machine...

Area of Science:

• Engineering
• Human Factors
• Computer Science

Background:

• Procedures are critical for safety in high-risk industries.
• Current guidelines lack quantitative analysis for diverse influencing factors.
• Subjective assessments limit the integration of performance-influencing variables.

Purpose of the Study:

• To assess machine learning models for predicting procedure performance.
• To identify key operator and procedure characteristics influencing performance.
• To develop data-driven approaches for procedure design.

Main Methods:

• Utilized a 25-participant, four-procedure oil extraction simulation.
• Developed and compared logistic regression (LR), random forest (RF), and decision tree (DT) algorithms.
• Employed Boruta feature selection and 10-fold cross-validation for model optimization.

Main Results:

• RF, DT, and LR models achieved AUCs of 0.78, 0.77, and 0.75, respectively.
• Models significantly outperformed LR with operator features only (AUC 0.61).
• Key predictors included operator experience, familiarity, and text-based metrics (word/character counts).

Conclusions:

• Machine learning offers a promising method for predicting step-level procedure performance.
• Models can guide procedure design, but require further validation.
• Text characteristics like brevity correlate with correct step execution.