Automating sedation state assessments using natural language processing.

Area of Science:

• Medical Informatics
• Natural Language Processing
• Clinical Decision Support

Background:

• Procedural sedation aims to control pain and patient movement for safe procedure completion.
• Clinicians assess sedation adequacy for titration and documentation.
• NLP offers potential for real-time classification of sedation states from audio recordings.

Purpose of the Study:

• To evaluate the accuracy of NLP algorithms in detecting sedation states during procedures.
• To determine if NLP can reliably classify movement and pain during procedural sedation.

Main Methods:

• A prospective observational study analyzed transcribed audio recordings from 82 procedures.
• Multiple NLP pipelines were evaluated, including Bag-of-Words (BOW), an ensemble model, and a transformer-based RoBERTa model.
• Text classification performance was assessed using metrics like AUC-ROC, F1 score, precision, and recall.

Main Results:

• The RoBERTa model achieved the highest performance with an AUC-ROC of 0.97 and an F1 score of 0.87.
• The ensemble model showed strong performance (AUC-ROC 0.96), while the BOW model had lower scores.
• All evaluated NLP models demonstrated varying degrees of accuracy in classifying sedation states.

Conclusions:

• Transformer-based NLP models, specifically RoBERTa, show significant promise for accurate sedation state classification.
• Automated assessments via NLP can enhance documentation efficiency and reduce clinician burden.
• Further research is needed to validate real-time application and explore downstream clinical benefits.