Comparison of NLP machine learning models with human physicians for ASA Physical Status classification.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• The American Society of Anesthesiologist's Physical Status (ASA-PS) classification system is crucial for assessing patient comorbidities.
• Subjectivity and inter-rater variability in ASA-PS assessments hinder objective clinical decision-making.
• Automating ASA-PS classification could standardize pre-anesthesia evaluations.

Purpose of the Study:

• To develop and validate natural language processing (NLP) models for objective ASA-PS classification.
• To compare the performance of NLP models against board-certified anesthesiologists.
• To assess the feasibility of using NLP for streamlining clinical workflows.

Main Methods:

• Utilized a large dataset of 717,389 surgical cases (October 2004-May 2023).
• Trained and tuned NLP models including ClinicalBigBird, BioClinicalBERT, and Generative Pretrained Transformer 4.
• Validated NLP models against reference labels created by board-certified anesthesiologists.

Main Results:

• The ClinicalBigBird NLP model achieved an Area Under the Receiver Operating Characteristic Curve (AUC) of 0.915.
• ClinicalBigBird demonstrated superior performance compared to anesthesiologists in specificity (0.901 vs. 0.897), precision (0.732 vs. 0.715), and F1-score (0.716 vs. 0.713).
• All performance metrics showed statistically significant improvements (p < 0.01).

Conclusions:

• NLP models, particularly ClinicalBigBird, offer a robust and objective method for ASA-PS classification.
• Automated ASA-PS classification using NLP can enhance accuracy and efficiency in clinical settings.
• This technology has the potential to significantly streamline pre-anesthesia evaluation workflows.