American Society of Anesthesiologists Physical Status Classification: Hybrid Machine Learning-Large Language Model Ensemble Retrospective Validation Study

The American Society of Anesthesiologists Physical Status (ASA-PS) classification system fundamentally shapes perioperative care delivery but suffers from poor interrater reliability (0.4-0.6). Machine learning (ML) models and large language models (LLMs) offer complementary remedies: ML processes structured data consistently but cannot interpret the nuance of clinical narratives, while LLMs parse narrative text and generate explanations but adhere inconsistently to diagnostic criteria.

This study aimed to develop and evaluate a parallel ML-LLM ensemble that combines the complementary strengths of both approaches for automated ASA-PS classification.

We retrospectively analyzed 3000 adult surgical encounters from the University of Arkansas for Medical Sciences (UAMS) between August 2024 and May 2025. Cases were randomly allocated to training (n=2000), validation (N=500), and held-out test sets (N=500). The validation set was used exclusively for architecture comparison and hyperparameter tuning, and the held-out test set was reserved for final model performance validation. We developed multiple architectures, including traditional ML models (Extreme Gradient Boosting [XGBoost], Light Gradient Boosting Machine [LightGBM], ExtraTrees), an LLM-only baseline (Generative Pretrained Transformer-4o [GPT-4o]), and hybrid approaches. The parallel ensemble processed structured data through XGBoost and unstructured clinical notes through GPT-4o independently, with outputs combined via weighted averaging. Model performance was evaluated using macro-F₁-score, exact match accuracy, and ±1-class accuracy.

A theoretical performance ceiling of macro-F₁-score=0.59 was determined through evaluation by an expert panel of 2 board-certified anesthesiologists (AF and GF) who independently rated 50 patient charts for ASA-PS scores. On the true held-out test set (N=500), the parallel ensemble (α=0.30) achieved a macro-F₁-score of 0.530 (95% CI 0.350-0.600), exact match accuracy of 0.630 (95% CI 0.580-0.670), and ±1-class accuracy of 0.980 (95% CI 0.968-0.992), exceeding the frozen LLM-only baseline evaluated on the same set (macro-F₁-score 0.493, exact match accuracy 0.621,±1-class accuracy 0.981). On the validation set used for architecture selection, the ensemble outperformed structured-only ML (XGBoost macro-F₁-score=0.333, P<.001), and the LLM-only model reached a macro-F₁-score of 0.642 (95% CI 0.525-0.691), nominally exceeding the expert panel ceiling. Per-class analysis attributed this to a perfect score on the 2 validation ASA-PS 5 cases. Excluding that class, the adjusted macro-F₁-score was approximately 0.55, within the range of the parallel ensemble. The LLM component also generated explanations detailing comorbidities, severity descriptors, and functional status indicators.

The parallel ML-LLM ensemble achieved performance approaching the theoretical ceiling established by the expert panel benchmark while providing interpretable clinical explanations. The ±1-class accuracy of 0.980 (95% CI 0.968-0.992) may enhance operational safety by minimizing extreme misclassifications. This approach demonstrates how complementary AI architectures can enhance perioperative risk assessment, which is particularly valuable given current health care workforce shortages.