Updated: May 28, 2026

Artificial Intelligence-Based System for Detecting Attention Levels in Students
Published on: December 15, 2023
Burhan Dost1, Engin I Turan2, Esra Turunc1
1Department of Anesthesiology and Reanimation, Ondokuz Mayis University Faculty of Medicine, Samsun, Türkiye.
You might also read
Articles linked to this work by shared authors, journal, and citation graph.
This study evaluated how well artificial intelligence models can predict a patient's preoperative risk level, known as the American Society of Anesthesiologists (ASA) physical status, compared to human doctors. By analyzing data from over 400,000 cases, researchers found that these digital tools show moderate to strong agreement with clinical assessments. Large language models performed better than older machine learning methods, suggesting these technologies could eventually assist doctors in evaluating surgical risk.
Area of Science:
Background:
The American Society of Anesthesiologists physical status classification serves as a primary tool for preoperative risk evaluation. This system relies on subjective judgment, which frequently leads to inconsistent results among different medical professionals. No prior work had resolved the exact reliability of automated prediction tools in this specific clinical context. That uncertainty drove the need for a rigorous synthesis of existing diagnostic performance data. Prior research has shown that machine learning approaches are increasingly applied to automate various medical scoring tasks. However, the overall agreement between these computational models and human assessors remains poorly defined. This gap motivated a comprehensive investigation into whether digital systems can match traditional clinical expertise. The current landscape of perioperative risk stratification requires clearer evidence regarding the utility of these emerging technologies.
Purpose Of The Study:
The researchers report a pooled quadratic weighted kappa of 0.69, indicating moderate-to-substantial agreement. This metric quantifies the concordance between automated predictions and human-assigned classifications across the included datasets.
The authors utilized the PROBAST+AI tool to evaluate study quality, while the GRADE framework was applied to determine the certainty of the evidence. These methods ensure a standardized assessment of the included research.
The meta-regression analysis revealed that studies mixing adult and pediatric patient data exhibited lower performance. This suggests that demographic heterogeneity may challenge the predictive accuracy of these models.
The aim of this systematic review and meta-analysis was to quantitatively synthesize current evidence regarding the diagnostic performance of digital ASA classification systems. Researchers sought to address the uncertainty surrounding the reliability of automated models compared to human judgment. The study specifically investigated the magnitude of agreement between computational predictions and clinician-assigned statuses. This initiative was motivated by the limitations of current subjective preoperative risk assessment methods. By aggregating data from multiple databases, the authors intended to clarify the potential utility of these tools. The project focused on identifying whether machine learning systems could serve as dependable adjuncts in clinical practice. Understanding the performance of these models is necessary for their eventual integration into surgical workflows. This work provides a rigorous evaluation of the current state of automated risk stratification technology.
Main Methods:
Review Approach involved a systematic search of five major databases including PubMed and Scopus from inception through October 2025. Investigators identified studies comparing automated predictions against human-assigned classifications in real or simulated datasets. The team applied the PROBAST+AI tool to evaluate the quality of every included publication. Certainty of evidence was determined using the GRADE framework to ensure rigorous synthesis. Statistical analysis utilized random-effects models to pool the primary outcome of quadratic weighted kappa. Meta-regression was conducted to explore factors influencing model performance across the diverse studies. The researchers calculated secondary outcomes including sensitivity, specificity, and overall accuracy for all models. This structured methodology allowed for a comprehensive quantitative assessment of current diagnostic performance.
Main Results:
Key Findings From the Literature show a pooled quadratic weighted kappa of 0.69, reflecting moderate-to-substantial agreement between models and clinicians. The analysis of 402,336 cases across thirteen studies provided the basis for these results. Pooled accuracy for the systems reached 0.66, while sensitivity was 0.51 and specificity was 0.78. Subgroup analysis revealed that large language models achieved significantly higher agreement than traditional machine learning models. This performance difference was statistically significant with a P-value of 0.04. Meta-regression indicated that combining adult and pediatric data resulted in lower predictive performance. The data demonstrate that digital systems can effectively replicate human risk stratification to a meaningful degree. These results highlight the varying capabilities of different computational architectures in clinical settings.
Conclusions:
Synthesis and Implications indicate that artificial intelligence systems achieve moderate-to-substantial concordance with human clinicians when assigning risk classifications. The authors propose that these tools could function as reliable adjuncts for preoperative stratification. Large language models demonstrated superior performance compared to traditional machine learning approaches in this meta-analysis. The researchers suggest that combining diverse patient populations, such as adults and children, may negatively impact model performance. These findings support the integration of digital models into clinical workflows to assist with risk assessment tasks. The evidence suggests that automated systems provide a consistent alternative to subjective human evaluation. Future implementation should focus on the specific benefits of advanced language-based models in perioperative settings. This review provides a foundation for understanding the diagnostic capabilities of current computational risk assessment tools.
The study analyzed 402,336 cases across thirteen distinct investigations. This large sample size provides a robust foundation for evaluating the diagnostic performance of the automated systems.
Large language models showed significantly higher agreement with clinicians compared to traditional machine learning models, with a P-value of 0.04. This indicates a statistically significant performance advantage for newer language-based architectures.
The authors propose that these systems could serve as reliable adjuncts for preoperative risk stratification. They suggest that the observed concordance supports the potential utility of digital tools in clinical practice.