Related Experiment Video
Updated: Aug 10, 2025
Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack
Published on: May 15, 2020
Validation of Automated Data Extraction From the Electronic Medical Record to Provide a Pediatric Risk Assessment
Eleonore Valencia1, Steven J Staffa2, Yousuf Aslam2
1From the Departments of Cardiology.
Insights
An automated Pediatric Risk Assessment (PRAm) score shows good agreement with the original NSQIP-derived score, offering a feasible alternative to manual data entry for pediatric surgical risk stratification.
Area of Science:
- Pediatric Surgery
- Health Informatics
- Clinical Risk Assessment
Background:
- Pediatric postoperative mortality, though low, necessitates risk stratification using tools like the Pediatric Risk Assessment (PRAm) score.
- Manual data entry for PRAm scores can be inaccurate, inefficient, and increase physician workload.
- Electronic derivation of risk scores aims to improve accuracy and efficiency in clinical practice.
Purpose of the Study:
- To develop an electronically derived, automated PRAm score.
- To evaluate the agreement and correlation of the automated PRAm score with existing NSQIP-derived and manually entered scores.
- To assess the discriminatory ability of the automated PRAm score compared to other methods.
Main Methods:
- Retrospective observational study of pediatric patients (<18 years) undergoing noncardiac surgery (2017-2021).
- Automated PRAm score developed using International Classification of Disease (ICD) codes.
- Statistical analyses included Fleiss Kappa, Spearman correlation, ICC, and ROC curve analysis with AUC.
Main Results:
- Agreement and correlation were higher between automated and NSQIP-derived PRAm scores (rho=0.78, ICC=0.80) compared to manual and NSQIP scores (rho=0.73, ICC=0.78).
- The manual score demonstrated the highest discrimination (AUC=0.976), followed by NSQIP (AUC=0.904) and automated scores (AUC=0.880).
- The automated PRAm score showed good agreement and correlation with established methods.
Conclusions:
- An electronically derived, automated PRAm score is feasible for risk stratification in pediatric surgery.
- This automated approach can potentially reduce clerical workload and enhance efficiency.
- The automated PRAm score offers a promising tool for improving clinical outcomes and resource utilization in pediatric surgical care.
Background:
Although the rate of pediatric postoperative mortality is low, the development and validation of perioperative risk assessment models have allowed for the stratification of those at highest risk, including the Pediatric Risk Assessment (PRAm) score. The clinical application of such tools requires manual data entry, which may be inaccurate or incomplete, compromise efficiency, and increase physicians' clerical obligations. We aimed to create an electronically derived, automated PRAm score and to evaluate its agreement with the original American College of Surgery National Surgical Quality Improvement Program (ACS NSQIP)-derived and validated score.
Methods:
We performed a retrospective observational study of children <18 years who underwent noncardiac surgery from 2017 through 2021 at Boston Children's Hospital (BCH). An automated PRAm score was developed via electronic derivation of International Classification of Disease (ICD) -9 and -10 codes. The primary outcome was agreement and correlation among PRAm scores obtained via automation, NSQIP data, and manual physician entry from the same BCH cohort. The secondary outcome was discriminatory ability of the 3 PRAm versions. Fleiss Kappa, Spearman correlation (rho), and intraclass correlation coefficient (ICC) and receiver operating characteristic (ROC) curve analyses with area under the curve (AUC) were applied accordingly.
Results:
Of the 6014 patients with NSQIP and automated PRAm scores (manual scores: n = 5267), the rate of 30-day mortality was 0.18% (n = 11). Agreement and correlation were greater between the NSQIP and automated scores (rho = 0.78; 95% confidence interval [CI], 0.76-0.79; P <.001; ICC = 0.80; 95% CI, 0.79-0.81; Fleiss kappa = 0.66; 95% CI, 0.65-0.67) versus the NSQIP and manual scores (rho = 0.73; 95% CI, 0.71-0.74; P < .001; ICC = 0.78; 95% CI, 0.77-0.79; Fleiss kappa = 0.56; 95% CI, 0.54-0.57). ROC analysis with AUC showed the manual score to have the greatest discrimination (AUC = 0.976; 95% CI, 0.959,0.993) compared to the NSQIP (AUC = 0.904; 95% CI, 0.792-0.999) and automated (AUC = 0.880; 95% CI, 0.769-0.999) scores.
Conclusions:
Development of an electronically derived, automated PRAm score that maintains good discrimination for 30-day mortality in neonates, infants, and children after noncardiac surgery is feasible. The automated PRAm score may reduce the preoperative clerical workload and provide an efficient and accurate means by which to risk stratify neonatal and pediatric surgical patients with the goal of improving clinical outcomes and resource utilization.
