A Large Language Model Approach to Functional Status Scale Assessment.

Area of Science:

• Artificial Intelligence in Medicine
• Pediatric Critical Care
• Health Informatics

Background:

• Estimating functional status in critically ill children is crucial for care planning and outcomes assessment.
• The Functional Status Scale (FSS) is a common tool, but manual scoring can be time-consuming and subjective.
• Leveraging advanced AI like GPT-4o offers potential for automated and efficient FSS scoring.

Purpose of the Study:

• To develop and evaluate a fine-tuned generative pretrained transformer (GPT)-4o artificial intelligence (AI) model, named FSS-AI, for estimating FSS scores in critically ill children.
• To assess the accuracy and agreement of AI-generated FSS scores compared to manually assigned scores across different hospitalization timepoints.

Main Methods:

• A secondary analysis of a prospective cohort of mechanically ventilated children (1 month to 18 years) was performed.
• Patient notes from baseline, PICU transfer, and hospital discharge were used to train and test the custom GPT-4o model (FSS-AI).
• FSS-AI performance was evaluated by comparing its generated scores against prospectively determined manual FSS scores using weighted Cohen's Kappa and accuracy metrics.

Main Results:

• FSS-AI analyzed 428 patient notes, demonstrating moderate agreement with manual FSS scores at baseline (Kappa=0.59) and discharge (Kappa=0.51).
• The model showed good discrimination between normal and abnormal FSS scores, with highest accuracy (0.90) and PPV (0.95) at baseline.
• FSS-AI identified new morbidities at discharge with 75% accuracy and 56% sensitivity.

Conclusions:

• A custom GPT-4o model (FSS-AI) can effectively estimate FSS scores in critically ill children at various hospitalization stages.
• The AI tool shows moderate agreement with manual scoring and can differentiate between normal and abnormal functional status.
• FSS-AI demonstrates potential as an efficient tool for assessing functional status and identifying new morbidities in pediatric critical care settings.