Generative Pre-trained Transformer for Pediatric Stroke Research: A Pilot Study

Area of Science:

• Neurology
• Artificial Intelligence
• Medical Informatics

Background:

• Pediatric stroke is a significant cause of childhood morbidity.
• The International Pediatric Stroke Study (IPSS) has collected extensive data through collaborative efforts.
• Research in pediatric stroke is often challenged by data collection complexities.

Purpose of the Study:

• To explore the utility of Generative Pre-trained Transformer (GPT), an advanced artificial intelligence program, for entering pediatric stroke data into the IPSS.
• To assess the accuracy and efficiency of GPT in data entry compared to human raters.

Main Methods:

• Fifty deidentified clinical notes from pediatric ischemic stroke or cerebral venous sinus thrombosis patients were used.
• Domain-specific prompts were engineered for an offline GPT model to answer IPSS database questions.
• GPT responses were compared against human raters, with percent agreement assessed for 114 queries.

Main Results:

• GPT performance varied, with initial iterations showing occasional human-level accuracy (17.5% of 20 patients).
• After four prompt refinement iterations, GPT achieved 93.6% agreement in data entry.
• The model correctly entered 2247 out of 2400 assessed items (93.6%).

Conclusions:

• A tailored generative AI model with domain-specific prompts shows potential for research data entry in pediatric stroke.
• Further refinement is necessary to improve GPT's accuracy for independent data entry.
• GPT can be utilized collaboratively with human oversight to reduce research effort.