Related Experiment Video
Updated: Mar 4, 2026
Measuring the Functional Abilities of Children Aged 3-6 Years Old with Observational Methods and Computer Tools
Published on: June 20, 2020
Development and Performance Validation of an Automated Generative Pretrained Transformer-Based Evaluation Tool for
Yu-Jeng Ju1, Yi-Ching Wang1, Fan Chou2
1School of Occupational Therapy, College of Medicine, National Taiwan University, Taipei, Taiwan.
Objectives:
To develop an automated generative pretrained transformer (GPT)- physiotherapy evidence database (PEDro) evaluation tool (GPT-PEDro ET) and validate its performance (including intrarater reliability and concurrent validity) for evaluating the methodological quality of RCTs using the PEDro scale.
Design:
A psychometric validation study with a repeated measurements design.
Setting:
Research laboratory.
Participants:
One hundred and twenty-five RCTs on neurofacilitation interventions in stroke rehabilitation were retrieved from the PEDro database.
Interventions:
Not applicable.
Main Outcomes And Measures:
The primary outcome was RCT quality scores evaluated using the GPT-PEDro ET tool at both total score and individual item levels across 2 rounds of evaluation. Intrarater reliability was determined using the intraclass correlation coefficient (ICC) for total score and prevalence-adjusted and bias-adjusted κ (PABAK) for individual items. Concurrent validity was assessed with ICC (total score) and PABAK (individual item), Bland-Altman analysis with 95% limits of agreement, and heteroscedasticity testing.
Results:
The GPT-PEDro ET tool achieved almost perfect intrarater reliability (total score ICC=1.00; individual item PABAK=0.94-1.00). The concurrent validity was moderate to high at both the total score level (ICCs=0.83, 0.83) and the individual item level for all the items (PABAKs=0.68-0.97).
Conclusions:
The results suggest that the GPT-PEDro ET tool may be a useful automated tool for evaluating the methodological quality of RCTs. It shows potential for reducing manual evaluation workload and supporting clinical and research applications.
