An AI-Assisted Adaptive Boolean Rubric for exercise prescription evaluation: A pilot validation study

The quality assessment of personalized exercise prescriptions is currently hampered by the subjectivity and inefficiency of traditional rating scales. Artificial intelligence (AI) presents a transformative opportunity for objective, scalable evaluation.

This pilot study aimed to develop an AI-assisted evaluation framework and assess the feasibility, reliability, and efficiency of its core component, the Adaptive Precision Boolean Rubric (Adaptive-PBR).

Based on ACSM guidelines, we developed a 50-item Precision Boolean Rubric (PBR) and a 10-item Likert scale. To maximize ecological validity, we utilized GPT-4 (via ChatGPT Plus Web Interface) with a dual-run consistency protocol to generate case-specific, 20-item Adaptive-PBRs. Twelve experts evaluated five diverse clinical cases (yielding 180 rating points) using all three instruments under randomized conditions.

The Adaptive-PBR demonstrated excellent inter-rater reliability (ICC = 0.83), significantly outperforming the Likert scale (ICC = 0.65) and matching the full PBR (ICC = 0.82). Quantitatively, it achieved high scoring precision (Median 0.78, IQR 0.70-0.85) while reducing evaluation time by approximately 63 % (mean 7.1 vs. 19.5 min) compared to the full PBR. Crucially, the Adaptive-PBR mitigated the subjective variability and experience-related bias observed with the Likert scale.

The AI-assisted Adaptive-PBR establishes a feasible, reliable, and efficient evaluation standard. By combining granular criteria with AI-driven adaptability, it offers a robust alternative to subjective scales, with immediate potential as a quality assurance tool in clinical training and practice.