Exploratory Evaluation of Learning Behaviors Using a ChatGPT-Based Question-Generating Bot Among Occupational Therapy...

This exploratory study examined occupational therapy (OT) students' learning behaviors using a custom-built ChatGPT version 4.0 (OpenAI, Inc., San Francisco, United States) based question-generating bot. Particularly, this study examined changes in mock examination scores before and after one month of use, correlations between usage logs and score improvements, and learning behavior clusters to identify distinct engagement patterns.

This study comprised 29 OT students, using a one-group pre-post design. The intervention comprised one month of access to a ChatGPT-based bot, developed to generate multiple-choice questions in anatomy, kinesiology, and physiology. Participants were instructed to use the bot as part of their self-study and submit weekly reports via Google Forms (Google LLC, Mountain View, California, United States), documenting the number of questions generated, the number of sessions, and total usage time. Performance was assessed using two 50-item mock exams administered before (Exam I) and after (Exam II) the intervention. Paired t-tests were used to compare the exam scores. Correlations were calculated between the score gains (Exams II-I) and the learning logs. Ward's hierarchical cluster analysis was performed to explore the distinct patterns of learning behavior.

In total, 28 datasets were analyzed. Exam scores significantly improved between Exam I (17.1 ± 4.4) and Exam II (22.1 ± 7.6), with a mean gain of +5.0 ± 6.0 (t-statistic=4.40, degrees of freedom=27, p<0.001, Cohen's d=0.83). Correlation analysis indicated weak-to-moderate associations between Exam II scores and the number of questions (r=0.39, p<0.05) and sessions (r=0.36, p<0.10), but no significant association with total usage time. Cluster analysis identified three groups: Cluster 1 (n=14) with low activity and minimal gains; Cluster 2 (n=9) with more questions but little improvement; and Cluster 3 (n=6) with moderate activity but high total usage time, achieving the greatest improvements.

This exploratory study examined OT students' learning behaviors when using a ChatGPT-based question-generating bot. The findings suggest that increased total time investment may be more relevant to performance gains than the generated question quantity. Future studies should incorporate qualitative analyses of the learning processes and controlled comparisons to further validate these patterns.