A Reproducible Survey-Scoring And C5.0 Decision-Tree Workflow For Classifying Self-Reported Higher-Order Thinking In...

Area of Science:

• Educational Technology
• Artificial Intelligence in Education
• Data Science in Education

Background:

• Generative artificial intelligence (AI) adoption in higher education necessitates transparent methods for analyzing learner data.
• Existing approaches lack standardized procedures for scoring and organizing survey data related to AI use in academic settings.

Purpose of the Study:

• To present a reproducible survey-scoring and C5.0 decision-tree workflow for classifying self-reported higher-order thinking (HOT) among college students using generative AI.
• To demonstrate a protocol for learner profiling in AI-supported educational environments.

Main Methods:

• A C5.0 decision-tree workflow was developed for classifying higher-order thinking (HOT) based on survey data.
• The protocol included participant recruitment, data collection, response screening, score calculation, and model construction.
• A dataset of 776 undergraduate students in China was used for demonstration, with HOT operationalized as a self-reported questionnaire score.

Main Results:

• The C5.0 model identified eight key learner variables influencing HOT, including AI anxiety, trust, smartphone use, procrastination, academic performance, upbringing, emotions, and attitudes.
• The model achieved 89.52% accuracy on the training subset and 86.21% on the testing subset.
• Class-specific evaluation revealed weaker recall for the 'Low-HOT' class compared to the 'High-HOT' class.

Conclusions:

• The developed workflow provides an interpretable, auxiliary tool for survey-based learner profiling in generative AI contexts.
• The model's uneven performance necessitates caution, suggesting it as a demonstration rather than a validated screening tool.
• This protocol supports researchers needing repeatable procedures for analyzing learner data in AI-integrated education.