Related Experiment Video
Updated: Jun 28, 2025

Project-Based Learning Guidelines for Health Sciences Students: An Analysis with Data Mining and Qualitative Techniques
Published on: December 9, 2022
Exploring the use of ChatGPT to analyze student course evaluation comments
Kathryn A Fuller1, Kathryn A Morbitzer1,2, Jacqueline M Zeeman1
1UNC Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
Artificial intelligence (AI) tools like ChatGPT can efficiently analyze student course evaluations, showing high agreement with instructor findings and significantly reducing analysis time. While beneficial, AI should supplement, not replace, human judgment in educational assessments.
Area of Science:
- Educational Technology
- Artificial Intelligence in Education
- Higher Education Assessment
Background:
- Artificial intelligence (AI) tools, notably ChatGPT, present emerging applications in higher education.
- AI can potentially reduce faculty workload, particularly in time-consuming tasks like analyzing open-ended course evaluation comments.
- Current methods for extracting insights from qualitative feedback in course evaluations are often labor-intensive.
Purpose of the Study:
- To investigate the efficacy of ChatGPT in analyzing open-ended student course evaluation comments.
- To compare the time efficiency of AI-driven analysis versus traditional instructor analysis.
- To assess the level of agreement between themes identified by instructors and those generated by ChatGPT.
Main Methods:
- Instructors independently performed thematic analysis on student course evaluation comments using predefined prompts.
- Two independent ChatGPT user accounts analyzed the same comments using similar prompts.
- Quantitative measures, including percent agreement and time taken, were calculated for both methods.
Main Results:
- High agreement was observed between instructor-identified themes and ChatGPT-generated themes (course-related topics: 0.71-0.82).
- ChatGPT analysis was significantly faster, averaging 10.50-12.50 minutes compared to instructors' 27.50 minutes.
- ChatGPT accounts showed higher inter-rater reliability than agreement with instructors for most prompts.
Conclusions:
- ChatGPT demonstrates significant potential as an assistive tool for analyzing qualitative data in health professions education course evaluations.
- The AI tool offers a time-efficient alternative for theme generation, with substantial agreement with human analysis.
- It is recommended to use AI as a supplementary tool, integrating its outputs with expert judgment rather than relying on it exclusively.
More Related Videos
Related Concept Videos
Comparing Experimental Results: Student's t-Test
Student t Distribution
The Student t distribution was developed by William S. Goset (1876–1937) of the...
Statistical Analysis: Overview
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
Study Design in Statistics
Does aspirin reduce the risk of heart attacks? Is one brand of fertilizer more effective at growing roses than another? Is fatigue as dangerous to a driver as the influence of alcohol? Questions like these are answered using randomized experiments with proper...
Self-Evaluation: Self-Enhancement and Self-Verification
Statistical Methods to Analyze Parametric Data: Student t-Test and Goodness-of-Fit Test
The Student's t-test is a statistical test that examines if there is a statistically significant difference between the means of two groups. This test is instrumental when dealing with...

