Real Customization or Just Marketing: Are Customized Versions of Generative AI Useful?

Area of Science:

• Artificial Intelligence in Education
• Natural Language Processing Applications
• Educational Technology

Background:

• Large Language Models (LLMs) are transforming higher education, with customization enabling tailored student support.
• OpenAI's recent feature allows natural language interface-based customization of GPT models for specific tasks.
• This facilitates the creation of specialized AI tutors for subjects like statistics.

Purpose of the Study:

• To assess the potential of customized GPTs in higher education.
• To evaluate and compare the performance of a custom-built Business Statistics Virtual Professor (BSVP) against ChatGPT-4 Turbo.
• To determine if customized LLMs offer significant advantages over general-purpose models for statistics education.

Main Methods:

• Developed a Business Statistics Virtual Professor (BSVP) for university students.
• Collected authentic student questions from statistics courses and posed them to both BSVP and ChatGPT-4 Turbo.
• Blindly evaluated responses based on quality, depth, and personalization, followed by statistical comparison.

Main Results:

• The customized BSVP exhibited a more relatable and friendly communication style, including humor.
• BSVP provided superior responses to specific requests, such as programming exercises in R.
• No statistically significant differences were found in overall performance, quality, depth, or course alignment between BSVP and ChatGPT-4 Turbo.

Conclusions:

• Customized AI assistants, trained with specific prompts, offer benefits as virtual aids for students.
• However, these specialized models do not represent a substantial improvement over general models like ChatGPT-4 Turbo.
• Further research is needed to fully understand the impact of customized LLMs in educational settings.