Related Experiment Video
Updated: Jun 26, 2025
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
547
Comparing and assessing four AI chatbots' competence in economics
Patrik T Hultberg1, David Santandreu Calonge2, Firuz Kamalov3
1Department of Economics and Business, Kalamazoo College, Kalamazoo, Michigan, United States of America.
Plos One
|May 8, 2024
Summary
Artificial Intelligence (AI) chatbots like GPT-3.5, GPT-4, Bard, and LLaMA 2 show varying accuracy in economics. Performance differences increase with prompt complexity, impacting academic use.
Area of Science:
- Economics
- Artificial Intelligence
- Educational Technology
Background:
- AI chatbots offer academic support but have variable accuracy.
- Differences in response accuracy and explanation quality exist among AI models.
- Understanding AI chatbot capabilities is crucial for academic integrity and learning.
Purpose of the Study:
- To evaluate the accuracy and explanation quality of four AI chatbots (GPT-3.5, GPT-4, Bard, LLaMA 2) in university-level economics.
- To compare chatbot performance across different levels of cognitive complexity using Bloom's taxonomy.
- To determine if AI chatbot performance varies significantly with increasing economics prompt complexity.
Main Methods:
- Utilized a standard economics test and advanced problems for university-level assessment.
- Applied Bloom's taxonomy to structure prompts and evaluate cognitive complexity.
- Compared the accuracy of conclusions and quality of explanations from four distinct AI chatbots.
Main Results:
- The null hypothesis of equal performance was rejected, indicating significant differences among chatbots.
- AI chatbot performance varied notably in accuracy and explanation quality for economics.
- Performance disparities widened as the complexity of economics prompts increased.
Conclusions:
- AI chatbots exhibit significant performance variations in economics education.
- Prompt complexity is a key factor influencing AI chatbot effectiveness.
- Findings inform students in selecting appropriate AI tools and educators in curriculum design.
More Related Videos
Related Concept Videos
01:16Non-equilibrium in the Cell
4.4K
An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...
4.4K
01:29Econometric Views (EViews)
141
Econometric Views, often stylized as EViews, is a package that merges statistical analysis with econometric studies. It is designed to provide tools for time series analysis, forecasting, and econometric model simulation. The software originated from MicroTSP software and has evolved significantly since its inception in 1981. The history of EViews is marked by a continuous effort to enhance its computational speed and user interface. It was initially developed for large computing systems but...
141
01:20Introduction to Cognitive Psychology
474
Cognitive psychology is the field of psychology dedicated to examining how people think. It attempts to explain how and why we think the way we do by studying the interactions among human thinking, emotion, creativity, language, and problem-solving, as well as other cognitive processes. Cognitive psychology studies how information is processed and manipulated in remembering, thinking, and knowing.
This field emerged in the mid-20th century, following a period dominated by behaviorism, which...
This field emerged in the mid-20th century, following a period dominated by behaviorism, which...
474
01:08The Availability Heuristic
6.0K
A heuristic is a general problem-solving framework (Tversky & Kahneman, 1974). You can think of these as mental shortcuts that are used to solve problems. Different types of heuristics are used in different types of situations, and the impulse to use a heuristic occurs when one of five conditions is met (Pratkanis, 1989):
6.0K
01:14Decision Making: Traditional Method
4.0K
The process of hypothesis testing based on the traditional method includes calculating the critical value, testing the value of the test statistic using the sample data, and interpreting these values.
First, a specific claim about the population parameter is decided based on the research question and is stated in a simple form. Further, an opposing statement to this claim is also stated. These statements can act as null and alternative hypotheses, out of which a null hypothesis would be a...
First, a specific claim about the population parameter is decided based on the research question and is stated in a simple form. Further, an opposing statement to this claim is also stated. These statements can act as null and alternative hypotheses, out of which a null hypothesis would be a...
4.0K
02:16Stereotype Content Model
14.7K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
14.7K