David vs. Goliath: comparing conventional machine learning and a large language model for assessing students' concept...

Area of Science:

• Educational research
• Artificial intelligence in education
• Physics education

Background:

• Large language models (LLMs) show promise for educational research and assessment.
• LLMs have limitations including knowledge hallucination, lack of explainability, and high resource costs.
• Conventional machine learning (ML) offers more researcher control but optimal use cases with LLMs are unclear.

Purpose of the Study:

• To compare the performance of conventional ML algorithms against a large language model in assessing student physics concept use.
• To determine the optimal circumstances for employing conventional ML versus LLMs in educational assessment.

Main Methods:

• A physics problem-solving task was used to assess students' concept application.
• Conventional machine learning algorithms were implemented and compared to a large language model.
• Model classifications were analyzed to understand decision-making processes.

Main Results:

• Combined conventional machine learning algorithms demonstrated superior performance compared to the large language model.
• Analysis of model decisions provided insights into classification differences.
• Conventional ML proved more effective in this specific student assessment context.

Conclusions:

• Conventional machine learning algorithms can outperform large language models in specific educational assessment tasks.
• Conventional ML can effectively supplement LLMs, especially when labeled data is available.
• Choosing between conventional ML and LLMs depends on specific research needs and data availability.