Item Difficulty Modeling Using Fine-tuned Small and Large Language Models.

Area of Science:

• Educational Measurement
• Natural Language Processing
• Machine Learning

Background:

• Item difficulty modeling is crucial for large-scale assessments.
• Existing methods face challenges with data imbalance and feature extraction.
• Evaluating the efficacy of small and large language models (LLMs) for this task is needed.

Purpose of the Study:

• To investigate and enhance item difficulty modeling using advanced language models.
• To develop and validate novel data augmentation strategies.
• To compare the performance of small language models (SLMs) and LLMs in predicting item difficulty.

Main Methods:

• Implementation of novel data augmentation techniques: augmentation on the fly and distribution balancing.
• Fine-tuning of SLMs (BERT, RoBERTa) and evaluation of domain-specific models (BioClinicalBERT, PubMedBERT).
• Exploration of LLM (GPT-4) capabilities with chain-of-thought prompting and rationale generation; utilization of embedding-based methods (NV-Embed-v2).

Main Results:

• Augmentation strategies significantly improved performance, outperforming benchmarks and mitigating data imbalance.
• Fine-tuned SLMs achieved lower root mean squared error than the top model in the BEA 2024 Shared Task.
• LLMs showed generalization but struggled with difficulty prediction; ensemble learning with SLMs enhanced accuracy.

Conclusions:

• Novel data augmentation strategies are highly effective for item difficulty modeling.
• Fine-tuned SLMs, particularly through ensemble methods, offer superior performance over LLMs for this specific task.
• Further research is needed to improve LLM performance, potentially through increased training data or advanced reasoning techniques.