Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Cochran's Q Test01:17

Cochran's Q Test

230
Cochran's Q Test is a nonparametric statistical test used to determine if there are potential differences in the outcomes of three or more related groups on a binary (yes/no) or dichotomous outcome. It is essentially an extension of the McNemar Test, which is limited to two related samples - Cochran's Q test can handle three or more related samples, making it more versatile in scenarios where subjects are measured under multiple conditions. The test statistic follows a Chi-Square...
230
Group Design02:01

Group Design

8.9K
The most basic experimental design involves two groups: the experimental group and the control group. The two groups are designed to be the same except for one difference— experimental manipulation. The experimental group gets the experimental manipulation—that is, the treatment or variable being tested—and the control group does not. Since experimental manipulation is the only difference between the experimental and control groups, we can be sure that any differences between...
8.9K
Multiple Comparison Tests01:13

Multiple Comparison Tests

3.9K
Multiple comparison test, abbreviated as MCT, is a post hoc analysis generally performed after comparing multiple samples with one or more tests. An MCT will help identify a significantly different sample among multiple samples or a factor among multiple factors.
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
3.9K
Confirmation Biases01:31

Confirmation Biases

5.5K
The confirmation bias is the tendency to focus on information that confirms our existing beliefs and ignore information that is inconsistent with our expectations. For example, if you think that your professor is not very nice, you notice all of the instances of rude behavior exhibited by the professor while ignoring the countless pleasant interactions he is involved in on a daily basis. Have you ever fallen prey to the confirmation bias, either as the source or target of such bias?
5.5K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same authorSame journal

Multimodal Test Item Parameter Prediction From Text, Images, and Metadata: Fusing Together AI Vision and Language Models.

Educational and psychological measurement·2026
See all related articles

Related Experiment Video

Updated: Jun 7, 2025

Author Spotlight: Validation of SICOLE-R for Assessing Cognitive and Reading Skills in Spanish-Speaking Children and Its Role in Personalized Education
09:00

Author Spotlight: Validation of SICOLE-R for Assessing Cognitive and Reading Skills in Spanish-Speaking Children and Its Role in Personalized Education

Published on: August 16, 2024

707

Field-Testing Multiple-Choice Questions With AI Examinees: English Grammar Items.

Hotaka Maeda1,2

  • 1Smarter Balanced, Santa Cruz, CA, USA.

Educational and Psychological Measurement
|November 18, 2024
PubMed
Summary

This study introduces using artificial intelligence (AI) examinees for field-testing educational assessments, showing promising results for item analysis and calibration. While not as accurate as human data, AI field-testing offers significant potential for cost and time savings in assessment development.

Keywords:
BERTartificial intelligencenatural language processingpretestingquestion difficulty prediction

More Related Videos

Experimental Paradigm for Measuring the Effect of Induced Emotion on Grammar Learning
05:33

Experimental Paradigm for Measuring the Effect of Induced Emotion on Grammar Learning

Published on: January 29, 2020

6.0K
Examining Online Syntactic Processing of Spoken Complex Sentences in Chinese Using Dual-Modal Interference Tasks
08:32

Examining Online Syntactic Processing of Spoken Complex Sentences in Chinese Using Dual-Modal Interference Tasks

Published on: September 5, 2019

5.6K

Related Experiment Videos

Last Updated: Jun 7, 2025

Author Spotlight: Validation of SICOLE-R for Assessing Cognitive and Reading Skills in Spanish-Speaking Children and Its Role in Personalized Education
09:00

Author Spotlight: Validation of SICOLE-R for Assessing Cognitive and Reading Skills in Spanish-Speaking Children and Its Role in Personalized Education

Published on: August 16, 2024

707
Experimental Paradigm for Measuring the Effect of Induced Emotion on Grammar Learning
05:33

Experimental Paradigm for Measuring the Effect of Induced Emotion on Grammar Learning

Published on: January 29, 2020

6.0K
Examining Online Syntactic Processing of Spoken Complex Sentences in Chinese Using Dual-Modal Interference Tasks
08:32

Examining Online Syntactic Processing of Spoken Complex Sentences in Chinese Using Dual-Modal Interference Tasks

Published on: September 5, 2019

5.6K

Area of Science:

  • Educational Measurement
  • Artificial Intelligence in Education
  • Psychometrics

Background:

  • Field-testing educational assessments is crucial but resource-intensive.
  • Current methods rely on human examinees, leading to significant time and cost investments.
  • The development of high-quality assessments necessitates efficient and effective field-testing procedures.

Purpose of the Study:

  • To introduce and evaluate an innovative method for field-testing educational assessment items using artificially intelligent (AI) examinees.
  • To demonstrate the feasibility of using AI examinees to generate item response data comparable to human examinees.
  • To explore the potential of AI in streamlining the assessment development lifecycle and reducing associated costs.

Main Methods:

  • Fine-tuning pre-trained transformer language models based on the 2-parameter logistic (2PL) item response model to simulate human test-taker behavior.
  • Utilizing AI examinees, each assigned a latent ability (θ), to predict response selection probabilities for multiple-choice English grammar questions.
  • Assessing the performance of the AI examinee model by comparing true and predicted 2PL correct response probabilities.

Main Results:

  • The best AI modeling approach achieved a correlation of .82 between true and predicted 2PL correct response probabilities (bias = 0.00, RMSE = 0.18).
  • AI-generated item response data demonstrated utility for calculating item proportion correct, item discrimination, conducting item calibration with anchors, distractor analysis, dimensionality analysis, and latent trait scoring.
  • The AI approach did not reach the accuracy levels of human examinee response data.

Conclusions:

  • AI examinees offer a viable and promising alternative for field-testing educational assessments, capable of generating data for various psychometric analyses.
  • While current AI models do not fully replicate human examinee accuracy, the potential for substantial resource savings in terms of time, cost, and logistical concerns is enormous.
  • Further refinement of AI models could revolutionize the field-testing process, enabling faster assessment development, improved item bank expansion, and enhanced test security.