Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Know when to trust: Making AI scoring more reliable for educational assessment.

Peter Organisciak1, Selcuk Acar2

  • 1University of Denver, 1999 E Evans Ave, Denver, CO, 80208, USA. peter.organisciak@du.edu.

Behavior Research Methods
|May 28, 2026
PubMed
Summary

This study enhances automated scoring using large language models (LLMs) with three new methods. These improvements increase the trustworthiness and accuracy of LLM-based educational scoring tools.

Related Concept Videos

Reliability and Validity01:29

Reliability and Validity

Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
Measures of Intelligence01:29

Measures of Intelligence

Psychologists measure intelligence by using standardized tests that produce a score known as the intelligence quotient or IQ. To understand IQ tests, it's important to recognize the key principles behind their construction: validity, reliability, and standardization.
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this; it...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Evaluating differences in latent means across studies: Extending meta-analytic confirmatory factor analysis with the analysis of means.

Research synthesis methods·2026
Same author

Measuring flexibility: A text-mining approach.

Frontiers in psychology·2023
Same author

Clustering and switching in divergent thinking: Neurophysiological correlates underlying flexibility during idea generation.

Neuropsychologia·2021
Same author

Schizophrenia and creativity: A meta-analytic review.

Schizophrenia research·2017

Area of Science:

  • Artificial Intelligence
  • Educational Measurement
  • Psychometrics

Background:

  • Large language models (LLMs) offer new possibilities for automated scoring in education.
  • Existing LLM scoring tools require enhancements for improved trustworthiness and accuracy.

Purpose of the Study:

  • To introduce and evaluate three novel improvements for LLM-based automated scoring tools.
  • To enhance the reliability and validity of LLM-generated scores in educational contexts.

Main Methods:

  • Utilized data from over 20,000 responses to the Alternative Uses Test from 2,000+ participants.
  • Implemented and assessed three techniques: model self-confidence, weighted probabilistic scoring, and ensemble modeling.
  • Evaluated improvements by comparing LLM scores with human judges' assessments.
Keywords:
Divergent thinking tasksEducational assessmentEnsemble modelingLarge language modelsModel self-confidenceWeighted probabilistic scoring

Related Experiment Videos

Main Results:

  • All three introduced techniques demonstrated statistically significant positive results.
  • Improvements were observed in correlation with human judges (from r=0.781 to r=0.823) and error reduction.
  • The methods proved compatible as drop-in improvements for existing scoring techniques.

Conclusions:

  • Model self-confidence, weighted probabilistic scoring, and ensemble modeling enhance LLM-based automated scoring.
  • These adjustments boost the dependability and applicability of LLMs for quantitative text analysis in education.
  • The findings support the use of LLMs for more trustworthy educational scoring.