Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Controls in Experiments01:13

Controls in Experiments

18.7K
When conducting an experiment, it is crucial to have control to reduce bias and accurately measure the dependent variables. It also marks the results more reliable. Controls are elements in an experiment that have the same characteristics as the treatment groups but are not affected by the independent variable. By sorting these data into control and experimental conditions, the relationship between the dependent and independent variables can be drawn. A randomized experiment always includes a...
18.7K
Reliability and Validity01:29

Reliability and Validity

14.3K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
14.3K
Random and Systematic Errors01:20

Random and Systematic Errors

15.7K
Scientists always try their best to record measurements with the utmost accuracy and precision. However, sometimes errors do occur. These errors can be random or systematic. Random errors are observed due to the inconsistency or fluctuation in the measurement process, or variations in the quantity itself that is being measured. Such errors fluctuate from being greater than or less than the true value in repeated measurements. Consider a scientist measuring the length of an earthworm using a...
15.7K
Random and Systematic Errors01:20

Random and Systematic Errors

900
900
Uncertainty in Measurement: Accuracy and Precision03:37

Uncertainty in Measurement: Accuracy and Precision

113.2K
Scientists typically make repeated measurements of a quantity to ensure the quality of their findings and to evaluate both the precision and the accuracy of their results. Measurements are said to be precise if they yield very similar results when repeated in the same manner. A measurement is considered accurate if it yields a result that is very close to the true or the accepted value. Precise values agree with each other; accurate values agree with a true value. 
113.2K
What is an Experiment?01:12

What is an Experiment?

19.6K
An experiment is a planned activity carried out under controlled conditions. The purpose of an experiment is to investigate the relationship between two variables. When one variable causes change in another, we call the first variable the explanatory or independent variable. The affected variable is called the response or dependent variable. In a randomized experiment, the researcher manipulates values of the explanatory variable and measures the resulting changes in the response variable. The...
19.6K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Representing objects and features in long-term memory: A case for direct feature-feature binding.

Journal of experimental psychology. Learning, memory, and cognition·2026
Same author

Assessing qualitative individual differences with Bayesian hierarchical latent-mixture models.

Psychological methods·2026
Same author

Random-effects psychophysics for studying individual differences in perception and cognition.

Psychonomic bulletin & review·2026
Same author

Localizing structure in individual differences: A visual illusion case study.

Attention, perception & psychophysics·2025
Same author

Estimating correlations in low-reliability settings with constrained hierarchical models.

Behavior research methods·2025
Same author

On aggregation invariance of multinomial processing tree models.

Behavior research methods·2024
Same journal

Proficiency order invariance of MLE, MAP, EAP, and WLE in item response theory.

The British journal of mathematical and statistical psychology·2026
Same journal

Bias and precision in true-score estimation.

The British journal of mathematical and statistical psychology·2026
Same journal

Polychoric correlations under the assumption of elliptical latent traits.

The British journal of mathematical and statistical psychology·2026
Same journal

Regularized reduced rank regression for mixed predictor and response variables.

The British journal of mathematical and statistical psychology·2026
Same journal

A multiple-choice SDT model for cognitive diagnosis models.

The British journal of mathematical and statistical psychology·2026
Same journal

Modular item response and structural equation modelling via measurement and uncertainty preserving parametric modelling.

The British journal of mathematical and statistical psychology·2026
See all related articles

Related Experiment Video

Updated: Mar 17, 2026

Operant Procedures for Assessing Behavioral Flexibility in Rats
08:30

Operant Procedures for Assessing Behavioral Flexibility in Rats

Published on: February 15, 2015

21.7K

The role of reliability in experiments.

Jeffrey N Rouder1, Mahbod Mehrvarz1, Martin Schnuerch2

  • 1University of California, Irvine, Irvine, California, USA.

The British Journal of Mathematical and Statistical Psychology
|March 16, 2026
PubMed
Summary
This summary is machine-generated.

Psychology experiment reliability measures are complicated by sample size elements. This study proposes three analysis levels—foundational, intermediate, and final—to better assess experimental properties and improve data interpretation.

Keywords:
experimental designhierarchical modelsindividual differencesreliability

More Related Videos

Reliable Mechanochemistry: Protocols for Reproducible Outcomes of Neat and Liquid Assisted Ball-mill Grinding Experiments
13:05

Reliable Mechanochemistry: Protocols for Reproducible Outcomes of Neat and Liquid Assisted Ball-mill Grinding Experiments

Published on: January 23, 2018

11.2K
A Within-Subject Experimental Design using an Object Location Task in Rats
09:28

A Within-Subject Experimental Design using an Object Location Task in Rats

Published on: May 6, 2021

5.3K

Related Experiment Videos

Last Updated: Mar 17, 2026

Operant Procedures for Assessing Behavioral Flexibility in Rats
08:30

Operant Procedures for Assessing Behavioral Flexibility in Rats

Published on: February 15, 2015

21.7K
Reliable Mechanochemistry: Protocols for Reproducible Outcomes of Neat and Liquid Assisted Ball-mill Grinding Experiments
13:05

Reliable Mechanochemistry: Protocols for Reproducible Outcomes of Neat and Liquid Assisted Ball-mill Grinding Experiments

Published on: January 23, 2018

11.2K
A Within-Subject Experimental Design using an Object Location Task in Rats
09:28

A Within-Subject Experimental Design using an Object Location Task in Rats

Published on: May 6, 2021

5.3K

Area of Science:

  • Psychology
  • Experimental Design
  • Statistical Analysis

Background:

  • Reliability analysis in psychology experiments is often overemphasized.
  • Experimental sample size involves both number of individuals and replicate trials, complicating traditional reliability measures.
  • Existing reliability metrics may not adequately capture the nuances of experimental data.

Purpose of the Study:

  • To propose a multi-level analytical framework for psychology experiments.
  • To differentiate between foundational task properties, trial-level variability, and combined individual/trial sample size effects.
  • To advocate for a more comprehensive approach to assessing experimental reliability.

Main Methods:

  • Distinguishing three levels of analysis: foundational (task properties), intermediate (number of trials), and final (individuals and trials).
  • Utilizing example statistics like intraclass correlation (foundational) and reliability (intermediate).
  • Highlighting the utility of hierarchical models for integrating these analytical levels.

Main Results:

  • Reliability, as commonly used, represents an intermediate level of analysis, focusing on trials but not individuals.
  • Foundational analysis captures task properties independent of sample size.
  • A final level considers both individuals and trials, exemplified by uncertainty in correlation coefficients.

Conclusions:

  • Researchers should consider all three analytical levels for a thorough understanding of experimental data.
  • Reliability measures alone are insufficient for communicating foundational task properties or interpreting correlations.
  • Hierarchical models are crucial for implementing a comprehensive, multi-level analysis in psychological research.