Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Crossover Experiments01:16

Crossover Experiments

Crossover experiments, also called the repeated-measurements design, is a study design in which all experimental units are exposed to all treatments in different periods. Crossover experiments are generally used in psychology, the pharmaceutical industry, agriculture, and medicine.
Crossover designs are performed even with smaller sample sizes since the samples can act as their controls. These are better than simple randomized trials since patients are exposed to all the treatments.
Bioequivalence Experimental Study Designs: Repeated Measures, Cross-Over, Carry-Over, and Latin Square Designs01:15

Bioequivalence Experimental Study Designs: Repeated Measures, Cross-Over, Carry-Over, and Latin Square Designs

Bioequivalence experimental study designs play a pivotal role in testing the effectiveness of various treatments. Key among these are the repeated measures, cross-over, carry-over, and Latin square designs. In the repeated measures design, each subject receives all treatments, allowing for temporal comparisons. This type of design is useful in reducing variability but requires careful planning to avoid bias.The cross-over design, an economical method, involves sequential administration of...
Group Design02:01

Group Design

The most basic experimental design involves two groups: the experimental group and the control group. The two groups are designed to be the same except for one difference— experimental manipulation. The experimental group gets the experimental manipulation—that is, the treatment or variable being tested—and the control group does not. Since experimental manipulation is the only difference between the experimental and control groups, we can be sure that any differences between the two are due to...
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches01:23

Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches

Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...
Bias in Epidemiological Studies01:29

Bias in Epidemiological Studies

Biases can arise at various stages of research, from study design and data collection to analysis and interpretation. Recognizing and addressing these biases is essential to ensure the validity and reliability of epidemiological findings.Broadly speaking, biases in epidemiology fall into three main categories: selection bias, information bias, and confounding. A more detailed description of possible biases is:
Confounding in Epidemiological Studies01:27

Confounding in Epidemiological Studies

Confounding in statistical epidemiology represents a pivotal challenge, referring to the distortion in the perceived relationship between an exposure and an outcome due to the presence of a third variable, known as a confounder. This variable is associated with both the exposure and the outcome but is not a direct link in their causal chain. Its presence can lead to erroneous interpretations of the exposure's effect, either exaggerating or underestimating the true association. This phenomenon...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Comparison of on-site versus NOAA's extreme precipitation intensity-duration-frequency estimates for six forest headwater catchments across the continental United States.

Stochastic environmental research and risk assessment : research journal·2026
Same author

Shared decision-making in general practice: Development of a complex health care intervention targeting multiple long-term conditions.

Patient education and counseling·2026
Same author

Longitudinal, Multi-Cycle Evaluation of Passive Function Improvement in People with Arm Spasticity Treated with Botulinum Toxin A.

Toxins·2026
Same author

A symptom network approach to schizophrenia in the CATIE study: processing speed as the central cognitive impairment.

BJPsych open·2026
Same author

A Watershed-Scale Multi-Approach Assessment of Design Flood Discharge Estimates used in Hydrologic Risk Analyses for Forest Road Stream Crossings and Culverts.

Journal of hydrology·2026
Same author

The NEuroCOUGH Chronic Cough Registry: a protocol for a pan-European observational study.

ERJ open research·2025
Same journal

When I say … 'in situ simulation'.

Medical education·2026
Same journal

Examiner training and calibration for simulated clinical examinations: A scoping review.

Medical education·2026
Same journal

When systems set the limits of supervision.

Medical education·2026
Same journal

From psychometrics to partnerships: Broadening what counts as validity evidence.

Medical education·2026
Same journal

When I say the clinical digital divide.

Medical education·2026
Same journal

Urology education under fire: Quantitative benchmarking of trainer adaptation and competency preservation in wartime Sudan.

Medical education·2026
See all related articles

Related Experiment Video

Updated: Jun 13, 2026

Probing the Limits of Egg Recognition Using Egg Rejection Experiments Along Phenotypic Gradients
07:34

Probing the Limits of Egg Recognition Using Egg Rejection Experiments Along Phenotypic Gradients

Published on: August 22, 2018

Generalisability in unbalanced, uncrossed and fully nested studies.

Ajit Narayanan1, Michael Greco, John L Campbell

  • 1School of Computing and Mathematical Sciences, Auckland University of Technology, Auckland, New Zealand.

Medical Education
|May 7, 2010
PubMed
Summary
This summary is machine-generated.

New formulas allow for accurate estimation of reliability and generalizability (G) coefficients in complex health care professional performance data. These methods can predict the minimum number of raters needed, even with unbalanced studies.

More Related Videos

A Within-Subject Experimental Design using an Object Location Task in Rats
09:28

A Within-Subject Experimental Design using an Object Location Task in Rats

Published on: May 6, 2021

Related Experiment Videos

Last Updated: Jun 13, 2026

Probing the Limits of Egg Recognition Using Egg Rejection Experiments Along Phenotypic Gradients
07:34

Probing the Limits of Egg Recognition Using Egg Rejection Experiments Along Phenotypic Gradients

Published on: August 22, 2018

A Within-Subject Experimental Design using an Object Location Task in Rats
09:28

A Within-Subject Experimental Design using an Object Location Task in Rats

Published on: May 6, 2021

Area of Science:

  • Psychometrics
  • Health Professions Education
  • Statistical Modeling

Background:

  • Multi-source, multi-level feedback is increasingly used for healthcare professional performance assessment.
  • Existing methods struggle with unbalanced, uncrossed, and fully nested data common in such assessments.
  • Accurate estimation of true score variance among professionals is challenging under these data conditions.

Purpose of the Study:

  • To introduce extensions to reliability and generalizability (G) formulas for handling complex data structures in performance feedback.
  • To develop decision (D) formulas for predicting the minimum number of raters required in unbalanced studies.
  • To demonstrate the feasibility and relevance of these new statistical approaches using artificial and real-world datasets.

Main Methods:

  • Developed extensions to G-theory formulas to accommodate unbalanced, uncrossed, and fully nested data.
  • Incorporated variances from raters, ratees, and items at multiple analysis levels.
  • Created D-formulas for predicting minimum rater numbers in unbalanced studies.
  • Validated D-formula predictions against actual G-coefficients using independent datasets.
  • Introduced a combined G-coefficient formula for multi-sourced reliability estimation.

Main Results:

  • The proposed formulas successfully estimate reliability and generalizability in complex, unbalanced datasets.
  • Extraneous variance components can be identified and removed to refine true score variance estimation.
  • Validation confirmed the ability to accurately predict the minimum number of raters needed, even for unbalanced studies.

Conclusions:

  • Calculating G and D coefficients is feasible for psychometric data from doctor performance feedback, even with unbalanced, uncrossed, and fully nested structures.
  • Accurate calculations require separating variance at rater and ratee levels.
  • Utilizing the average number of raters per ratee is crucial for deriving reliable coefficients.