Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Multiple Comparison Tests01:13

Multiple Comparison Tests

Multiple comparison test, abbreviated as MCT, is a post hoc analysis generally performed after comparing multiple samples with one or more tests. An MCT will help identify a significantly different sample among multiple samples or a factor among multiple factors.
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
Significance Testing: Overview01:04

Significance Testing: Overview

Significance testing is a set of statistical methods used to test whether a claim about a parameter is valid. In analytical chemistry, significance testing is used primarily to determine whether the difference between two values comes from determinate or random errors. The effect of a particular change in the measurement protocol, analyst, or sample itself can cause a deviation from the expected result. In the case of a suspected deviation/outlier, we need to be able to confirm mathematically...
Comparing the Survival Analysis of Two or More Groups01:20

Comparing the Survival Analysis of Two or More Groups

Survival analysis is a cornerstone of medical research, used to evaluate the time until an event of interest occurs, such as death, disease recurrence, or recovery. Unlike standard statistical methods, survival analysis is particularly adept at handling censored data—instances where the event has not occurred for some participants by the end of the study or remains unobserved. To address these unique challenges, specialized techniques like the Kaplan-Meier estimator, log-rank test, and Cox...
Bonferroni Test01:10

Bonferroni Test

The Bonferroni test is a statistical test named after Carlo Emilio Bonferroni, an Italian mathematician best known for Bonferroni inequalities. This statistical test is a type of multiple comparison test to determine which means are different than the rest. Bonferroni test can minimize the Type 1 error by reducing the significance level alpha, which otherwise increases with sample pairs.
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
Quantifying and Rejecting Outliers: The Grubbs Test01:02

Quantifying and Rejecting Outliers: The Grubbs Test

Sometimes, a data set can have a recorded numerical observation that greatly  deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier.  To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This number is...
Test for Homogeneity01:23

Test for Homogeneity

The goodness–of–fit test can be used to decide whether a population fits a given distribution, but it will not suffice to decide whether two populations follow the same unknown distribution. A different test, called the test for homogeneity, can be used to conclude whether two populations have the same distribution. To calculate the test statistic for a test for homogeneity, follow the same procedure as with the test of independence. The hypotheses for the test for homogeneity can be stated as...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Error, power, and cluster separation rates of pairwise multiple testing procedures.

Psychological methods·2013
Same author

Controlling the false discovery rate with constraints: the Newman-Keuls test revisited.

Biometrical journal. Biometrische Zeitschrift·2007
Same author

Multiplicity, directional (type III) errors, and the null hypothesis.

Psychological methods·2002
See all related articles

Related Experiment Video

Updated: Jun 28, 2026

Large-scale Reconstructions and Independent, Unbiased Clustering Based on Morphological Metrics to Classify Neurons in Selective Populations
12:27

Large-scale Reconstructions and Independent, Unbiased Clustering Based on Morphological Metrics to Classify Neurons in Selective Populations

Published on: February 15, 2017

Cluster formation as a measure of interpretability in multiple testing.

Juliet Popper Shaffer1

  • 1University of California, Department of Statistics, 367 Evans Hall 3860, Berkeley, CA 94720-3860, USA. shaffer@stat.berkeley.edu

Biometrical Journal. Biometrische Zeitschrift
|October 22, 2008
PubMed
Summary

This study introduces new measures for evaluating multiple testing procedures based on their ability to identify distinct groups of treatments. It compares methods by their potential for cluster separation, offering a novel perspective beyond traditional error control and power metrics.

More Related Videos

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments
08:12

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

Published on: March 1, 2022

Related Experiment Videos

Last Updated: Jun 28, 2026

Large-scale Reconstructions and Independent, Unbiased Clustering Based on Morphological Metrics to Classify Neurons in Selective Populations
12:27

Large-scale Reconstructions and Independent, Unbiased Clustering Based on Morphological Metrics to Classify Neurons in Selective Populations

Published on: February 15, 2017

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments
08:12

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

Published on: March 1, 2022

Area of Science:

  • Statistics
  • Multiple Hypothesis Testing
  • Data Analysis

Background:

  • Traditional multiple testing procedures primarily focus on error control and statistical power.
  • Existing methods often overlook the pattern of rejected hypotheses, which can be crucial for interpreting results, especially in treatment comparisons.
  • Cluster analysis and model selection implicitly consider patterns, but a formal framework for evaluating this aspect in multiple testing is lacking.

Purpose of the Study:

  • To introduce novel measures for assessing the cluster separation potential of multiple testing procedures.
  • To evaluate how effectively different methods can divide treatments into distinct subsets based on population parameters (P-clusters).
  • To compare the performance of established procedures like Benjamini-Hochberg (BH) and Newman-Keuls (NK) using these new pattern-based measures.

Main Methods:

  • Development of new quantitative measures to assess the probability of separating treatments into outcome clusters.
  • Focus on the number and accuracy (true vs. false) of identified outcome clusters.
  • Comparison of the Benjamini-Hochberg (BH) and Newman-Keuls (NK) procedures using these cluster separation metrics.

Main Results:

  • The study proposes a new framework for analyzing multiple testing outcomes beyond simple error rates and power.
  • It highlights that different procedures, even with similar false discovery rate (FDR) control, can yield distinct patterns of treatment grouping.
  • The proposed measures quantify the potential of methods to reveal underlying population structures (P-clusters) through sample data.

Conclusions:

  • The pattern of hypothesis rejections, specifically cluster separation potential, is a critical, yet often overlooked, aspect of multiple testing.
  • New measures offer a more comprehensive evaluation of multiple testing procedures, aiding in the interpretation of results for practical applications.
  • Understanding cluster separation is vital for selecting appropriate methods in fields like treatment comparisons where grouping is key.