Inferences about competing measures based on patterns of binary significance tests are questionable

Area of Science:

• Psychological measurement
• Statistical inference
• Research methodology

Background:

• Demonstrating the validity of new measures is crucial, often involving incremental validity testing against existing measures.
• Regression methods and binary significance tests (BST) are commonly used to argue for a new measure's incremental validity.
• The traditional BST approach relies on specific patterns of statistical significance when measures are jointly tested.

Purpose of the Study:

• To evaluate the reliability of arguments for incremental validity based on binary significance tests (BST).
• To identify the conditions under which BST can lead to erroneous conclusions in validity studies.
• To propose and illustrate alternative, more robust methods for assessing measure validity and incremental contribution.

Main Methods:

• Analysis of statistical inference in regression models, specifically focusing on binary significance testing (BST).
• Simulation or examination of scenarios with modest statistical power to assess the rate of false conclusions from BST.
• Application of alternative inferential methods to real-world data, using construal level as a case study.

Main Results:

• Binary significance test (BST) arguments for incremental validity can result in incorrect conclusions in up to 30% of cases, particularly with modest statistical power.
• The 'black and white' approach of relying solely on significance/non-significance is shown to be misleading.
• Alternative methods provide stronger and more accurate inferences about measure validity and added information.

Conclusions:

• Relying on binary significance tests for incremental validity can lead to substantial error rates.
• Researchers should adopt more nuanced statistical approaches for evaluating new measures.
• Accurate assessment of measure validity and incremental contribution is essential for advancing psychological science.