Robustness of statistical methods when measure is affected by ceiling and/or floor effect

Area of Science:

• Statistics
• Measurement Theory
• Data Analysis

Background:

• Ceiling and floor effects (CFE) are common in data collection, potentially distorting statistical analyses.
• Understanding CFE's impact on various statistical tests is crucial for accurate data interpretation.
• Existing statistical methods may not adequately address the challenges posed by CFE.

Purpose of the Study:

• To investigate the performance of eight statistical methods under varying degrees of ceiling and floor effects (CFE).
• To evaluate CFE's influence on effect size estimates, confidence intervals, and equivalence testing.
• To explore the utility of data transformations and measurement-theoretic models in mitigating CFE.

Main Methods:

• Simulation study examining Welch's t-test, F-test, Mann-Whitney, Kruskal-Wallis, Scheirer-Ray-Hare, trimmed t-test, Bayesian t-test, and two one-sided tests.
• Utilized probability distributions derived from the principle of maximum entropy to manipulate CFE magnitude.
• Applied parametric methods to log- or logit-transformed data and analyzed data using Wald distribution and ordered logistic regression.

Main Results:

• Bias and uncertainty generally increased with CFE, leading to more certain but biased inferences.
• Data transformation improved performance, with specific transformations (log, logit) showing effectiveness for certain distributions.
• CFE hindered detection of main effects and interactions in factor designs; F-test frequently misidentified effects.

Conclusions:

• Generalized linear models based on abstract measurement theory are recommended to address CFE.
• Measure validation and calibration studies are essential for selecting appropriate statistical tools when CFE is present.
• Rank-based tests performed well on discrete data, but measurement-theoretic models may offer superior performance.