When possible, report a Fisher-exact P value and display its underlying null randomization distribution

Area of Science:

• Statistics
• Biostatistics
• Experimental Design

Background:

• Asymptotic P values are commonly reported in randomized experiments.
• Asymptotic P values can introduce irrelevant distributional assumptions, altering the research question.
• The Fisherian framework offers an alternative P value calculation based on the randomization procedure.

Purpose of the Study:

• To illustrate the Fisherian statistical framework in a crossover randomized experiment.
• To compare Fisher-exact P values with asymptotic P values for evaluating experimental results.
• To highlight the importance of the null randomization distribution in interpreting study outcomes.

Main Methods:

• Analysis of a crossover randomized experiment, considering the first period independently and then both periods.
• Focus on 10 outcomes to demonstrate differences between asymptotic and Fisher tests.
• Application of the Fisherian framework using the actual randomization procedure.

Main Results:

• For some outcomes, asymptotic P values were substantially lower than minimum attainable Fisher-exact P values.
• For other outcomes, the Fisher-exact null randomization distribution significantly differed from the assumed bell-shaped distribution.
• Differences highlight the impact of distributional assumptions in traditional P value calculations.

Conclusions:

• Fisher-exact P values should be preferred in randomized experiments, particularly for small sample sizes.
• Examining the shape of the null randomization distribution can yield valuable scientific insights.
• Using Fisher-exact P values ensures the P value directly addresses the question posed by the randomization procedure.