Type I error robustness of ANOVA and ANOVA on ranks when the number of treatments is large

Agricultural screening trials often involve a large number (t) of treatments in a complete block design with limited replication (b = 3 or 4 blocks). The null hypothesis of interest is that of no differences between treatments. For the commonly used analysis of variance (ANOVA) procedure, most texts do not discuss agreement between actual and nominal Type I error rates in the presence of nonnormality, in this small b, large t, situation. Similarly, for the Friedman and the increasingly popular "ANOVA on ranks" procedures, it is not easy to find results concerning null performance given b small and t large. In this article, we therefore present results, from two different bodies of theory, that provide useful insight concerning null performance of these ANOVA and rank procedures when t is large. The two types of theory are (i) the classical approach based on moment approximations to the permutation distribution, and (ii) central-limit-theory-based asymptotics in the nonstandard t--> infinity situation. Both approaches demonstrate the validity of standard ANOVA and of ANOVA on within-block ranks, under nonnormality when t is large. Choice of the procedure to be used on a given data set should therefore be based on consideration of power properties. In general, ANOVA on ranks will be superior to standard ANOVA for data with frequent extreme values.