Significance testing of many variables. Problems and solutions

More than one variable of interest in pharmaco-EEG studies is the rule rather than the exception. The many-variables situation may arise from several types of EEG variables and/or from repeated measurements of these variables through time. In many cases, multivariate analysis techniques are not suitable for application. As an alternative, the analyst usually applies individual (univariate) tests for all variables and/or time points. This significance testing of many variables causes problems because of the so-called 'alpha-inflation', i.e. the inflation of the probability for the 'error of the first kind' to reject a null hypothesis although it is valid. To counteract this effect (which invalidates the significance levels used for hypothesis testing), various procedures have been proposed some of which are discussed in the present paper. All procedures involve so-called 'alpha-adjustments', and two procedures are based upon the assumption that the investigator demands that at least a given percentage of all individual null hypotheses considered will be rejected or are not valid, respectively.