A Neutral Comparison of Multivariate Hypothesis Tests for Parallel Group Designs in Rare Disease Settings

Multivariate endpoints are increasingly used in clinical studies on rare diseases, where sample sizes tend to be small and distributional assumptions are often violated. In such settings, commonly applied parametric multivariate methods, such as classical MANOVA approaches, may suffer from inflated type-I error or reduced statistical power. Therefore, a neutral simulation-based comparison of four methodologically different approaches for the analysis of multivariate outcomes in parallel group designs was conducted. The four approaches included the nonparametric R package npmv (using permutation and randomization resampling for small samples), a semiparametric MANOVA approach from the R package MANOVA.RM with both a Wald-type statistics (WTS) and a modified ANOVA-type statistics (MATS) using all available variants including also bootstrap approaches, a classical parametric MANOVA, and multiple univariate Mann-Whitney tests with Holm correction. The simulation framework was informed by empirical distributions derived from anonymized clinical data and clinical expert guidance. Two clinically realistic effect types were mainly evaluated: (i) percent-responder effects for wound size reduction and (ii) additional location shifts in pain and pruritus visual analog scales (VAS) scores. In addition, scenarios are examined in which classical assumptions such as multivariate normality and homogeneity of covariance matrices are violated. The sample sizes considered ranged from 12 to 360 subjects. Both balanced and unbalanced designs were evaluated, with the objective of illustrating potential limitations of methods when applied to small samples. Outcomes were assessed in terms of type-I error control and statistical power. Overall, npmv, the MATS using especially parametric bootstrap within the semiparametric MANOVA.RM package maintained reliable type-I error control, whereas the semiparametric MANOVA.RM approach with the WTS using its asymptotic approximation showed very liberal type-I error behavior to almost no type-I error control in small sample scenarios, and especially in unbalanced settings. Bootstrap approaches especially of the WTS substantially improved type-I error control compared with the asymptotic WTS. The nonparametric npmv approach achieved good power values, especially in small samples, as did the modified ATS approach using parametric bootstrap from the MANOVA.RM package, which achieved very similar results. The univariate Mann-Whitney U test with Holm correction tended to yield conservative results for small samples; however, despite the univariate nature of this approach, it achieved a high and comparable level of statistical power in many scenarios. Classical MANOVA also yielded good results in certain scenarios; however, it exhibited rather liberal type-I error rates, primarily in heteroscedastic scenarios and unbalanced designs, which can be explained by a violation of the assumption of homoscedastic covariance matrices. These results highlight the strengths and limitations of a multi-aspect analysis in parallel group designs especially for small sample settings, as common in rare diseases, and provide guidance for the analysis in rare disease research.