Replicability of studies following a dual-criterion design

Replicability of results is regarded as the corner stone of science. Recent research seems to raise doubts about whether this requirement is generally fulfilled. Often, replicability of results is defined as repeating a statistically significant result. However, since significance may not imply scientific relevance, dual-criterion study designs that take both aspects into account have been proposed and investigated during the last decade. Originally developed for proof-of-concept trials, the design could be appropriate for phase III trials as well. In fact, a dual-criterion design has been requested for COVID-19 vaccine applications by major health authorities. In this article, replicability of dual-criterion designs is investigated. It turns out that the probability to replicate a significant and relevant result can become as low as 0.5. The replication probability increases if the effect estimator exceeds the minimum relevant effect in the original study by an extra amount.