Sample size and consensus methods used in Delphi studies are heterogeneous: meta-research study

Few studies have examined sample size and consensus methods in Delphi studies, leaving limited knowledge in the field of biomedicine. We aimed to analyze these methodological aspects in published articles in which the Delphi method was used.

This was a meta-research study. We analyzed open-access full-text articles in English, retrieved from PubMed, that reported results of primary studies that used a Delphi method. The analysis included the number of planned and conducted rounds, the planned sample size and justification for the sample size, the number of invited and included participants in each round, and the methods used to achieve consensus.

Analysis included 1875 articles describing 1904 Delphi studies. The description of the study design was provided in most articles (N = 1157; 62%). The word Delphi was mentioned in the title of 636 articles (34%), and the word consensus appeared in the title of 502 articles (27%). Only 19% of studies reported planned sample size, ranging in the overall sample from 5 to 420 participants (median 19-25). The rationale for their size was described in only 16% of the studies. Invited participants were usually reported for the first round, ranging in the overall sample from 3 to 7468 (median 20-37). The number of participants included was also most often stated for the first round, and in the overall sample, it ranged from 3 to 2295 (median 14-27.5). For all other Delphi rounds used in the studies, a decreasing number of participants was reported. Likert point scales were used in most studies (66%), most often as a single five-point scale. A percentage threshold for consensus was used in most studies, with 80% being the most common. Other numerical definitions most often used were the median, mean, and importance ranking.

Sample sizes and consensus methods vary significantly in studies using the Delphi method. Therefore, consensus guidelines on sample sizes and recommendations for consistent reporting of consensus criteria need to be developed.

Researchers often use the Delphi method to collect and combine opinions from experts, usually through several rounds of questionnaires, to reach agreement on important topics such as clinical guidelines or research priorities. However, there is little clear guidance on how many experts should take part in these studies or how consensus should be defined. In this study, we examined 1904 Delphi studies published in biomedical journals. We looked at how many participants the authors planned to include, how many they actually invited and retained in each round, and how they defined when experts had reached agreement. We found large differences between studies. Only about one in five studies reported how many participants they planned to include, and even fewer explained why they chose that number. The number of participants ranged from just a few to several hundred. Definitions of consensus also varied widely, with thresholds ranging from 50% to 100% agreement, most often 80%. These findings show that Delphi studies are conducted and reported in very different ways. To make Delphi research more transparent, comparable, and reliable, there is a need for consensus-based guidelines on how to plan, conduct, and report Delphi studies, including how to choose the number of participants and how to define consensus.