Sample size in quantitative instrument validation studies: A systematic review of articles published in Scopus, 2021

Due to the range of conflicting criteria regarding minimum sample size needed for a scale/questionnaire validation study, the objective of this review is to analyze sample sizes used in published journal articles to contribute a pragmatic perspective to the discussion on sample sizes.

A sample of 1999 articles published in a Scopus-indexed journal about the validation of a scale or questionnaire during 2021 were analyzed for this study. Abstracts from these articles were tabulated by two data entry professionals and any discrepancies were reviewed by the author. The sample size data was grouped by highest quartile of the journal publishing the article and further sub-categorized based on the inclusion of medical patients or students in each study's population.

From the total sample, 1750 articles provided sufficient information in their summary to determine the sample size used. Of these, the majority were published in quartile 1 (784) and quartile 2 (620) journals. Mean values by quartile ranged from 389 (quartile 3) to 2032 (quartile 1), but extreme outliers limited the usefulness of the simple mean. Thus, outlier-removed means were calculated, and in most cases, these sample size values were higher for studies involving students and lower for studies involving patients.

This study is limited by its focus on a single database and by including all phases of validation from initial quantitative instrument design studies (which tend to have the lowest sample sizes) up to international macro-studies (which can have hundreds of thousands of participants.) Nevertheless, the results of this study provide an additional practical perspective for the academic discussion regarding minimum sample size based on accepted practice.