Related Experiment Video
Updated: Sep 3, 2026
Comparing Bibliometric Analysis Using PubMed, Scopus, and Web of Science Databases
Published on: October 24, 2019
Null studies in ophthalmology journals: a large-scale analysis of PubMed indexed research
Idan Hecht1, Ruti Sella2, Piotr Kanclerz3
1Department of Ophthalmology, Mayo Clinic, Rochester, Minnesota; Helsinki Retina Research Group, University of Helsinki, Helsinki, Finland.
Purpose:
Null studies, in which the primary outcome is not statistically significant, are essential for unbiased evidence-based medicine and help reveal selective dissemination of research. We estimated the prevalence of null studies and characterized differences between null and positive studies in ophthalmology journals.
Design:
Retrospective analysis of original studies published from January 1, 2014, through December 31, 2025.
Subjects:
Original studies published in 90 PubMed-indexed journals listed in the Ophthalmology category of the 2025 Journal Citation Reports.
Methods:
Study characteristics were derived using rule-based processing, except classification of the main study result as null versus positive, which used a large language model pipeline (gpt-4.1) applied to abstracts. Large language model performance was evaluated against a prespecified manual validation set of 500 abstracts, achieving kappa 0.87 (prespecified threshold 0.75).
Main Outcome Measures:
Prevalence and differences in study characteristics between null and positive studies.
Results:
A total of 93,963 studies were included, of which 19,231 (20.5%) were null studies. The proportion of null studies declined from 25.6% in 2014 to 15.1% in 2025 (p <0.001). Positive studies were more often funded than null studies (68.4% vs 58.0%; risk difference 95% CI 9.6 to 11.2; p<0.001) and were more often prospective in design (17.5% vs 16.0%; risk difference 95% CI 0.9 to 2.0; p<0.001), whereas randomized design was similarly common between groups (6.5% vs 6.5%, p=0.867). Positive studies were also more often drug-focused (13.6% vs 11.8%), device-focused (9.3% vs 6.9%), or surgery-focused (13.4% vs 9.3%) (all p<0.001). Null studies were published in journals with lower mean impact factor (2.59±1.62 vs 2.73±1.69; mean difference 95% CI 0.11 to 0.17; p<0.001), and worse mean journal rank (40.11 ± 22.6 vs 37.80 ± 22.2; mean difference 95% CI -2.67 to -1.95; p<0.001) compared to positive studies.
Conclusions:
In this large-scale analysis of ophthalmology journals, one in five studies had a null primary outcome. Overall, null studies became less common over time and were more often published in lower-impact, lower-ranked journals and were more likely to be unfunded. These findings suggest systematic differences in the characteristics and dissemination of null versus positive studies in ophthalmology.
