Meta-analytic rain cloud plots: Improving evidence communication through data visualization design principles

Results of meta-analyses are of interest not only to researchers but often to policy-makers and other decision-makers (e.g., in education and medicine), and visualizations play an important role in communicating data and statistical evidence to the broader public. Therefore, the potential audience of meta-analytic visualizations is broad. However, the most common meta-analytic visualization - the forest plot - uses non-optimal design principles that do not align with data visualization best practices and relies on statistical knowledge and conventions not likely to be familiar to a broad audience. Previously, the Meta-Analytic Rain Cloud (MARC) plot has been shown to be an effective alternative to a forest plot when communicating the results of a small meta-analysis to education practitioners. However, the original MARC plot design was not well-suited for meta-analyses with large numbers of effect sizes as is common across the social sciences. This paper presents an extension of the MARC plot, intended for effective communication of moderate to large meta-analyses (k = 10, 20, 50, 100 studies). We discuss the design principles of the MARC plot, grounded in the data visualization and cognitive science literature. We then present the methods and results of a randomized survey experiment to evaluate the revised MARC plot in comparison to the original MARC plot, the forest plot, and a bar plot. We find that the revised MARC plot is more effective for communicating moderate to large meta-analyses to non-research audiences, offering a 0.30, 0.34, and 1.07 standard deviation improvement in chart users' scores compared to the original MARC plot, forest plot, and bar plot, respectively.