High social media attention scores are not reflective of study quality: an altmetrics-based content analysis

Recent literature has demonstrated the associations between social media attention, as measured by altmetric attention score (AAS), and higher citation rates across medical disciplines. Despite increasing use of AAS, an understanding of factors associated with higher AAS and social media attention remains lacking. Furthermore, if this increased attention correlates with a higher methodological quality and lower biases has not been determined. Therefore, the purpose of the current study was to determine the relationship between methodological quality, study biases and the AAS in randomized controlled trials (RCTs). All RCTs from 2016 in the New England Journal of Medicine (NEJM), Journal of the American Medical Society (JAMA), and Lancet were extracted and the (1) AAS; (2) Methodological Bias (JADAD Scale); Study Bias (Cochrane Risk-of-Bias tool for RCTs) recorded. A total of 296 RCTs with a median (range) AAS and citation rate per article of 234.0(7-4079) and 165.0(4-3257), respectively, were included. The AAS was positively associated with citation rate (β 0.19, 95% CI 0.10-0.29; P < 0.001). Methodological bias was not associated with the AAS (β - 36.3, 95% CI - 83.5-10.9; P = 0.131), but was negatively associated with higher citation rates (β - 66.4, 95% CI - 106.0 to - 26.9; P = 0.001). The number of study biases was not associated with the AAS (β 43.7, 95% CI - 6.3-93.7;P = 0.086), but was positively associated with a higher citation rate (β 64.5, 95% CI 22.4-106.6; P = 0.003). The online attention of RCTs in medical journals was not necessarily reflective of high methodological quality and minimal study biases, but was associated with higher citation rates. Researchers and clinicians should critically examine each article despite the amount of online attention an article receives as the AAS does not necessarily reflect article quality.