Assessing Message Deployment During Public Health Emergencies Through Social Media: Empirical Test of Optimizing...

During an infodemic, timely, reliable, and accessible information is crucial to combat the proliferation of health misinformation. While message testing can provide vital information to make data-informed decisions, traditional methods tend to be time- and resource-intensive. Recognizing this need, we developed the rapid message testing at scale (RMTS) approach to allow communicators to repurpose existing social media advertising tools and understand the full spectrum of audience engagement.

We had two main objectives: (1) to demonstrate the use of the RMTS approach for message testing, especially when resources and time are limited, and (2) to propose and test the efficacy of an outcome variable that measures engagement along a continuum of viewing experience.

We developed 12 versions of a single video created for a vaccine confidence project in India. We manipulated video length, aspect ratio, and use of subtitles. The videos were tested across 4 demographic groups (women or men, younger or older). We assessed user engagement along a continuum of viewing experience: obtaining attention, sustaining attention, conveying the message, and inspiring action. These were measured by the percentage of video watched and clicks on the call-to-action link.

The video advertisements were placed on Facebook for over 4 consecutive days at the cost of US $450 and garnered a total of 3.34 million impressions. Overall, we found that the best-performing video was the shorter version in portrait aspect ratio and without subtitles. There was a significant but small association between the length of the video and users' level of engagement at key points along the continuum of viewing experience (N=1,032,888; χ²₄=48,261.97; P<.001; V=.22). We found that for the longer video, those with subtitles held viewers longer after 25% video watch time than those without subtitles (n=15,597; χ²₁=7.33; P=.007; V=.02). While we found some significant associations between the aspect ratio, the use of subtitles, and the number of users watching the video and clicking on the call-to-action link, the effect size for those were extremely small.

This test served as a proof of concept for the RMTS approach. We obtained rapid feedback on formal message attributes from a very large sample. The results of this test reinforce the need for platform-specific tailoring of communications. While our data showed a general preference for a short video in portrait orientation and without subtitles among our target audiences on Facebook, that may not necessarily be the case in other social media platforms such as YouTube or TikTok, where users go primarily to watch videos. RMTS testing highlights nuances that communication professionals can address instead of being limited to a "one size fits all" approach.