Reading Grade Level Estimates of Public Health Communication on Social Media Vary Due to Text Preprocessing

Area of Science:

• Health Communication
• Computational Linguistics

Background:

• Readability formulas are widely used to assess health communication materials.
• These formulas can produce varying estimates due to differences in text preprocessing.
• This variability is often overlooked in computerized assessments.

Purpose of the Study:

• To illustrate the impact of text preprocessing decisions on readability estimates for short-form online content.
• To highlight the importance of reporting preprocessing methods and the limitations of readability formulas.
• To compare different preprocessing scenarios for health-related Tweets.

Main Methods:

• Manually counted words, sentences, and syllables in 100 Tweets from U.S. state health agencies (2012-2022).
• Applied Simplified Measure of Gobbledygook and Flesch-Kincaid formulas under 7 preprocessing scenarios, varying inclusion of URLs, hashtags, and numbers.
• Compared estimates against a restrictive baseline excluding these elements using Wilcoxon signed-rank tests and linear regression.

Main Results:

• Significant differences in median grade level estimates were found across preprocessing scenarios.
• No significant differences were observed in the proportion of Tweets meeting an 8th-grade readability benchmark.
• Baseline estimates did not fully account for the observed variation in readability scores.

Conclusions:

• Readability formulas are unreliable for assessing short-form online content like Tweets.
• Directly reporting word, sentence, and syllable counts is recommended over formula-based estimates.
• Conducting sensitivity analyses for readability assessments is advised to understand formula limitations.