An Efficient Message Evaluation Protocol: Two Empirical Analyses on Positional Effects and Optimal Sample Size

Reliable and valid message evaluation has a central role in effective health communication and message effects research. The authors have employed a message testing protocol to efficiently acquire valid and reliable message evaluation results: (a) use multiple messages, (b) recruit evaluators from the target population, (c) use valid and reliable effectiveness measures, (d) expose an evaluator to multiple messages, and (e) ensure enough evaluations per message. Two secondary analyses of anti-tobacco message evaluation studies provide evidence for reliability and validity regarding points (d) and (e). Seven studies where adult smokers evaluated the effectiveness of various anti-smoking campaign messages were examined. The first analysis shows that the position in which a message appears has little or no impact on its evaluation, supporting the validity of multiple-exposure design. The second analysis suggests having 25 evaluations per message can achieve a fair balance between accuracy and efficiency.