Evaluating Large Language Models for Post-Publication Promotion: A Blinded Comparative Study of Social Media Posts in...

Area of Science:

• Scientific communication
• Artificial intelligence in research

Background:

• Social media platforms are vital for promoting scientific research, but content creation is time-consuming.
• Large language models (LLMs) present a potential solution for efficient and effective research promotion.

Purpose of the Study:

• To systematically evaluate the performance of four leading LLMs in generating social media posts for academic research.
• To compare LLM-generated posts based on factual accuracy, policy compliance, clarity, and overall quality.

Main Methods:

• A blinded, offline evaluation of GPT-5, Gemini 2.5 Pro, Grok-3, and Perplexity Pro was conducted.
• LLMs generated X-style posts for 36 open access articles, scored by a single blinded rater on a five-domain rubric.
• Secondary measures included character count, hashtag use, and readability.

Main Results:

• All tested LLMs produced factually accurate and policy-compliant posts.
• GPT-5 and Perplexity Pro achieved the highest overall quality scores, followed closely by Gemini 2.5 Pro; Grok-3 scored lower.
• Significant differences were observed in call-to-action quality and post length, with varying readability levels across models.

Conclusions:

• LLMs are capable of generating reliable, high-quality social media content for research promotion.
• Model selection should align with audience targeting and communication goals, with human oversight remaining crucial.
• LLMs can serve as scalable tools to increase the visibility and accessibility of scientific research.