Assessing the Capability of Large Language Model Chatbots in Generating Plain Language Summaries

Area of Science:

• Scientific communication
• Artificial Intelligence in Research

Background:

• Plain language summaries (PLSs) are crucial for making scientific research accessible to a non-expert audience.
• Crafting effective PLSs presents challenges, particularly for non-native English-speaking researchers.
• The potential of large language model (LLM) chatbots to aid in PLS generation is underexplored.

Purpose of the Study:

• To compare the quality and readability of human-written PLSs with those generated by various LLM chatbots.
• To assess the effectiveness of LLM chatbots in producing accessible scientific summaries.

Main Methods:

• A cross-sectional study comparing 30 human-written PLSs against PLSs generated by six LLM chatbots (ChatGPT, Claude, Copilot, Gemini, Meta AI, Perplexity).
• Readability assessed using Flesch reading ease (FR) scores and Flesch-Kincaid (FK) grade levels.
• Text quality evaluated by three authors based on predefined criteria.

Main Results:

• LLM-generated PLSs demonstrated lower FK grade levels and higher FR ease scores compared to human-written PLSs (p < 0.0001).
• No statistically significant difference in overall quality scores was found between human-generated PLSs and those from individual LLM chatbots.
• LLM chatbots produced PLSs of comparable quality to those written by humans.

Conclusions:

• LLM chatbots can generate PLSs with enhanced readability, making research accessible to individuals with lower educational backgrounds.
• The quality of LLM-generated PLSs is on par with human-written summaries.
• Authors can leverage LLM chatbots for PLS generation, with particular advantages for researchers in developing countries, though accuracy checks are essential.