Evaluating large language models for abstract evaluation tasks: an empirical study

Large language models (LLMs) show great promise as tools for assisting scientific peer review, but their agreement with human experts in quantitative assessment of academic content needs further investigation. This study examined ChatGPT-5, Gemini-3-Pro, and Claude-Sonnet-4.5's consistency and reliability in evaluating conference abstracts compared to one another and to human reviewers.

Three LLMs independently graded 160 abstracts from a regional conference, while 14 human reviewers each assessed a subset using an identical rubric with eight criteria scored on a 1-5 scale. We compared AI and human scoring patterns using boxplots, calculated intraclass correlation coefficients (ICCs) for inter-rater reliability both among LLMs and between human and LLMs, and examined Bland-Altman plots to identify agreement patterns and systematic bias.

Three LLMs demonstrated high internal consistency with narrow interquartile ranges and few outliers in composite scores, while human reviewers exhibited greater scoring variability. LLMs also achieved good-to-excellent agreement with each other across all criteria (ICCs: 0.59-0.87). ChatGPT and Claude reached moderate agreement with human reviewers on overall quality and content-specific criteria, with ICCs = 0.45-0.60 for composite score, impression, clarity, objective, and results. The two LLMs' concordance with humans achieved fair levels on subjective dimensions, with ICC ranging from 0.23-0.38 for impact, engagement, and applicability. Gemini performed notably worse, showing fair agreement on half the criteria and poor reliability on impact and applicability. Bland-Altman analysis revealed acceptable or negligible systematic bias, with mean differences of 0.24 (ChatGPT), 0.42 (Gemini), and -0.02 (Claude) from human mean ratings.

With appropriate model selection, LLMs could reach moderate agreement with human experts on abstract overall quality and objective criteria, supporting their potential use for pre-screening low-quality submissions or serving as additional reviewers. Their ability to apply rubrics consistently across large volumes of abstracts offers advantages in efficiency and standardization that exceed human feasibility. However, LLMs' reduced performance on subjective dimensions indicates that they should complement rather than replace human judgment in abstract evaluation, with expert review remaining essential for comprehensive assessment.