Utilizing large language models to evaluate medical research abstracts

The current evaluation of abstracts for national conferences faces challenges in consistency due to subjectivity and implicit biases. This study explores the feasibility of using ChatGPT-3.5-turbo models to grade and rank medical abstracts.

Two ChatGPT-3.5-turbo models were trained to evaluate abstracts submitted to the American Society of Plastic Surgery and American Hernia Society annual meetings and then asked to grade and rank each meeting's 2023 abstracts. Grades and ranks were compared between human reviewers and ChatGPT.

American Hernia Society: ChatGPT was fine-tuned on 153 abstracts from 2021 and scored 167 from 2023 on a 5-point scale, with human scoring 2.820 and ChatGPT scoring 2.749. Average difference between scores was 0.072, average distance was 0.701, and Spearman coefficient was 0.143. American Society of Plastic Surgery: ChatGPT was fine-tuned on the top 200 abstracts from 2021 to 2022 and scored 200 from 2023 on a 20-point scale, with human scoring 17.131 and ChatGPT scoring them 17.032. Average difference was 0.104, average distance was 0.718, and Spearman coefficient was 0.137. For quartiles based on ChatGPT-generated abstract scores, average correct ranking was 1.146 ± 1.042 and relative correct ranking was 2.579 ± 0.707 for American Hernia Society and 1.083 ± 1.050 and 2.493 ± 0.755, respectively, for American Society of Plastic Surgery. A bridging algorithm showed comparable performance, with Spearman coefficient of 0.032 for American Hernia Society and 0.023 for American Society of Plastic Surgery.

Despite ChatGPT models being trained to grade and rank abstracts, there was little correlation with human scores. This work serves as a benchmark example of ChatGPT's potential integration into the abstract review process.