Classification of Cochrane Plain Language Summaries by Conclusiveness Using Transformer-Based Models and ChatGPT:...

Area of Science:

• Natural Language Processing
• Machine Learning in Healthcare
• Biomedical Informatics

Background:

• Cochrane plain language summaries (PLSs) enhance public accessibility to systematic review findings.
• Inconsistent presentation of conclusions in PLSs can hinder comprehension and decision-making.
• Classifying PLSs by conclusiveness can improve clarity and support informed health choices.

Purpose of the Study:

• Develop and evaluate deep learning language models for classifying PLSs into three conclusiveness levels: conclusive, inconclusive, and unclear.
• Compare the performance of these specialized models against a general-purpose large language model (GPT-4o).

Main Methods:

• Utilized a dataset of 4405 Cochrane PLSs, classified into three conclusiveness levels.
• Fine-tuned Scientific Bidirectional Encoder Representations from Transformers (SciBERT) and Longformer models.
• Evaluated model performance using balanced accuracy and Area Under the Receiver Operating Characteristic Curve (AUCROC), comparing against manual verification and ChatGPT outputs.

Main Results:

• SciBERT achieved 56.6% balanced accuracy; Longformer achieved 60.9% balanced accuracy.
• AUCROC values for SciBERT ranged from 0.67 to 0.91; for Longformer, from 0.67 to 0.86.
• Both specialized models underperformed compared to ChatGPT (74.2% accuracy, higher precision, recall, and Cohen κ).

Conclusions:

• Transformer-based models showed limited success in classifying PLS conclusiveness due to semantic nuances.
• Fine-tuned models struggled to generalize to new PLSs, with performance dropping significantly.
• General-purpose large language models like GPT-4o currently provide more dependable results for classifying biomedical texts.