Large Language Models for Supporting Clear Writing and Detecting Spin in Randomized Controlled Trials in Oncology:...

Area of Science:

• Oncology
• Medical Informatics
• Scientific Publishing

Background:

• Randomized controlled trials (RCTs) are crucial for oncology intervention evaluation.
• Reporting "spin" in RCTs can mislead readers about true efficacy, particularly in conclusions.

Purpose of the Study:

• To investigate the capability of large language models (LLMs) in detecting spin within oncology RCT reports.
• To assess LLMs' performance in identifying spin, especially in the conclusions section.

Main Methods:

• 250 oncology RCTs were randomly sampled from major medical journals.
• Three commercial LLMs classified trials as positive/negative using varying text inputs (conclusions only to full abstract).
• LLM performance was evaluated against human annotations using accuracy, precision, recall, and F1-score.

Main Results:

• The GPT-o1 model achieved high F1-scores across all input conditions, with the highest at 0.98 when provided with methods, results, and conclusions.
• Analysis of misclassified trials revealed patterns indicative of spin, such as absent primary endpoint results or emphasis on subgroup analyses.
• These spin patterns were rarely present in correctly classified negative trials.

Conclusions:

• LLMs can effectively detect potential spin in oncology RCT reporting by identifying discrepancies between conclusions and full abstracts.
• This LLM-based approach can serve as a supplementary tool to improve transparency in scientific reporting.
• Further development is needed to address more complex trial designs.