RoBuster-Corpus Annotated With Risk of Bias Text Spans in Randomized Controlled Trials in Physiotherapy and...

Area of Science:

• Medical Informatics
• Clinical Trial Methodology
• Natural Language Processing

Background:

• Manual risk of bias (RoB) assessment in randomized clinical trials (RCTs) is time-consuming and cognitively demanding for systematic reviews.
• Automating RoB assessment can expedite the identification of bias indicators within RCT texts.
• A lack of annotated text span corpora and clear guidelines hinders the development and evaluation of large language models (LLMs) for RoB assessment.

Purpose of the Study:

• To develop precise RoB text span annotation instructions addressing subjectivity in RoB assessment.
• To create an annotated corpus (RoBuster) of RCTs for fine-tuning and evaluating LLMs.
• To enhance interannotator agreement (IAA) in RoB span and risk judgment annotation.

Main Methods:

• Leveraged the revised Cochrane RoB 2 tool to create visual instructional placards for RoB annotation.
• Expert annotators used these placards to annotate 41 full-text RCTs in physiotherapy and rehabilitation, forming the RoBuster corpus.
• Evaluated IAA for text span and risk judgment annotations, and assessed LLM (GPT-3.5) performance on RoB span extraction.

Main Results:

• The RoBuster corpus contains 41 RCTs with over 28,427 tokens and 22 RoB classes.
• Visual instructions significantly increased IAA for text span annotations by over 17 percentage points.
• LLM (GPT-3.5) demonstrated varied agreement with expert annotations, particularly struggling with subjective RoB questions.

Conclusions:

• RoB annotation remains challenging, especially for subjective RoB questions and when annotation data is scarce.
• Visual instructional placards improve IAA in RoB assessment and annotation.
• LLMs show potential for RoB span extraction but require further refinement for complex and subjective assessments.