Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study

Area of Science:

• Medical research methodology
• Artificial intelligence in healthcare
• Evidence synthesis

Background:

• Assessing bias in randomized controlled trials (RCTs) is crucial for systematic reviews but is time-consuming.
• Machine learning (ML) systems like RobotReviewer offer potential for semi-automating this process.

Purpose of the Study:

• To evaluate the time savings and usability of RobotReviewer for bias assessment in RCTs.
• To compare semi-automated bias assessment with manual methods.

Main Methods:

• A user study involving 41 volunteers assessing four RCT articles using the Cochrane Risk of Bias tool.
• Comparison of manual bias assessment with a semi-automated process where ML suggestions were provided.
• Measurement of time taken, ML suggestion acceptance rates, and usability using the System Usability Scale (SUS).

Main Results:

• Semi-automated assessment was quicker (755s) than manual assessment (824s), a 25% time saving.
• Reviewers accepted 91% of ML bias judgments and 62% of text highlights.
• The system achieved a high SUS score of 77.7, indicating good to excellent usability.

Conclusions:

• Semi-automation, with human validation of ML suggestions, enhances efficiency in evidence synthesis.
• RobotReviewer effectively expedites bias assessment for RCTs and is highly usable.