Exploiting the systematic review protocol for classification of medical abstracts

Area of Science:

• Medical Informatics
• Computational Linguistics
• Evidence-Based Medicine

Background:

• Systematic reviews are crucial for synthesizing medical evidence.
• Manual abstract screening is time-consuming and labor-intensive.
• Automated document classification offers potential efficiency gains.

Purpose of the Study:

• To evaluate the utility of automatic document classification in medical systematic reviews.
• To assess if a 'per-question' classification approach enhances performance over a 'global' method.
• To determine the effectiveness of a human-machine workflow for abstract screening.

Main Methods:

• Utilized a dataset of 47,274 abstracts from a systematic review on elderly healthcare dissemination.
• Employed the complement naïve Bayes machine learning algorithm.
• Compared a global classification method with a novel per-question method training multiple classifiers.

Main Results:

• The per-question method achieved significantly higher recall (99.2%) and precision (63%) than the global method (recall 67.8%, precision 37.9%).
• A human-machine workflow reached a recall of 99.6% with 17.8% precision.
• The per-question approach demonstrated superior performance in identifying relevant documents.

Conclusions:

• The per-question classification method enhances recall in systematic reviews compared to global methods.
• Neither automated method alone is sufficient; a semi-automatic approach with human oversight is recommended.
• Integrating automatic classification into a human-machine workflow effectively reduces workload and improves recall for systematic reviews.