Discriminating between empirical studies and nonempirical works using automated text classification.

Area of Science:

• Information Science
• Biomedical Informatics
• Research Methodology

Background:

• Systematic mixed studies reviews require accurate differentiation between empirical and nonempirical research.
• Manual screening of large bibliographic databases is time-consuming and resource-intensive.
• Automated text classification offers a potential solution to accelerate study identification.

Purpose of the Study:

• To identify the most performant automated text classification algorithm for distinguishing empirical from nonempirical studies.
• To enhance the efficiency of systematic mixed studies reviews through improved study selection.

Main Methods:

• Trained and validated multiple algorithms on 8050 manually categorized database records.
• Utilized a Boolean mixed filter as a baseline comparison.
• Extracted features from metat a saurus, including observable terms and concepts, and measured performance using accuracy, sensitivity, precision, and specificity.

Main Results:

• Decision trees algorithm achieved the highest performance, exceeding the baseline Boolean filter's accuracy by 30%.
• Incorporating both concepts and observable terms improved classification accuracy.
• Using full texts did not yield significant performance gains over title, abstract, and keywords.

Conclusions:

• Automated text classification, particularly using decision trees, can substantially accelerate the screening of research records for systematic reviews.
• This approach facilitates more efficient and accurate inclusion of relevant studies in systematic reviews.