Automatic evidence quality prediction to support evidence-based decision making.

Area of Science:

• Medical Informatics
• Computational Linguistics
• Evidence-Based Medicine

Background:

• Manual appraisal of medical evidence quality is time-consuming for practitioners.
• The increasing volume of medical literature necessitates efficient methods for evidence quality assessment.
• Automated tools can support clinicians at the point-of-care by predicting evidence quality.

Purpose of the Study:

• To develop and evaluate a fully automatic approach for predicting the quality of medical evidence.
• To assist healthcare practitioners in making clinical decisions by providing rapid quality assessments of medical literature.
• To compare the performance of the automated system against human judgments.

Main Methods:

• Extracted relevant information from medical article abstracts and metadata.
• Utilized a specialized corpus and supervised machine learning for quality grade prediction.
• Employed rule-based approaches for feature extraction (e.g., publication types) and a sequence of high-precision classifiers.
• Introduced the Average Error Distance (AED) metric for system evaluation.

Main Results:

• Achieved high F-scores (up to 0.99) for automatic extraction of high-quality publication types.
• Obtained 63.84% accuracy and an AED of 0.271 for evidence quality classification.
• Human evaluations demonstrated that the system's performance (AED and accuracy) is comparable to human performance.

Conclusions:

• The structured text classification framework demonstrates performance comparable to human judgment.
• The automated approach aids in efficient evidence quality prediction for point-of-care decision-making.
• The classification approach and evaluation technique are portable across different evidence grading scales.