Active Annotation in Evaluating the Credibility of Web-Based Medical Information: Guidelines for Creating Training...

Area of Science:

• Medical Informatics
• Computer Science
• Machine Learning

Background:

• The proliferation of online medical misinformation necessitates robust methods for credibility assessment.
• Machine learning presents a viable technological solution for identifying and mitigating web-based health falsehoods.

Purpose of the Study:

• To present a framework for creating and curating machine learning training datasets for evaluating the credibility of online medical information.
• To provide guidelines for preparing datasets to train machine learning models aimed at combating medical misinformation.
• To support researchers in both medical and computer science fields.

Main Methods:

• Developed an annotation protocol for medical experts to evaluate the credibility of medical statements.
• Implemented a preprocessing pipeline including representation learning, clustering, and reranking, termed 'active annotation', to address limited initial labels.
• Utilized qualitative analysis of experimental data to refine the annotation protocol.

Main Results:

• Generated over 10,000 annotations for medical statements across diverse topics (e.g., psychiatry, vaccines, autism) at a low cost (under $7000) using certified medical professionals.
• Released the curated dataset publicly to aid research efforts.
• Demonstrated the efficiency of the active annotation framework in identifying non-credible medical statements.

Conclusions:

• The qualitative analysis confirmed the effectiveness of the proposed active annotation method for dataset creation.
• The developed framework and released dataset can advance the fight against online medical misinformation.