Constructing a disease database and using natural language processing to capture and standardize free text clinical...

Area of Science:

• Medical Informatics
• Public Health
• Natural Language Processing

Background:

• Timely extraction of infectious disease information is vital for population health research.
• A significant barrier is the absence of effective methods for mining large health datasets.
• Identifying clinical factors and social determinants of health from unstructured text remains challenging.

Purpose of the Study:

• To develop and evaluate a natural language processing (NLP) framework for extracting key clinical and social determinants of health information from free text.
• To demonstrate the framework's utility in constructing databases for pandemic surveillance using COVID-19 case reports.
• To assess the framework's performance against established methods.

Main Methods:

• Database construction and the development of specialized NLP modules for identifying clinical and social determinants of health.
• Utilizing COVID-19 case reports for practical data construction and surveillance demonstration.
• Implementing a detailed evaluation protocol to quantify the framework's effectiveness.

Main Results:

• The proposed NLP framework demonstrated superior performance, achieving an F1-score approximately 1-3% higher than benchmark methods.
• The system successfully identified disease presence and symptom frequency in patient data.
• Analysis revealed the effectiveness of transfer learning for infectious diseases with similar clinical presentations.

Conclusions:

• The developed NLP framework offers an effective solution for extracting critical health information from unstructured text.
• The approach significantly improves infectious disease surveillance and analysis capabilities.
• Transfer learning shows promise for enhancing predictive accuracy in similar infectious disease research.