Extraction of Social Determinants of Health From Electronic Health Records Using Natural Language Processing

Area of Science:

• Health Informatics
• Artificial Intelligence in Healthcare
• Population Health Management

Background:

• Social Determinants of Health (SDoH) significantly impact health outcomes and inequalities.
• Extracting SDoH from Electronic Health Records (EHR) is crucial for policy development and research.
• Automated extraction methods, like AI and NLP, offer improved efficiency and cost-effectiveness.

Purpose of the Study:

• To autonomously extract comprehensive SDoH details from EHR using a natural language processing (NLP)-based AI pipeline.
• To evaluate the performance of an open-source NLP pipeline against an industrial benchmark for SDoH extraction.
• To identify key SDoH factors prevalent in oncology patient populations.

Main Methods:

• A curated dataset of 1,000 BC Cancer clinical documents was used for training and validation.
• An open-source NLP pipeline was developed and optimized, considering subtype word positions.
• The optimized pipeline was applied to extract SDoH from 13,258 oncology documents, compared against an industrial benchmark.

Main Results:

• The open-source NLP pipeline achieved an average F1 score of 0.88, outperforming the benchmark by 5%.
• The pipeline successfully extracted 13 SDoH factors and over 60,000 SDoH instances.
• Frequently extracted SDoH factors included tobacco use, employment, marital status, alcohol consumption, and living status.

Conclusions:

• An NLP pipeline demonstrates significant potential for extracting SDoH factors from clinical notes.
• The developed pipeline shows strong performance, particularly with limited data.
• Data set-specific adjustments are necessary for broader institutional application of the NLP pipeline.