Using Large Language Models to Abstract Complex Social Determinants of Health From Original and Deidentified Medical...

Area of Science:

• Health Informatics
• Natural Language Processing
• Social Determinants of Health

Background:

• Social determinants of health (SDoH), such as housing insecurity, significantly impact patient health outcomes.
• Efficiently abstracting structured data on SDoH is crucial for biomedical research and proactive healthcare interventions.
• Large language models (LLMs) demonstrate potential for complex data abstraction from unstructured clinical text.

Purpose of the Study:

• To evaluate the performance of GPT-3.5 and GPT-4 in identifying temporal aspects of housing insecurity from clinical notes.
• To compare LLM performance against manual abstraction, named entity recognition, and regular expressions.
• To assess the impact of deidentification on LLM performance in housing insecurity detection.

Main Methods:

• Utilized 25,217 clinical notes from 795 pregnant women for analysis.
• Compared GPT-3.5 and GPT-4's ability to identify current, past, and general housing status.
• Benchmarked LLM results against human abstractors, a named entity recognition model, and regular expressions.

Main Results:

• GPT-4 exhibited superior performance compared to GPT-3.5 and the named entity recognition model.
• GPT-4 achieved higher recall (0.924) than human abstractors (0.702) for current/past housing instability.
• GPT-4's precision was lower than human abstractors but improved on deidentified notes, while recall decreased.

Conclusions:

• LLMs offer a scalable and cost-effective method for SDoH data abstraction, particularly with high recall.
• Manual abstraction may yield slightly higher precision, but LLMs can support semi-automated processes.
• Human review remains essential for patient care decisions, and deidentification can negatively impact LLM recall.