Assessing the quality and performance of synthetic data augmentation to identify stigmatizing language in obstetric...

Stigmatizing language in clinical notes is associated with adverse birth outcomes. Identifying stigmatizing language using natural language processing (NLP) requires human-annotated datasets, which can be scarce.

Generate synthetic data containing stigmatizing language using Llama-3, assess quality, and examine NLP performance.

Synthetic data was generated for "Difficult Patient" and "Questioning Patient Credibility" using zero-shot (no examples), few-shot (examples), and fine-tuned few-shot (fine-tuned model and examples). Nurse researchers evaluated data quality. NLP models (e.g., ClinicalBERT) were trained using real-world (birth admission notes) and synthetic augmented datasets.

Fine-tuned few-shot generated higher quality synthetic data. ClinicalBERT F1-scores increased from 0.88 to 0.95 with 10% synthetic data augmentation for Difficult Patient and from 0.68 to 0.82 with 50% augmentation for Questioning Patient Credibility.

Synthetic data improved performance with varied gains, showing potential to address data scarcity. Findings support developing tools to identify stigmatizing language, providing measurable indicators to inform policy.