Can Generative LLMs Help Classify Imbalanced Real-World Data? Exploring Rare Diseases on Social Media

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Social Media Analysis

Background:

• Developmental and Epileptic Encephalopathies (DEEs) are rare, severe neurological disorders.
• Social media platforms are valuable sources for understanding patient and family experiences with rare diseases.
• Data imbalance poses a significant challenge in analyzing rare disease discussions online.

Purpose of the Study:

• To evaluate the effectiveness of generative large language models (LLMs) for classifying DEE-related social media posts.
• To compare zero-shot prompt-based classification with synthetic data augmentation for addressing data imbalance.

Main Methods:

• Utilized CamemBERT as a baseline model for binary classification tasks.
• Implemented zero-shot prompt-based classification strategy.
• Employed generative LLMs to create synthetic data for augmenting the minority class (DEE-related posts).
• Compared performance metrics (macro/positive F1, precision, recall) across different strategies and synthetic data proportions.

Main Results:

• Zero-shot prompt-based classification showed underperformance.
• The addition of a small proportion (2%) of synthetic data significantly improved all evaluated metrics.
• Increasing the proportion of synthetic data beyond 2% led to a decrease in classification precision.

Conclusions:

• Generative LLMs show promise in identifying rare disease experiences on social media.
• Hybrid approaches combining LLM fine-tuning with domain-specific synthetic data are effective for mitigating data imbalance in rare disease research.
• Further research and validation across diverse models and datasets are warranted to confirm these findings.