FairREAD: Re-fusing demographic attributes after disentanglement for fair medical image classification

Recent advancements in deep learning have shown transformative potential in medical imaging, yet concerns about fairness persist due to performance disparities across demographic subgroups. Existing methods aim to address these biases by mitigating sensitive attributes in image data; however, these attributes often carry clinically relevant information, and their removal can compromise model performance-a highly undesirable outcome. To address this challenge, we propose Fair Re-fusion After Disentanglement (FairREAD), a novel, simple, and efficient framework that mitigates unfairness by re-integrating sensitive demographic attributes into fair image representations. FairREAD employs orthogonality constraints and adversarial training to disentangle demographic information while using a controlled re-fusion mechanism to preserve clinically relevant details. Additionally, subgroup-specific threshold adjustments ensure equitable performance across demographic groups. Comprehensive evaluations and out-of-distribution testing on large-scale clinical X-ray datasets demonstrate that, given demographic attributes of each patient, FairREAD is able to significantly reduce unfairness metrics while maintaining diagnostic accuracy. Our code is available at: https://github.com/Advanced-AI-in-Medicine-and-Physics-Lab/FairREAD/.