Membership Inference or Data Split Bias? Identifying False Positives in Synthetic Medical Image Privacy Audits

Membership Inference Attacks (MIAs) are the current standard for auditing the privacy of synthetic medical data. A high attack score is widely interpreted as a significant privacy leak, but this interpretation assumes the MIA is learning a true membership signal. We argue that these attacks may instead be learning experimental artefacts, specifically data split bias. To test this, we conduct experiments on synthetic brain MRI images from a StyleGAN3 model. We design a Primary MIA (member vs. non-member data) and a critical Control Experiment (disjoint subsets of member data alone). This member-only setup lacks any genuine membership distinction between classes, leaving only data split bias. Our results show the Control Experiment's AUCs match or exceed the Primary MIA's (+0.0205 on the 5k-pool Small corpus and +0.0113 on the 30k-pool Big corpus). This compelling evidence demonstrates that a high MIA score can be a false positive for a privacy breach, as the classifier is simply learning statistical artefacts. We conclude that MIA results are unreliable without rigorous control experiments to validate that they are detecting genuine privacy risks, not experimental bias.