Pose-invariant and occlusion-robust neonatal facial pain assessment

Neonatal Facial Pain Assessment (NFPA) is essential to improve neonatal pain management. Pose variation and occlusion, which can significantly alter the facial appearance, are two major and still unstudied barriers to NFPA. We bridge this gap in terms of method and dataset. Techniques to tackle both challenges in other tasks either expect pose/occlusion-invariant deep learning methods or first generate a normal version of the input image before feature extraction, combining these we argue that it is more effective to jointly perform adversarial learning and end-to-end classification for their mutual benefit. To this end, we propose a Pose-invariant Occlusion-robust Pain Assessment (POPA) framework, with two novelties. We incorporate adversarial learning-based disturbance mitigation for end-to-end pain-level classification and propose a novel composite loss function for facial representation learning; compared to the vanilla discriminator that implicitly determines occlusion and pose conditions, we propose a multi-scale discriminator that determines explicitly, while incorporating local discriminators to enhance the discrimination of key regions. For a comprehensive evaluation, we built the first neonatal pain dataset with disturbance annotation involving 1091 neonates and also applied the proposed POPA to the facial expression recognition task. Extensive qualitative and quantitative experiments prove the superiority of the POPA.