Automated detection of mouth opening in newborn infants

Automated behavioral measurement using machine learning is gaining ground in psychological research. Automated approaches have the potential to reduce the labor and time associated with manual behavioral coding, and to enhance measurement objectivity; yet their application in young infants remains limited. We asked whether automated measurement can accurately identify newborn mouth opening-a facial gesture involved in infants' communication and expression-in videos of 29 newborns (age range, 9-29 days, 55.2% female, 58.6% White, 51.7% Hispanic/Latino) during neonatal imitation testing. We employed a three-dimensional cascade regression computer vision algorithm to automatically track and register newborn faces. The facial landmark coordinates of each frame were input into a support vector machine (SVM) classifier, trained to recognize the presence and absence of mouth openings at the frame level as identified by expert human coders. The SVM classifier was trained using leave-one-infant-out cross-validation (training: N = 22 newborns, 95 videos, 354,468 frames), and the best classifier showed an average validation accuracy of 75%. The final SVM classifier was tested on different newborns from the training set (testing: N = 7 newborns, 29 videos, 118,615 frames) and demonstrated 76% overall accuracy in identifying mouth opening. An intraclass correlation coefficient of .81 among the SVM classifier and human experts indicated that the SVM classifier was, on a practical level, reliable with experts in quantifying newborns' total rates of mouth opening across videos. Results highlight the potential of automated measurement approaches for objectively identifying the presence and absence of mouth opening in newborn infants.