Coding infant engagement in the Face-to-Face Still-Face paradigm using deep neural networks

Area of Science:

• Developmental Psychology
• Computational Neuroscience
• Human-Computer Interaction

Background:

• The Face-to-Face Still-Face (FFSF) task is a standard observational method for assessing mother-infant socio-emotional interactions.
• Advancements in deep learning-based facial emotion recognition offer potential for automating complex behavioral coding tasks.

Purpose of the Study:

• To evaluate the accuracy of deep neural network (DNN) image classification models in coding infant engagement.
• To compare DNN performance against manual coding by trained independent raters.

Main Methods:

• Sixty-eight mother-infant dyads participated, completing the FFSF task across three time points.
• Infant engagement was manually coded second-by-second into four categories: positive affect, neutral affect, object/environment engagement, and negative affect.
• Four distinct DNN models were trained on 40,000 images from FFSF recordings.

Main Results:

• DNN models achieved a maximum classification accuracy of 99.5% for infant frames from FFSF task recordings.
• The highest inter-rater reliability for DNN models reached a Cohen's kappa value of 0.993.
• These results indicate high concordance between DNN classification and manual coding of infant engagement.

Conclusions:

• Deep neural networks demonstrate high accuracy in coding infant engagement from FFSF task videos.
• DNNs show potential for significantly improving the efficiency of coding observational data across various fields of human behavior research.
• Automated coding using DNNs could streamline research processes and expand the scope of behavioral analysis.