Spontaneous facial expression in unscripted social interactions can be measured automatically

Area of Science:

• Psychology
• Computer Science
• Behavioral Science

Background:

• Manual coding of facial actions is time-consuming and requires expertise.
• Automated coding of spontaneous facial expressions is crucial for advancing psychological theories and applications.
• Existing automated systems face challenges with variations in gender, ethnicity, pose, speech, and occlusion.

Purpose of the Study:

• To report a significant advancement in the automated coding of spontaneous facial actions.
• To evaluate the precision and robustness of automated facial action unit (AU) detection in unscripted social interactions.
• To assess the feasibility of applying automated FACS coding to observational research.

Main Methods:

• Manually coded 25 facial action units (AUs) for 80 participants in an unscripted social interaction using the Facial Action Coding System (FACS).
• Applied automated FACS coding to 12 frequently occurring AUs.
• Analyzed reliability using intraclass correlation for AU occurrence proportion and Matthew's correlation for frame-by-frame accuracy.

Main Results:

• Automated coding demonstrated very strong reliability for the proportion of time AUs occurred (mean ICC = 0.89).
• Frame-by-frame reliability was moderate to strong (mean Matthew's correlation = 0.61).
• AU detection showed minimal differences across gender, ethnicity, pose, and pixel intensity, with <6% of frames missed by automation.

Conclusions:

• Automated FACS coding has reached a sufficient level of precision and robustness for observational research.
• This technology can significantly ease the burden of manual coding in complex behavioral studies.
• The findings support the application of automated facial expression analysis in emotion and social interaction research.