A Cautionary Note on Predicting Social Judgments from Faces with Deep Neural Networks

Area of Science:

• Computer Vision
• Social Psychology
• Machine Learning

Background:

• Humans infer psychological states from faces, influencing social decisions despite potential inaccuracies.
• Automated prediction of these judgments using deep convolutional neural networks (DCNNs) is a growing area of research.

Purpose of the Study:

• To evaluate the generalizability and potential biases of DCNNs for predicting social and affective judgments from faces.
• To compare feature sets from face identification DCNNs, object recognition DCNNs, and facial geometry.

Main Methods:

• Compared prediction accuracy across multiple out-of-sample datasets using features from different DCNNs and facial geometry.
• Investigated the specificity of predictions by training models for one judgment and testing on others.
• Assessed robustness to superficial cues like hair style and color.

Main Results:

• DCNN features outperformed facial geometry in predicting social judgments.
• Object recognition DCNN features were sensitive to superficial cues.
• Face identification DCNNs exhibited non-specific predictions, correlating unrelated judgments (e.g., trustworthiness with femininity/criminality) and emotions with traits.

Conclusions:

• While DCNNs offer a framework for facial judgment prediction, they carry risks of unintended correlations and biases.
• Automated predictions may be driven by unintended, correlated judgments rather than the target trait.