Inter-rater agreement in trait judgements from faces

Area of Science:

• Psychology
• Social Psychology
• Perception Science

Background:

• Social evaluations from facial first impressions are a key research area.
• High inter-rater agreement is typically reported for facial judgments, but current measures may be flawed.
• Cronbach's alpha is commonly used but can be misleadingly inflated by sample size.

Purpose of the Study:

• To evaluate alternative measures of inter-rater agreement for facial judgments.
• To investigate agreement levels for traits with predicted low (parental resemblance), intermediate (attractiveness, dominance, trustworthiness), and high (age, gender) agreement.
• To examine the influence of face familiarity on agreement levels.

Main Methods:

• Compared Cronbach's alpha with alternative statistical measures for inter-rater agreement.
• Assessed agreement across various facial traits, including attractiveness, trustworthiness, dominance, parental resemblance, age, and gender.
• Investigated the impact of face familiarity (familiar vs. unfamiliar) on agreement.

Main Results:

• Alternative measures provide more informative assessments of inter-rater agreement than Cronbach's alpha.
• No significant influence of face familiarity on the level of agreement was observed.
• Trustworthiness and dominance judgments, similar to attractiveness, exhibit significant private taste, alongside shared taste.

Conclusions:

• Researchers should consider alternatives to Cronbach's alpha for analyzing social judgments of faces.
• Investigating the origins and value of private taste is crucial for understanding social perceptions.
• Shared and private taste contribute significantly to the variance in facial social evaluations.