On reliability of annotations in contextual emotion imagery

Area of Science:

• Psychology
• Computer Science
• Data Science

Background:

• Reliability in emotion-in-context recognition databases is crucial for accurate AI development.
• Existing databases may require refinement to ensure high-quality, trustworthy annotations.

Purpose of the Study:

• To document and analyze a relabeling process for an emotion-in-context database.
• To enhance the reliability and trustworthiness of emotion labels through experimental design.

Main Methods:

• Organized emotion categories into eight groups.
• Employed a large number of participants for data annotation (average 28 per day).
• Implemented a Z-Score filtering technique and strict experimental controls over 13.45 minutes daily.

Main Results:

• Fleiss' Kappa agreement values ranged from slight to almost perfect, indicating improved coherence.
• The relabeling process demonstrated a "coherent diversity" in experimental outcomes.
• Annotation reliability was positively impacted by the experimental setup.

Conclusions:

• A reduced number of emotion categories enhances annotation reliability.
• A large number of participants (voters) significantly benefits the trustworthiness of labels in contextual emotion imagery.
• The findings support a scalable and reliable method for database annotation in affective computing.