An Expectation Maximization approach to Joint Modeling of Multidimensional Ratings derived from Multiple Annotators

Area of Science:

• Machine Learning
• Human-Computer Interaction
• Developmental Psychology

Background:

• Human annotator ratings are crucial for ground truth estimation in various applications.
• These ratings are often multidimensional and subjective.
• Existing methods may not fully capture the complexity of multidimensional ratings.

Purpose of the Study:

• To propose an Expectation-Maximization (EM) based algorithm for modeling multidimensional human ratings.
• To reliably predict individual annotator ratings from observation features.
• To address challenges of annotator cognitive load and rating costs.

Main Methods:

• Developed an EM algorithm assuming a latent multidimensional ground truth.
• Trained a baseline model for direct prediction of annotator ratings.
• Compared the proposed model against the baseline under three settings: independent dimensions, joint distribution, and partial rating prediction.

Main Results:

• The proposed EM-based model demonstrated improved prediction of multidimensional ratings compared to a direct baseline.
• The model's performance varied across the three tested settings, highlighting the importance of considering rating dimension dependencies.
• Accurate prediction of individual annotator ratings was achieved, validating the model's effectiveness.

Conclusions:

• The EM-based algorithm provides a robust framework for modeling and predicting multidimensional human ratings.
• This approach has significant implications for applications requiring subjective assessments, such as evaluating child expressivity in autism studies.
• The model offers a pathway to optimize annotation processes, reducing costs and cognitive burden.