Interrater Reliability for Interdependent Social Network Data: A Generalizability Theory Approach.

Area of Science:

• Social Psychology
• Network Analysis
• Behavioral Observation

Background:

• Observational data in social networks often involve interdependent dyadic interactions.
• Assessing the reliability of observations across different raters is crucial for valid conclusions.
• Existing models may not fully capture the complexities of rater influence on dyadic observations.

Purpose of the Study:

• To propose novel interrater reliability coefficients for observational interdependent social network data.
• To extend the social relations model by incorporating rater effects for a comprehensive variance decomposition.
• To provide a statistically robust method for assessing the generalizability of observed social behaviors across raters.

Main Methods:

• Extended the social relations model using generalizability theory to include rater effects.
• Decomposed dyadic observational data variance into actor, partner, relationship, and rater effects, including interactions.
• Employed Markov chain Monte Carlo estimation of a Bayesian hierarchical linear model to calculate intraclass correlation coefficients (ICCs).

Main Results:

• The proposed model successfully decomposes dyadic observational data variance into key components.
• Developed ICCs quantify the generalizability of actor, partner, and relationship effects across different raters.
• Simulation studies confirmed the proposed method's low bias and adequate coverage for ICC estimation.

Conclusions:

• The novel interrater reliability coefficients offer a reliable way to assess observational data in social networks.
• The extended social relations model provides a more nuanced understanding of variance in dyadic interactions.
• This methodology enhances the rigor and generalizability of findings in social network research, as demonstrated with social mimicry data.