Analysing multisource feedback with multilevel structural equation models: Pitfalls and recommendations from a
Jana Mahlke1, Martin Schultze2, Michael Eid1
1Division of Methods and Evaluation, Department of Educational Science and Psychology, Freie Universität Berlin, Germany.
The British Journal of Mathematical and Statistical Psychology
|January 30, 2019
Summary
This study determined minimum sample sizes for validating multisource feedback using multilevel structural equation models. Accurate standard error estimation requires 400 self-ratings or four ratings from peers/subordinates.
Area of Science:
- Organizational Psychology
- Psychometrics
- Quantitative Research Methods
Background:
- Multisource feedback instruments, such as 360-degree feedback, are crucial for assessing employee performance and development.
- Validating these instruments often involves complex statistical models to ensure reliability and validity.
- Multilevel structural equation models (MSEM) are the preferred method for analyzing such data structures.
Purpose of the Study:
- To determine the minimal required sample sizes for a specific non-standard MSEM incorporating self-ratings and ratings from multiple sources (peers, subordinates).
- To investigate the impact of sample size on the accuracy of parameter and standard error estimation in this model.
- To evaluate the influence of convergent and discriminant validity on model estimation accuracy.
Main Methods:
- A Monte Carlo simulation study was employed to systematically vary sample sizes.
- A non-standard MSEM was utilized, distinguishing between level-2 (self-ratings) and level-1 (others' ratings) variables.
- Model fit was assessed using a corrected level-specific standardized root mean square residual (CLSRMR).
Main Results:
- Model parameters were accurately estimated even with small sample sizes (100 self-ratings, 2 ratings from peers/subordinates).
- Precise standard error estimation required larger samples: 400 self-ratings or at least four ratings from peers/subordinates.
- Smaller sample sizes primarily biased standard errors related to common method factors, with trade-offs observed between self- and other-ratings.
Conclusions:
- The study provides crucial sample size recommendations for researchers using this specific MSEM for multisource feedback validation.
- Researchers should be cautious about standard error bias with smaller sample sizes, particularly concerning common method factors.
- The corrected CLSRMR is recommended for model fit analysis, and the χ² test statistic should be interpreted with caution.
Related Concept Videos
00:46Feedback Inhibition
57.1K
Biochemical reactions are occurring constantly in cells, converting starting substances to different products, usually with the help of enzymes that speed the reactions. Without enzymes, it would take far too long for most reactions to occur to be useful to the cell!
57.1K
01:25Modeling with Differential Equations
66
Population dynamics can be described mathematically by considering the population size P(t) as a function of time. The rate of change of the population is then represented by the derivative of P(t). A simple assumption is that the rate of growth is proportional to the size of the population itself. This leads to an exponential growth model, where the population increases rapidly without bound. While this is a useful first approximation, it does not reflect realistic long-term...
66
03:10Chemical Equations
81.3K
Chemical equations represent the identities and relative quantities of substances involved in a chemical reaction. The substances undergoing reaction are called reactants, and their formulas are placed on the left side of the equation. The substances generated by the reaction are called products, and their formulas are placed on the right side of the equation. Plus signs (+) separate individual reactant and product formulas, and an arrow (→) separates the reactant and product (left and right)...
81.3K
01:01Feedback Loops
64.3K
In most cases, excessive hormone production is prevented by negative feedback—a loop that starts with a stimulus inducing the release of a particular substance, like a hormone, to maintain a certain level before triggering a signal that results in a decrease in further release of the hormone.
64.3K
02:59The Nernst Equation
46.8K
Nonstandard Reaction Conditions
The interconnection between standard cell potentials and various thermodynamic parameters such as the standard free energy change ΔG° and equilibrium constant K has been previously explored. For example, a redox reaction involving zinc(II) and tin(II) ions at 1 M concentration with Eºcell = +0.291 V and ΔG° = −56.2 kJ is spontaneous.
The interconnection between standard cell potentials and various thermodynamic parameters such as the standard free energy change ΔG° and equilibrium constant K has been previously explored. For example, a redox reaction involving zinc(II) and tin(II) ions at 1 M concentration with Eºcell = +0.291 V and ΔG° = −56.2 kJ is spontaneous.
46.8K
02:55Thermochemical Equations
35.9K
For a chemical reaction (the system) carried out at constant pressure – with the only work done caused by expansion or contraction – the enthalpy of reaction (also called the heat of reaction, ΔHrxn) is equal to the heat exchanged with the surroundings (qp).
35.9K