Related Experiment Video
Updated: Aug 5, 2026
08:27
Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits
Published on: September 27, 2019
Mechanisms of Effect Size Differences Between Researcher Developed and Independently Developed Outcomes: An
Joshua B Gilbert1, James Soland2
1Center for School Behavioral Health, Massachusetts General Hospital, Boston, Massachusetts; United States.
Multivariate Behavioral Research
|July 28, 2026
Summary
Researcher-developed (RD) and independently developed (ID) measures show different effect sizes in education research. Greater item-level variability in treatment effects explains some, but not all, of this observed difference.
Area of Science:
- Education Research
- Psychometrics
- Meta-Analysis
Background:
- Differences in effect sizes between researcher-developed (RD) and independently developed (ID) outcome measures are documented in education research.
- The underlying mechanisms explaining these differences remain poorly understood.
Purpose of the Study:
- To conduct a meta-analysis using item-level outcome data.
- To test potential mechanisms explaining effect size differences between RD and ID outcome measures.
Main Methods:
- Meta-analysis of 54 effect sizes from 32 studies.
- Analysis of item-level outcome data to examine predictors of effect sizes.
- Investigation of the role of standard deviations of item-specific treatment effects.
Main Results:
- Greater standard deviations of item-specific treatment effects predict larger effect sizes.
- This variability reduced the observed difference between RD and ID measures from 0.14 SDs to 0.12 SDs.
- The remaining difference is attributable to factors other than item-level variability.
Conclusions:
- Item properties, specifically variability in item-specific treatment effects, predict educational intervention outcomes.
- Understanding item-level data is crucial for developing theory in education research.
- Further research is needed to identify other factors contributing to the RD-ID effect size difference.
Related Concept Videos
01:13Multiple Comparison Tests
Multiple comparison test, abbreviated as MCT, is a post hoc analysis generally performed after comparing multiple samples with one or more tests. An MCT will help identify a significantly different sample among multiple samples or a factor among multiple factors.
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
02:01Group Design
The most basic experimental design involves two groups: the experimental group and the control group. The two groups are designed to be the same except for one difference— experimental manipulation. The experimental group gets the experimental manipulation—that is, the treatment or variable being tested—and the control group does not. Since experimental manipulation is the only difference between the experimental and control groups, we can be sure that any differences between the two are due to...
01:15Bioequivalence Experimental Study Designs: Repeated Measures, Cross-Over, Carry-Over, and Latin Square Designs
Bioequivalence experimental study designs play a pivotal role in testing the effectiveness of various treatments. Key among these are the repeated measures, cross-over, carry-over, and Latin square designs. In the repeated measures design, each subject receives all treatments, allowing for temporal comparisons. This type of design is useful in reducing variability but requires careful planning to avoid bias.The cross-over design, an economical method, involves sequential administration of...
01:28Methods of Medium Optimization
Optimizing growth media enhances microbial proliferation and maximizes product yield. Statistical experimental design methodologies provide structured and reproducible approaches, offering progressively higher levels of robustness and efficiency.The One-Factor-at-a-Time (OFAT) MethodThe One-Factor-at-a-Time (OFAT) method involves adjusting a single variable while keeping all others constant. However, it cannot detect interactions between variables, often leading to suboptimal outcomes when...
01:09Comparing Experimental Results: Student's t-Test
The t-test is a statistical method used to compare the sample mean with a population mean or compare two means from two data sets. The test statistic is calculated from the standard deviation, mean, and number of measurements in the data set at a selected confidence interval and then compared to a table of critical values at this confidence level. If the test statistic is smaller than the critical value, the null hypothesis is accepted. In this case, we state that the difference between the...
01:52Regression Toward the Mean
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when researchers try to extrapolate results...