Planned missing data in early literacy interventions: A replication study with an additional gold standard
Ralph C A Rippe1, Inge Merkelbach2
1Research Methods and Statistics, Institute of Education and Child Studies, Leiden University, Leiden, The Netherlands.
Insights
Replication of early literacy intervention results requires precise "gold standard" measures. Only the most accurate assessments successfully replicated findings, highlighting the importance of high-quality data in research.
Area of Science:
- Child Development
- Educational Psychology
- Research Methodology
Background:
- Late preterm infants in digital early literacy interventions showed varied outcomes.
- Replication of previous findings was complicated by the quality of assessment measures used.
- Gold standard measures, while sensitive, are often time-intensive and costly.
Purpose of the Study:
- To investigate the impact of different definitions of "gold standard" measures on the replication of digital early literacy intervention results.
- To explore how measure precision affects the reliability of findings in randomized controlled trials (RCTs).
Main Methods:
- A randomized controlled trial (RCT) involving 695 primary school pupils (normal and late preterm birth).
- Comparison of simpler, complete measures with high-quality, potentially incomplete "gold standard" measures in a subsample.
- Evaluation of five different definitions of gold standard alignment.
Main Results:
- Two out of five gold standard definitions improved precision compared to original results.
- The least precise gold standard definition diminished precision.
- Only the most precise gold standard definition successfully replicated the original study findings.
- Specific factors like alphabetical and writing-only measures yielded comparable estimates to original results.
Conclusions:
- The successful replication of RCT results using gold standard measures is contingent upon sufficiently high convergent validity.
- Careful selection and definition of high-quality measures are crucial for reliable research outcomes in early literacy interventions.
Introduction:
In a digital early literacy intervention RCT, children born late preterm fell behind peers when in a control condition, but outperformed them when assigned to the intervention. Results did however not replicate previous findings. Replication is often complicated by resource quality. Gold Standard measures are generally time-intensive and costly, while they closely align with, and are more sensitive to changes in, early literacy and language performance. A planned missing data approach, leaving these gold standard measures incomplete, might aid in addressing the origin(s) of non-replication.
Methods:
Participants after consent were 695 p Dutch primary school pupils of normal and late preterm birth. The high-quality measures, in additional to simpler but complete measures, were intentionally administered to a random subsample of children. Five definitions of gold standard alignment were evaluated.
Results:
Two out of five gold standard levels improved precision compared to the original results. The lowest gold standard level did not lead to improvement: precision was actually diminished. In two gold standard definitions, an alphabetical factor and a writing-only factor the model estimates were comparable to the original results. Only the most precise definition of the gold standard level replicated the original results.
Conclusion:
Gold standard measures could only be used to improve model efficiency in RCT-designs under sufficiently high convergent validity.
