Related Experiment Videos
Seemingly minor changes to a questionnaire can make a big difference to mean scores: a cautionary tale
Robert Goodman1, Alessandra C Iervolino, Stephan Collishaw
1King's College London, Institute of Psychiatry, London, UK. r.goodman@iop.kcl.ac.uk
Social Psychiatry and Psychiatric Epidemiology
|March 6, 2007
Summary
Minor modifications to research instruments like the Rutter parent questionnaire can significantly alter scores, impacting comparability. Researchers must validate changes to ensure reliable psychometric properties and consistent findings.
Area of Science:
- Psychometrics
- Child Mental Health Research
- Clinical Psychology
Background:
- Researchers often modify existing instruments without assessing psychometric properties.
- Altering measurement tools can inadvertently affect data interpretation.
Purpose of the Study:
- To compare psychometric properties of a standard and modified Rutter parent questionnaire.
- To investigate the impact of minor wording changes on child mental health measures.
Main Methods:
- Compared equivalent child mental health items from standard and modified Rutter questionnaires.
- Randomized 400 parents of children aged 5-7 years to complete one Rutter version and the Strengths and Difficulties Questionnaire (SDQ).
Main Results:
- Mean psychopathology scores differed significantly between Rutter versions due to response category changes.
- The Strengths and Difficulties Questionnaire (SDQ) showed comparable mean scores across groups.
- Validity, assessed by Rutter-SDQ correlations, was similar for both Rutter versions.
Conclusions:
- Even minor wording modifications in research instruments can substantially alter mean scores.
- This impacts the ability to compare or combine results from studies using different versions of a measure.
- Careful validation of modified instruments is crucial for research integrity.
Related Concept Videos
01:52Regression Toward the Mean
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when researchers try to extrapolate results...
02:16Surveys
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
01:19Testing a Claim about Standard Deviation
A complete procedure to test a claim about population standard deviation or population variance is explained here.
The hypothesis testing for the claim of population standard deviation (or variance) requires the data and samples to be random and unbiased. The population distribution also must be normal. There is no specific requirement on the sample size as the estimation is based on the chi-square distribution.
As a first step, the hypothesis (null and alternative) concerning the claim about...
The hypothesis testing for the claim of population standard deviation (or variance) requires the data and samples to be random and unbiased. The population distribution also must be normal. There is no specific requirement on the sample size as the estimation is based on the chi-square distribution.
As a first step, the hypothesis (null and alternative) concerning the claim about...
01:15One-Way ANOVA: Equal Sample Sizes
One-Way ANOVA can be performed on three or more samples with equal or unequal sample sizes. When one-way ANOVA is performed on two datasets with samples of equal sizes, it can be easily observed that the computed F statistic is highly sensitive to the sample mean.
Different sample means can result in different values for the variance estimate: variance between samples. This is because the variance between samples is calculated as the product of the sample size and the variance between the...
Different sample means can result in different values for the variance estimate: variance between samples. This is because the variance between samples is calculated as the product of the sample size and the variance between the...
01:37Statistical Significance
Once data is collected from both the experimental and the control groups, a statistical analysis is conducted to find out if there are meaningful differences between the two groups. A statistical analysis determines how likely any difference found is due to chance (and thus not meaningful). In psychology, group differences are considered meaningful, or significant, if the odds that these differences occurred by chance alone are 5 percent or less. Stated another way, if we repeated this...
01:29Reliability and Validity
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.