Generalizability theory reliability of written expression curriculum-based measurement in universal screening

Area of Science:

• Educational Psychology
• Psychometrics
• Academic Assessment

Background:

• Curriculum-based measurement (CBM) is widely used for progress monitoring and screening.
• Written expression CBM (WE-CBM) is a common tool, but its reliability for universal screening needs further examination.
• Generalizability theory provides a framework to assess the sources of measurement error.

Purpose of the Study:

• To evaluate the reliability of written expression curriculum-based measurement (WE-CBM) within a universal screening context.
• To apply generalizability theory to understand the variance components in WE-CBM scores.
• To determine the number and duration of WE-CBM probes needed for reliable student performance decisions.

Main Methods:

• A generalizability theory framework was used to analyze WE-CBM reliability.
• Three WE-CBM probes (7 minutes each) were administered to 145 students in grades 2-5 across three time points over one year.
• Writing samples were scored using metrics like correct minus incorrect word sequences (CIWS).

Main Results:

• Nearly half of the variance in WE-CBM scores was attributed to unsystematic error.
• Conventional screening procedures using a single 3-minute sample yielded inadequate reliability for relative or absolute decisions.
• Multiple probes (e.g., three 3-minute or two longer samples) were needed for reliable relative decisions, with three 7-minute samples insufficient for within-year growth decisions.

Conclusions:

• Current WE-CBM screening practices are unreliable for making critical student performance decisions.
• More extensive or frequent sampling is necessary to achieve adequate reliability in WE-CBM.
• Recommendations are provided for improving the reliability and utility of WE-CBM in educational settings.