Scale development with small samples: a new application of longitudinal item response theory

Area of Science:

• Psychometrics
• Statistical Modeling
• Clinical Research Methodology

Background:

• Measurement development in hard-to-reach populations presents significant methodological challenges.
• Item Response Theory (IRT) is a powerful statistical tool for measurement, but typically requires large sample sizes.
• Existing methods are often inadequate for developing instruments in populations with limited participant numbers.

Purpose of the Study:

• To describe the application of longitudinal IRT models as a pragmatic approach for instrument development when large samples are not feasible.
• To demonstrate the utility of longitudinal IRT in overcoming sample size limitations in measurement development.
• To provide a practical framework for researchers working with difficult-to-reach populations.

Main Methods:

• The study describes the statistical foundations and practical benefits of longitudinal IRT models.
• A simulation study was conducted to assess the model's ability to recover measurement structure and parameters across various sample sizes, item counts, and time points.
• The methods were exemplified using early-phase clinical trial data from a study on a rare condition.

Main Results:

• Simulation results indicate that the performance of longitudinal IRT models in parameter recovery depends on the interplay between sample size and the number of time points.
• The models demonstrate good performance even with small sample sizes, provided sufficient repeated measurements are available.
• Application to clinical trial data showed that conditional, longitudinal IRT models yield stable psychometric estimates from small samples.

Conclusions:

• Longitudinal IRT modeling allows for the estimation of psychometric characteristics from small sample sizes by leveraging repeated measurements.
• This approach enhances confidence in subsequent score comparisons and optimizes study designs for challenging populations.
• Longitudinal IRT is particularly advantageous for developing measures in rare diseases and hard-to-reach groups, despite potential limitations.