Related Experiment Video
Updated: Nov 12, 2025
In Vivo Vascular Injury Readouts in Mouse Retina to Promote Reproducibility
Published on: April 21, 2022
Calculation of test-retest variability in phase I/IIa clinical trials for Inherited Retinal Degenerations
Jacob Jeries Abou-Hanna1,2, Chris A Andrews3,4, Naheed W Khan3
1Department of Ophthalmology and Visual Sciences, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, Wisconsin, USA.
Abstract:
Background: Several novel treatments of inherited retinal degenerations have undergone phase I/IIa clinical trials with limited sample size, yet investigators must still determine if toxicity or an efficacy signal occurred or if the change was due to test-retest variability (TRV) of the measurement tool.Materials and Methods: Synthetic datasets were used to compare three types of TRV estimators under different sample sizes, mean drift, skewness, and number of baseline measurements.Results: Mixed effects models underestimated the standard deviation of measurement error (SDEM); the unbiased change score estimator method (UBS) was more accurate. The fixed effect model had less bias and smaller standard deviation than UBS if >2 baseline measurements. The change score estimator had no bias; other estimators introduced bias for lower variability. With sample size <10, all estimators had high variance. With sample size ≥10, the differences between methods were often minimal. The pooled estimator model did not capture drift, whereas a fixed effect regression or mixed effects models accounted for drift while maintaining an accurate measure of variance. With small sample sizes, the bootstrap estimates of SDEM were severe underestimates, while the jackknife estimates were mildly low but much better. The jackknife was more accurate for the unbiased change score method than for the pooled estimator.Conclusions: The ideal phase I/IIa study has ≥20 subjects and uses UBS or its fixed effect model generalization if >2 baseline measurements. With non-ideal study parameters, investigators should at least quantify the error estimate present in their data analysis.
Insights
For inherited retinal degeneration trials, the unbiased change score estimator (UBS) is best for determining treatment effects, especially with fewer than 10 subjects. Larger sample sizes (≥10) minimize differences between TRV estimators.
Area of Science:
- Ophthalmology
- Clinical Trial Design
- Biostatistics
Background:
- Novel treatments for inherited retinal degenerations require careful evaluation in early-phase clinical trials.
- Distinguishing true treatment effects from test-retest variability (TRV) is crucial but challenging with small sample sizes.
- Accurate estimation of measurement error is vital for interpreting trial outcomes.
Purpose of the Study:
- To compare the accuracy of different test-retest variability (TRV) estimators in simulated phase I/IIa clinical trials.
- To identify optimal TRV estimation methods under various conditions, including sample size and data characteristics.
- To provide guidance on selecting appropriate statistical methods for analyzing early-phase clinical trial data in ophthalmology.
Main Methods:
- Generation of synthetic datasets simulating clinical trial data with varying sample sizes, mean drift, skewness, and baseline measurements.
- Comparison of three TRV estimators: mixed-effects models, unbiased change score (UBS) estimator, and fixed-effect models.
- Evaluation of bootstrap and jackknife methods for estimating the standard deviation of measurement error (SDEM).
Main Results:
- The unbiased change score estimator (UBS) demonstrated higher accuracy than mixed-effects models in estimating measurement error.
- Fixed-effect models showed less bias and smaller standard deviation than UBS when more than two baseline measurements were available.
- All tested estimators exhibited high variance with sample sizes below 10; differences were minimal for sample sizes of 10 or greater.
- Pooled estimator models failed to account for data drift, unlike fixed-effect regression or mixed-effects models.
- Jackknife estimates provided more accurate SDEM than bootstrap estimates, particularly for the UBS method.
Conclusions:
- Phase I/IIa studies for inherited retinal degenerations ideally require at least 20 subjects.
- The unbiased change score (UBS) estimator or its fixed-effect model generalization (with >2 baseline measurements) is recommended.
- When ideal study parameters are not met, quantifying the error estimate within data analysis is essential.
More Related Videos
07:11Assessing Early Stage Open-Angle Glaucoma in Patients by Isolated-Check Visual Evoked Potential
Published on: May 25, 2020
10:11Fundus Photography as a Convenient Tool to Study Microvascular Responses to Cardiovascular Disease Risk Factors in Epidemiological Studies
Published on: October 22, 2014