Penalized GEE for Complex Carry-Over in Repeated-Measures Crossover Designs

Crossover designs are commonly employed in clinical and behavioral research, yet the statistical models used to analyze them often rely on unrealistic assumptions-either ignoring carry-over effects or modeling them as simple and homogeneous across treatment sequences. However, carry-over effects are frequently complex, varying by treatment order and interaction, and until now, no statistical methodology had been formally established to estimate such complex effects. This paper introduces a penalized semiparametric Generalized Estimating Equations (GEE) approach designed to estimate first order complex carry-over effects in crossover designs with repeated measurements. We first derive identifiability conditions under which complex carry-over effects become estimable. We then provide theoretical guarantees-building on an extension of the sandwich variance formula-showing that the proposed penalized estimator achieves asymptotic normality for the functional components and shrinks negligible carry-over effects toward zero, thereby enabling their practical identification. Through simulation studies and application to real data, the methodology demonstrates improved estimation accuracy when complex carry-over effects are present, outperforming models that assume simple or no carry-over. This work represents the first rigorous and generalizable approach for modeling complex carry-over effects in repeated-measures crossover designs.