Mastering rare event analysis: subsample-size determination in Cox and logistic regressions

Modern data science frequently encounters massive datasets that impose significant burdens on computational memory and processing time. Prior research has shown that optimal subsampling methods can reduce these burdens while minimizing the loss of statistical efficiency. These existing frameworks allow researchers to analyze smaller portions of a dataset to approximate the parameters of the full population. However, current methodologies often fail to provide a rigorous mathematical basis for deciding exactly how many observations are required for a specific analysis. This lack of guidance is particularly problematic when dealing with rare events where data points of interest are scarce. Statistical practitioners currently rely on heuristic approaches or arbitrary thresholds that may not preserve the necessary power for complex regressions. This absence of evidence motivated the development of formal tools to guide the selection of appropriate sample sizes within subsampling routines.

This research develops specific tools for determining the optimal subsample size across three distinct statistical modeling environments. The investigators focus on the Proportional Hazards (Cox) regression model for survival data characterized by infrequent occurrences. The study also addresses Logistic Regression (LR) within the context of both balanced and imbalanced datasets to ensure broad applicability. A secondary objective involves creating a novel optimal subsampling procedure specifically designed for imbalanced logistic data. The authors aim to bridge the existing gap between theoretical subsampling efficiency and practical implementation in large-scale studies. By providing these selection tools, the work seeks to optimize the trade-off between computational speed and statistical precision. The project intends to validate these new methodologies through both synthetic simulations and real-world massive data applications.

The researchers implemented an extensive simulation study to evaluate the performance of the proposed size-selection tools. They applied the Proportional Hazards (Cox) regression framework to a massive dataset from the UK Biobank containing colorectal cancer records. This specific survival analysis involved processing approximately 350 million rows of data to test the limits of the subsampling algorithm. For the Logistic Regression (LR) component, the team utilized linked birth and infant death records comprising roughly 28 million observations. The methodology incorporated a new optimal subsampling procedure tailored for imbalanced datasets where the event of interest is rare. Statistical comparisons focused on the minimized efficiency loss achieved by the selected subsample sizes relative to full-dataset benchmarks. The analytical framework prioritized the maintenance of predictive accuracy while drastically reducing the required computational memory.

The simulation study confirmed that the new tools effectively identify subsample sizes that maintain high statistical efficiency. Analysis of the UK Biobank colorectal cancer data demonstrated that the Cox regression model could be accurately estimated using the determined subsample. Results from the 28 million linked birth and infant death observations showed that the imbalanced logistic regression procedure performed reliably. The proposed tools successfully balanced the computational time requirements with the need for precise parameter estimation in rare event scenarios. The optimal subsampling procedure for imbalanced data yielded lower variance in coefficient estimates compared to standard random sampling. These findings suggest that the size-selection tools provide a robust solution for managing massive datasets without sacrificing analytical rigor. The empirical evidence supports the use of these tools across diverse data structures, including those with extreme class imbalances.

The development of these size-selection tools provides a necessary foundation for efficient large-scale data analysis in modern research. These findings imply that researchers can now make informed decisions regarding the volume of data needed for Cox and logistic regressions. The application to UK Biobank colorectal cancer data highlights the potential for these methods to accelerate epidemiological studies. Future investigations might apply these subsampling principles to other complex models beyond survival and logistic frameworks. The study establishes that imbalanced datasets require specialized subsampling procedures to ensure the accurate detection of rare events. Implementation of these tools could significantly reduce the environmental and financial costs associated with high-performance computing. The authors conclude that this framework represents a significant advancement in the practical utility of optimal subsampling for massive datasets.