Sampling for computational efficiency when conducting analyses in big data.

Area of Science:

• Epidemiology
• Biostatistics
• Health Informatics

Background:

• Big data research presents computational challenges, especially with bias-correction methods.
• Efficiently analyzing large datasets is crucial for public health research.

Purpose of the Study:

• To evaluate sampling methods for estimating lung cancer incidence by HIV status in a large cohort.
• To compare the accuracy, speed, and resource usage of different sampling techniques.

Main Methods:

• Utilized a cohort of nearly 30 million Medicaid beneficiaries to assess lung cancer incidence.
• Employed sampling schemes: divide-and-recombine, sub-cohort, and case-cohort, alongside inverse probability weighting for confounder control.
• Estimated incidence rate ratio (IRR), hazard ratio (HR), and risk ratio (RR).

Main Results:

• Observed 1113 lung cancer diagnoses in beneficiaries with HIV and 33,106 in those without.
• Sub-cohort and case-cohort sampling yielded estimates comparable to the full sample.
• These methods were faster and required less memory than divide-and-recombine, particularly for risk ratio estimation.

Conclusions:

• Sampling methods, specifically sub-cohort and case-cohort, provide efficient and accurate parameter estimation in big data analyses.
• These approaches reduce computational burden without significantly compromising results.
• HIV status may influence lung cancer incidence, warranting further investigation.