Apples-to-Apples: Age-Sex Standardisation of Public Chest X-ray Datasets

Area of Science:

• Medical Imaging
• Radiology Informatics
• Biostatistics

Background:

• Public chest radiograph datasets are crucial for developing and benchmarking AI models.
• Demographic variations across datasets can distort disease prevalence estimates, impacting model evaluation.
• NIH ChestX-ray14 and CheXpert are widely used but have differing demographic compositions.

Purpose of the Study:

• To quantify the impact of patient age and sex on prevalence differences between the NIH ChestX-ray14 and CheXpert datasets.
• To determine the proportion of observed prevalence gaps attributable solely to demographic factors.

Main Methods:

• Cross-sectional analysis of NIH ChestX-ray14 (n=112,120) and CheXpert (n=223,413) datasets.
• Harmonized age categories and sex (Male/Female); treated CheXpert uncertainty as negative.
• Calculated crude prevalence, compared datasets using two-proportion z-tests, and performed direct age-sex standardization.

Main Results:

• CheXpert showed higher crude prevalence for all five assessed labels (consolidation, atelectasis, pleural effusion, edema, cardiomegaly) compared to NIH.
• Age-sex standardization reduced the prevalence gap for all labels, explaining 25% to 40% of the differences.
• For example, the gap for atelectasis reduced by approximately 41% (4.85 to 2.84 percentage points).

Conclusions:

• Age and sex are significant confounders in comparing disease prevalence between chest radiograph datasets.
• Simple age-sex standardization substantially reduces apparent prevalence differences, improving cross-dataset comparability.
• Routine reporting of standardized prevalence alongside demographic data enhances fairness and interpretability in AI benchmarking.