Development of an expert-annotated chest X-ray dataset to support AI validation in tuberculosis diagnosis

To assess inter-rater agreement and diagnostic performance of six United States National Institute for Occupational Safety and Health-certified B readers (physicians certified in standardized chest X-ray (CXR) classification for pneumoconiosis) in diagnosing tuberculosis (TB) on CXR, with the goal of developing a reliable dataset to support external validation of artificial intelligence (AI) models for TB detection.

1097 CXRs from five institutions were analyzed by six B readers. Patients aged ≥ 15 years were included, excluding those with HIV or opportunistic infections. CXRs were classified as unremarkable or abnormal. Abnormalities were categorized using a modified International Labor Organization classification and classified as consistent or inconsistent with TB. Microbiological references included sputum smears, cultures, or molecular tests. Descriptive statistics summarized the CXR and microbiological findings. Inter-rater agreement was assessed with Fleiss' kappa. Diagnostic performance was evaluated by comparing CXR findings to microbiological references.

Of the 3117 readings, 69% of CXRs were abnormal, and 31% were unremarkable. Microbiological results confirmed that 87% of abnormal CXRs were TB cases, while 83% of unremarkable CXRs were non-TB. Inter-rater agreements were κ = 0.83 for all findings, κ = 0.67 for findings consistent with TB, and κ = 0.76 for active TB findings. For findings consistent with TB, the sensitivity, specificity, and accuracy ranged from 77.2% to 91.1%, 87.4% to 98.6%, and 84.1% to 90.1%, respectively.

B readers exhibited strong agreement and high accuracy in diagnosing TB on CXR, providing a robust dataset that could be used for external validation of AI models in TB diagnosis.

With substantial inter-rater agreement and strong diagnostic performance, B readers' interpretations of CXRs provide a reliable dataset to support external validation of AI models for TB diagnosis.

B readers can provide a reliable reference for validating AI models in TB diagnosis. Less variability and high accuracy were observed among B readers in CXR-based TB diagnosis.