Reliability of cancer screening questions from the National Health Interview Survey

The National Health Interview Survey (NHIS), a nationwide in-person and telephone survey of the civilian non-institutionalized population, is used to measure progress towards Healthy People objectives on cancer screening. Survey reports have been compared to medical record data to assess validity of these questions; however, few studies have assessed reliability, or response consistency over time. We assessed the test-retest reliability of questions on cervical, colorectal, breast and lung cancer screening from the 2021 and 2022 NHIS.

We surveyed 1,770 adults ages 18 + years with and without prior screening for breast, cervical, colorectal, and lung cancer from four U.S. health systems (three in Washington state and one in Michigan) to examine validity of the NHIS questions. Among people who completed the validity survey, we randomized individuals to receive an additional survey one month later (N = 944) and three months later (N = 822). We calculated Cohen's Kappa, Gwet's AC1, and simple concordance to assess reliability.

Among our study sample, 535 (56.7% of those assigned) to one month follow-up and 537 (65.0%) assigned to three month follow-up completed the reliability survey. Among these, 36 (6.6%) and 94 (17.6%) of those assigned to one and three month follow-up respectively were excluded from reliability analyses because they had additional screening tests after completing the validity survey. Cohen's Kappa showed strong-to-moderate reliability for being up to date with mammography screening: 0.83 at one month and 0.78 at three months; reliability for other screening tests was considerably lower: 0.61 and 0.60 for cervical, 0.57 and 0.62 for colorectal, and 0.72 and 0.58 for lung and one and three months, respectively.

We found strong-to-moderate reliability of responses recalling breast cancer screening at one and three months following an initial survey. Reliability may not be a major threat to self-reported cancer screening behavior.