Validation of Immune-Related Adverse Event (irAE) Case Definitions in a Real-World Lung Cancer Population

The use of real-world data is increasing to examine immune-related adverse event (irAE) incidence and risk factors in immune checkpoint inhibitor (ICI) users. We aimed to validate five case definition algorithms for irAE in a Johns Hopkins lung cancer registry.

We conducted a retrospective cohort study using linked electronic health record (EHR) and cancer registry data from a large academic healthcare system. The Lung Immunotherapy irAE Monitoring Registry assesses irAEs in a group of patients treated for lung cancer at Johns Hopkins Medicine from 2013 to 2020. We used data from inpatient, outpatient, and emergency department encounters, including International Classification of Disease (ICD)-10 codes and medication administration records to classify the presence or absence of irAEs using five distinct algorithms. These algorithms included three that used both diagnosis (Dx) and medication (Rx) codes, one that used Rx codes only, and one that used Dx codes only, ranging from most numerous criteria (most stringent) to least numerous criteria (least stringent). We compared all five algorithms' performances against chart review-ascertained irAE status and reported sensitivity (Se), specificity (Sp), positive predictive value (PPV), negative predictive value (NPV), and C-statistic (C-stat), with 95% confidence intervals (CI). We also explored algorithm performance by specific organ system toxicities and by Common Terminology Criteria for Adverse Events (CTCAE) severity.

The study cohort included 354 patients with ICI exposure for whom chart review-ascertained irAE status was available. A total of 89 (25.1%) experienced at least one irAE (38 pneumonitis, 12 arthritis, 12 colitis, 7 thyroiditis, and others). Across algorithm versions, Se ranged from 59.3% to 93.2% in descending order of algorithm stringency; Sp ranged from 21.0% to 77.6% in ascending order of algorithm stringency, and PPV ranged from 19.1% to 34.7%. The C-stat ranged from 0.57 (95% CI, 0.53-0.61) (Dx codes only) to 0.71 (0.64-0.77) (Rx codes only). For severe irAE (CTCAE Grade 3-5), all algorithms performed better than in the primary analysis, and four exceeded the threshold for usefulness as a measurement tool (maximum C-stat: 0.78 [0.71-0.85] [Rx codes only]). For severe tissue-specific toxicities, algorithmic detection of irAE pneumonitis, colitis, and hepatitis performed better than for the overall group of severe toxicities. Generally, the algorithm versions depicted a Se-Sp tradeoff depending on algorithm stringency.

In this validation study of five irAE case definition algorithms, a combination of ICD-10 codes and medication administration codes generally perform well to identify more severe irAE (CTCAE Grade 3-5), and severe pneumonitis, hepatitis, and colitis (common irAEs) among all possible irAE severity levels and sites. Medication codes alone perform well at identifying severe irAE, while the most stringent algorithm (mirroring guideline-recommended irAE treatment) has the highest Sp and PPV. Algorithms have utility for comparing the relative risk of irAE between regimens or patient subgroups.