Related Experiment Video
Updated: Jul 11, 2025
Author Spotlight: Workflow for Integrating POCUS Data into EHR for Managing Heart Failure Patients
Published on: July 12, 2024
Natural Language Processing for Adjudication of Heart Failure in a Multicenter Clinical Trial: A Secondary Analysis
Jonathan W Cunningham1,2, Pulkit Singh3, Christopher Reeder3
1Division of Cardiovascular Medicine, Brigham and Women's Hospital, Boston, Massachusetts.
Insights
Natural language processing (NLP) models show good agreement for heart failure (HF) hospitalization adjudication in multicenter trials, validating their use as a resource-efficient alternative to physician review.
Area of Science:
- Clinical Trials
- Health Informatics
- Cardiology
Background:
- Physician-led clinical events committees (CECs) are the gold standard for outcome adjudication in clinical trials but are resource-intensive.
- Automated adjudication using natural language processing (NLP) offers a potential alternative but requires validation in multicenter settings.
Purpose of the Study:
- To externally validate the Community Care Cohort Project (C3PO) NLP model for heart failure (HF) hospitalization adjudication in a multicenter clinical trial.
- To compare NLP model performance against the gold-standard CEC adjudication.
Main Methods:
- Retrospective analysis of the Influenza Vaccine to Effectively Stop Cardio Thoracic Events and Decompensated Heart Failure (INVESTED) trial data.
- Independent adjudication of hospitalizations by the central INVESTED CEC and the C3PO NLP model.
- Fine-tuning the C3PO NLP model and training a de novo NLP model using INVESTED trial data.
Main Results:
- The C3PO NLP model demonstrated good agreement with CEC adjudications (raw agreement, 87%; κ, 0.69).
- The model achieved high sensitivity (94%) and specificity (84%) for HF hospitalization identification.
- Fine-tuned and de novo NLP models showed improved agreement (93% and higher) with CECs, approaching human interrater reproducibility (94%).
Conclusions:
- The C3PO NLP model effectively adjudicates HF hospitalizations in a multicenter trial, supporting its use as a resource-efficient alternative to CECs.
- Model fine-tuning enhances agreement and approaches human reproducibility.
- Further research is needed to assess NLP's efficiency in identifying clinical events at scale in future multicenter trials.
Importance:
The gold standard for outcome adjudication in clinical trials is medical record review by a physician clinical events committee (CEC), which requires substantial time and expertise. Automated adjudication of medical records by natural language processing (NLP) may offer a more resource-efficient alternative but this approach has not been validated in a multicenter setting.
Objective:
To externally validate the Community Care Cohort Project (C3PO) NLP model for heart failure (HF) hospitalization adjudication, which was previously developed and tested within one health care system, compared to gold-standard CEC adjudication in a multicenter clinical trial.
Design, Setting, And Participants:
This was a retrospective analysis of the Influenza Vaccine to Effectively Stop Cardio Thoracic Events and Decompensated Heart Failure (INVESTED) trial, which compared 2 influenza vaccines in 5260 participants with cardiovascular disease at 157 sites in the US and Canada between September 2016 and January 2019. Analysis was performed from November 2022 to October 2023.
Exposures:
Individual sites submitted medical records for each hospitalization. The central INVESTED CEC and the C3PO NLP model independently adjudicated whether the cause of hospitalization was HF using the prepared hospitalization dossier. The C3PO NLP model was fine-tuned (C3PO + INVESTED) and a de novo NLP model was trained using half the INVESTED hospitalizations.
Main Outcomes And Measures:
Concordance between the C3PO NLP model HF adjudication and the gold-standard INVESTED CEC adjudication was measured by raw agreement, κ, sensitivity, and specificity. The fine-tuned and de novo INVESTED NLP models were evaluated in an internal validation cohort not used for training.
Results:
Among 4060 hospitalizations in 1973 patients (mean [SD] age, 66.4 [13.2] years; 514 [27.4%] female and 1432 [72.6%] male]), 1074 hospitalizations (26%) were adjudicated as HF by the CEC. There was good agreement between the C3PO NLP and CEC HF adjudications (raw agreement, 87% [95% CI, 86-88]; κ, 0.69 [95% CI, 0.66-0.72]). C3PO NLP model sensitivity was 94% (95% CI, 92-95) and specificity was 84% (95% CI, 83-85). The fine-tuned C3PO and de novo NLP models demonstrated agreement of 93% (95% CI, 92-94) and κ of 0.82 (95% CI, 0.77-0.86) and 0.83 (95% CI, 0.79-0.87), respectively, vs the CEC. CEC reviewer interrater reproducibility was 94% (95% CI, 93-95; κ, 0.85 [95% CI, 0.80-0.89]).
Conclusions And Relevance:
The C3PO NLP model developed within 1 health care system identified HF events with good agreement relative to the gold-standard CEC in an external multicenter clinical trial. Fine-tuning the model improved agreement and approximated human reproducibility. Further study is needed to determine whether NLP will improve the efficiency of future multicenter clinical trials by identifying clinical events at scale.
Related Concept Videos
01:30Heart Failure IV: Classification and Diagnostic Evaluation
01:22Heart Failure Drugs: β-Blockers
01:26Heart Failure III: Clinical Manifestations
01:30Heart Failure VII: Nursing Interventions
01:26Heart Failure Drugs: Inhibitors of Renin-Angiotensin System
01:22Heart Failure VI: Adjunct Therapies