Automating Adjudication of Cardiovascular Events Using Large Language Models

Area of Science:

• Artificial Intelligence
• Clinical Trials
• Medical Informatics

Background:

• Cardiovascular event adjudication in clinical trials is critical but traditionally manual, leading to delays, inconsistencies, and high costs.
• Manual review of clinical documents for event adjudication is time-consuming and prone to human error.

Purpose of the Study:

• To develop and evaluate a two-stage framework using large language models (LLMs) to automate the adjudication of cardiovascular deaths in clinical trials.
• To improve the efficiency, consistency, and transparency of cardiovascular event adjudication.

Main Methods:

• A two-stage framework employing large language models (LLMs) for automated adjudication.
• Stage 1: A few-shot LLM extracts structured evidence (event, negation, date, span) from unstructured clinical documents.
• Stage 2: A Tree-of-Thoughts adjudicator uses Clinical Endpoint Committee (CEC) guidelines for classification and rationale generation.

Main Results:

• The LLM-based framework achieved high precision (0.96) and F1 score (0.82) in extracting structured evidence from clinical documents.
• The adjudication stage demonstrated 0.68 accuracy using GPT-4 Tree-of-Thoughts, outperforming a baseline summarizer-plus-adjudicator approach.
• The CLEART score (0.67) quantified rationale quality, identifying temporal reasoning and relevance as areas for improvement.

Conclusions:

• Automated adjudication of cardiovascular deaths using LLMs offers a promising solution to enhance efficiency and reduce variability in clinical trials.
• The proposed framework provides an auditable rationale, increasing transparency in the adjudication process.
• Further refinement is needed to optimize temporal reasoning and relevance in automated adjudication systems.