Next-generation multicenter studies: using artificial intelligence to automatically process unstructured health...

Manual abstraction of real-world data (RWD) from unstructured health records (HRs) remains resource intensive, error prone, and highly variable across institutions. Large language models (LLMs) offer a scalable alternative, but their performance in multicenter oncology settings is not fully validated.

We conducted a multicenter study within the French Large & Unified Cancer Cohort (LUCC) consortium to compare the accuracy of artificial intelligence (AI)-based data extraction against manual abstraction by clinical research professionals. A fine-tuned LLM was applied to de-identified unstructured HRs in PDF format to extract 31 variables from lung cancer patients across 10 centers. Ground truth was defined as concordant values across sources, with discrepant cases adjudicated by a blinded expert. The primary endpoint was the extraction error rates. Secondary endpoints included per-variable performance, interinstitutional variability, F1-score for multiple-choice variables, added value of hybrid AI-human workflows, and survival analyses.

Among 10 327 patients with AI-based extraction, 311 were included in the test cohort. Across 8708 datapoints for 28 variables with only one correct answer, the LLM achieved a 7.0% error rate, outperforming manual abstraction (14.2%, P <0.001). The F1-scores of three multiple-choice variables were superior (gene alterations 0.97 versus 0.86, comorbidities 0.86 versus 0.76, metastatic sites 0.71 versus 0.69). Interinstitutional variance was lower with AI (0.12% versus 0.39%). A hybrid approach with targeted human review of 30% of low-confidence AI outputs further decreased error rates to 4.4%. Survival analyses based on AI-extracted data closely matched ground truth, with similar median overall and progression-free survival.

In a multicenter setting, our AI pipeline yielded lower error rates and greater consistency than manual abstraction. These findings support the feasibility of next-generation, AI-enabled multicenter studies to generate high-quality RWD at scale, with potential applicability in prospective clinical trials.