Development and Validation of an Artificial Intelligence Tool for Automated Population, Intervention, Comparator,...

The European Union Health Technology Assessment regulation (EU-HTAR) requires laborious population, intervention, comparator, outcome (PICO) scoping and consolidation across member states during joint clinical assessments. We developed a tool to automate PICO extraction from health technology assessment (HTA) documents and clinical guidelines.

We developed a proof-of-concept artificial intelligence (AI) pipeline using retrieval-augmented generation and large language models to process documents, translate to English, extract PICOs, and consolidate findings. The system was validated using 2 oncology cases from nonsmall cell lung cancer (NSCLC) and hepatocellular carcinoma (HCC). We curated data sets for both cases, comprising 49 reports totaling 5463 pages from 16 unique countries. Performance was assessed using recall (sensitivity) and precision (positive predictive value) and compared with human data extraction.

The AI system demonstrated strong recall performance, achieving 0.807 for NSCLC and 0.837 for HCC. The AI outperformed human extraction in the NSCLC case (0.81 vs 0.71 recall) while matching human performance in HCC (0.84 vs 0.81). Extraction of the comparator element showed high recall (0.911 NSCLC; 0.967 HCC). The AI pipeline required 96 minutes of computational time beyond translation, saving approximately 25 hours of human screening work.

We showed that AI-based PICO extraction was feasible and valuable in 2 use-cases. The system showed good recall and had the potential to reduce workload in the joint clinical assessment. It may be useful for the assessors and coassessors in the EU-HTAR process, the industry in gaining insight into PICO expectations, and national HTA bodies in understanding the expected PICO wishes of other nations.