Criteria2Query 3.0: Leveraging generative large language models for clinical trial eligibility query generation

Area of Science:

• Clinical Informatics
• Artificial Intelligence in Medicine
• Health Data Science

Background:

• Automated patient identification for clinical research is a significant challenge.
• Existing methods for translating eligibility criteria into database queries are often manual and time-consuming.
• The Criteria2Query (C2Q) system aims to streamline this process.

Purpose of the Study:

• To introduce and evaluate Criteria2Query (C2Q) 3.0, a system utilizing GPT-4 for semi-automatic conversion of clinical trial eligibility criteria into executable database queries.
• To assess the performance of GPT-4 in concept extraction and SQL query generation for clinical trial eligibility criteria.

Main Methods:

• C2Q 3.0 employed three GPT-4 prompts for concept extraction, SQL query generation, and reasoning.
• Concept extraction was benchmarked against manual annotations from 20 clinical trials.
• SQL generation accuracy and reasoning quality were evaluated by domain experts on subsets of clinical trials.

Main Results:

• GPT-4 achieved an F1-score of 0.891 for concept extraction from 518 concepts.
• The system identified 29 errors in SQL generation, with logic errors being most frequent (34.48%).
• Reasoning evaluation showed high coherence (mean 4.70) and usefulness (mean 4.37), but lower readability (mean 3.95).

Conclusions:

• GPT-4 significantly enhances the accuracy of extracting clinical trial eligibility criteria concepts within the C2Q 3.0 system.
• The findings highlight the potential of large language models in clinical research data management.
• Continued investigation is necessary to fully establish the reliability of large language models for clinical applications.