Combining human and machine intelligence for clinical trial eligibility querying.

Area of Science:

• Clinical Informatics
• Natural Language Processing
• Biomedical Data Science

Background:

• Converting complex clinical trial eligibility criteria into computable cohort queries is challenging.
• Existing methods often lack the flexibility to handle nuanced language and require significant manual effort.
• Automated tools struggle with the inherent complexity and ambiguity of clinical trial protocols.

Purpose of the Study:

• To develop and evaluate Criteria2Query (C2Q) 2.0, a system that integrates machine efficiency with human intelligence for converting clinical trial eligibility criteria into cohort queries.
• To enhance modules for negation scope detection, temporal, and value normalization.
• To assess the usability and usefulness of C2Q 2.0 with domain experts.

Main Methods:

• C2Q 2.0 was developed with real-time user intervention capabilities for criteria selection, simplification, error correction, and concept mapping.
• Enhanced modules for negation scope detection, temporal, and value normalization were evaluated against a gold standard of 1010 COVID-19 trial eligibility criteria.
• Usability and usefulness were assessed via a task-oriented evaluation with 10 research coordinators using 5 Alzheimer's disease trials, collecting data through logs, questionnaires, and usability scales.

Main Results:

• The enhanced modules achieved high accuracies: negation scope detection (0.924), temporal normalization (0.916), and value normalization (0.966).
• C2Q 2.0 demonstrated moderate usability (3.84/5) and high learnability (4.54/5).
• Users favored modified cohort queries (4.1/5) and user engagement features (4.3/5), with an average of 9.9 modifications per study.

Conclusions:

• Human-computer collaboration, facilitated by features engaging domain experts, is crucial for overcoming limitations in automated natural language processing outputs.
• C2Q 2.0 proves useful and user-friendly, enhancing the adoption and effectiveness of NLP in clinical trial recruitment.
• Integrating human intelligence with machine capabilities is key to improving the efficiency and accuracy of cohort query generation from complex eligibility criteria.