Development and Evaluation of a Generative AI Chatbot for Database Searching in Systematic Review

Area of Science:

• Information Science
• Medical Librarianship
• Artificial Intelligence in Research

Background:

• Systematic reviews (SRs) demand extensive, reproducible searches, a process that is time-consuming and requires specialized expertise.
• Generative AI (GAI) presents an opportunity to simplify SR search strategy development, but empirical evidence is limited.

Purpose of the Study:

• To outline a method for creating a custom ChatGPT-based chatbot for SR search strategy development.
• To assess the performance of this GAI-assisted approach.

Main Methods:

• A cross-sectional evaluation utilized ChatGPT-4.0 to develop a chatbot mimicking a medical librarian for PICO-informed searches.
• The chatbot was trained on methodological references and refined through pilot testing.
• Queries were generated for 50 Cochrane SRs (2024) using P-I-O prompts for PubMed and Embase.

Main Results:

• The chatbot achieved an 83.7% retrieval rate during pilot testing.
• In the main evaluation, the chatbot identified a median of 67.4% of included studies (72.0% for indexed studies).
• Performance decreased when outcomes were not in abstracts or interventions had varied terminology.

Conclusions:

• GAI chatbots offer a rapid method for generating initial SR search strategies, achieving moderate study identification rates.
• These tools serve as valuable starting points but cannot fully replace the expertise of information professionals.
• Enhancing GAI performance may involve integrating librarian expertise, structured prompts, and controlled vocabularies, necessitating further research and transparent reporting.