Artificial Intelligence Applications Versus Manual Methods For Literature Retrieval: A Comparative Analysis

Area of Science:

• Nursing Education
• Scholarly Writing
• Artificial Intelligence

Background:

• Generative artificial intelligence (AI) and large language models are increasingly used in nursing education, practice, and scholarly writing.
• AI applications show promise in reducing production time for scholarly work, particularly literature reviews.
• However, the accuracy of AI in identifying references for literature reviews remains under-investigated.

Purpose of the Study:

• To compare the accuracy of AI-generated citations with human-generated citations in literature reviews.
• To evaluate the performance of AI literature review applications in reference identification.

Main Methods:

• A comparative exploratory design was employed.
• References from four human-written literature reviews (two published, two unpublished) were compared with references from two AI applications (Consensus and Elicit).
• Three prompting strategies, including ChatGPT-4, were used to evaluate AI performance.

Main Results:

• The agreement between AI and human-generated reference lists varied significantly, ranging from 0% to 63.6%.
• The Consensus application demonstrated a higher mean match rate (21.3%) compared to Elicit (3.7%).
• The use of ChatGPT-4 prompts and the publication status of reviews did not significantly affect the results.

Conclusions:

• The examined AI literature review applications have potential but also limitations.
• AI tools may support, but not substitute, human effort in scholarly literature review processes.