Large language models vs. conventional search in identifying delirium guidelines for a scoping review

A literature search of information sources is a key methodological step for evidence synthesis, including scoping reviews. This study compares the performance of large language models with conventional systematic literature search methods in identifying published delirium clinical practice guidelines.

Comparative study using parallel searches. Four large language models (ChatGPT, Google Gemini, Claude and Grok) were queried using structured prompts, with results assessed for precision, sensitivity, and efficiency, versus a conventional systematic literature search using established bibliographic databases and grey literature sources.

Large language models sourced 17 of the 45 guidelines identified through a comparable conventional search, missing 28, but also identified two guidelines not found through the conventional search. The sensitivity of individual large language models ranged from 9% to 29%, using the conventional search as a reference standard. Of the 19 guidelines retrieved through large language models, 63% were deemed high quality, compared with 56% identified through the conventional search. Precision of individual large language models varied from 22% to 60%, far exceeding the precision of the conventional search (1.7%). Large language model-based searches required substantially less execution time compared to conventional methods.

While large language models have a significantly lower yield in retrieving clinical guidelines compared to the conventional search, their efficiency, along with their capacity to identify some unique articles, suggests a possible role in complementing conventional search methods in scoping reviews.