Benchmarking AI chatbots: assessing their accuracy in identifying hijacked medical journals

Area of Science:

• Bibliometrics and Scholarly Communication
• Artificial Intelligence in Research Integrity

Background:

• Questionable journals, including hijacked ones, pose significant threats to academic integrity and the research community.
• Identifying hijacked journals is crucial for researchers to avoid predatory publishing practices.
• Artificial intelligence (AI) chatbots present a potential avenue for early detection of hijacked journals.

Purpose of the Study:

• To analyze and benchmark the performance of various AI chatbots in identifying hijacked medical journals.
• To assess the capability of AI chatbots in distinguishing between legitimate and hijacked journal titles and websites.

Main Methods:

• A dataset of 31 hijacked journals (21 previously identified, 10 newly detected) and their legitimate counterparts was compiled.
• Seven AI chatbots (ChatGPT, Gemini, Copilot, DeepSeek, Qwen, Perplexity, Claude) were benchmarked.
• Three question types were used to evaluate chatbots' ability to provide information, identify hijacked sites, and verify legitimate ones.

Main Results:

• AI chatbots demonstrated an ability to provide general information about hijacked journals.
• However, current AI chatbots were unable to reliably identify legitimate or hijacked journal titles.
• Copilot showed better performance compared to other chatbots but still exhibited errors.

Conclusions:

• Existing AI chatbots are not yet dependable tools for detecting hijacked journals.
• There is a risk that current AI chatbots may inadvertently promote hijacked journals.
• Further advancements in AI are required to enhance accuracy and reliability in identifying scholarly publishing threats.