Verifying Outputs in Scopus AI to Promote Critical AI Literacy in Education and Research

Area of Science:

• Health Professions Education
• Artificial Intelligence in Research

Background:

• Generative artificial intelligence (AI) tools are increasingly used in academic research.
• Evaluating the accuracy and reliability of AI-generated content is crucial for maintaining scholarly standards.

Purpose of the Study:

• To assess the factual accuracy and citation fidelity of Scopus AI outputs.
• To examine the reliability of AI-generated responses to a health care research question.

Main Methods:

• A mixed-methods content verification approach was used.
• Scopus AI generated a summary, expanded summary, and concept map in response to a health care research question.
• Statements were quantitatively classified as accurate, misleading, or incorrect, with qualitative analysis of citations and sources.

Main Results:

• 90% of statements in the Scopus AI output were accurate; 10% were misleading.
• No incorrect or hallucinated content was found.
• Scopus AI cited legitimate sources, but occasionally referenced secondary sources without clarification.

Conclusions:

• Scopus AI provides generally reliable academic content, but user verification is necessary.
• The study highlights the need for critical evaluation of AI-generated material and improved citation transparency.
• A classroom activity, "Fact-Check the Bot," was developed to enhance AI literacy and critical assessment skills.