Creating a biomedical knowledge base by addressing GPT inaccurate responses and benchmarking context.

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Medicine
• Gerontology and Neurodegenerative Diseases

Background:

• Generative AI models like GPT can produce inaccurate information ('hallucinations').
• Retrieval Augmented Generation (RAG) systems require robust evaluation methods.
• Aging, dementia, Alzheimer's disease, and diabetes research necessitates reliable knowledge bases.

Purpose of the Study:

• To develop and evaluate GNQA, a GPT-based knowledge base utilizing RAG for aging and dementia research.
• To implement a context provenance tracking mechanism to ensure response accuracy and traceability.
• To introduce RAGAS, an automated system for evaluating RAG performance, combining human and AI assessments.

Main Methods:

• A corpus of 3,000 peer-reviewed publications on aging, dementia, Alzheimer's, and diabetes was integrated into a RAG system.
• A context provenance tracking mechanism was developed to link responses to source material.
• The RAG Assessment System (RAGAS) was created, incorporating human expert and citizen scientist feedback alongside AI-driven evaluation.

Main Results:

• Human respondents approved GNQA responses 76% of the time.
• RAGAS achieved 90% answer relevance for expert-posed questions and 74% for GPT-generated questions.
• The study established RAGAS as a benchmark for continuous RAG system performance assessment.

Conclusions:

• GNQA, powered by RAG and enhanced with provenance tracking, offers a reliable and verifiable knowledge resource for aging and dementia research.
• RAGAS provides a robust framework for evaluating RAG systems, crucial for mitigating AI-generated inaccuracies.
• GNQA is accessible via the open-source GeneNetwork.org web service, promoting broader research accessibility.