Retrieval Augmented Generation (RAG) for Natural Language Querying of Immunogenicity Data for Protein Drugs

Area of Science:

• Biotechnology
• Artificial Intelligence
• Immunology

Background:

• Biologics immunogenicity is critical for drug safety and efficacy.
• Natural language querying of complex biologics data presents challenges.
• Retrieval-augmented generative (RAG) AI offers a potential solution for data synthesis.

Purpose of the Study:

• To evaluate the strengths and limitations of RAG AI for querying biologics immunogenicity data.
• To assess the performance of different large language models (LLMs) within a RAG system.
• To identify areas for improvement in RAG systems for scientific literature analysis.

Main Methods:

• Package inserts for 663 biologics were retrieved from DailyMed.
• A RAG system integrated NLP, retrieval, and LLM components (ChatGPT, Gemini, DeepSeek, Llama).
• LLMs were queried on factors influencing anti-drug antibody (ADA) incidence and tolerability.

Main Results:

• The RAG system retrieved relevant contexts but noted inaccuracies for non-antibody protein drugs.
• All LLMs identified key determinants of ADA incidence and tolerability factors.
• LLM outputs varied in detail and comprehensiveness, but generally showed relevance and faithfulness to source text.
• Domain-specific evaluation confirmed accurate trend identification and knowledge gap highlighting.

Conclusions:

• RAG-based systems can retrieve and synthesize immunogenicity data from multiple sources.
• Limitations exist, particularly in the retriever's effectiveness and accuracy for specific drug types.
• Further refinement of RAG systems is warranted for reliable scientific data querying.