PodGPT: An audio-augmented large language model for research and education

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Biomedical Informatics

Background:

• Scientific podcasts offer rich, specialized audio content across STEMM disciplines.
• Existing large language models (LLMs) can be enhanced by incorporating this diverse dialogue.
• Publicly accessible podcast data represents an underutilized resource for AI development.

Purpose of the Study:

• To introduce PodGPT, a computational framework leveraging STEMM podcast data to improve LLMs.
• To enhance LLMs' understanding of scientific terminology, cultural contexts, and natural language nuances.
• To augment LLMs with retrieval augmented generation (RAG) for real-time access to scientific literature.

Main Methods:

• Transcribed over 3,700 hours of STEMM podcast audio to generate 42 million text tokens.
• Developed PodGPT to integrate podcast dialogue and improve LLM comprehension.
• Implemented RAG using a vector database of Creative Commons PubMed Central and The New England Journal of Medicine articles.

Main Results:

• PodGPT showed an average improvement of 3.51 percentage points over standard benchmarks.
• Augmentation with RAG pipeline evidence improved performance by 3.81 percentage points.
• Demonstrated a 4.06 percentage point improvement in zero-shot multilingual transfer ability.

Conclusions:

• PodGPT effectively harnesses podcast content to advance LLMs in STEMM research and education.
• The framework enhances natural language processing and conversational AI capabilities.
• PodGPT offers improved access to scientific knowledge and literature through advanced AI.