PodGPT: an audio-augmented large language model for research and education

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computational Linguistics

Background:

• Scientific podcasts offer a rich source of specialized terminology and expert dialogues.
• Existing large language models (LLMs) can be enhanced by incorporating diverse educational content.
• Science, Technology, Engineering, Mathematics, and Medicine (STEMM) fields generate vast amounts of audio content.

Purpose of the Study:

• To develop a computational framework, PodGPT, to improve LLMs using STEMM podcast data.
• To enhance LLM understanding of natural language nuances, cultural contexts, and scientific/medical knowledge.
• To integrate retrieval augmented generation (RAG) for real-time access to scientific literature.

Main Methods:

• Transcribing over 3700 hours of STEMM podcast audio to generate 42 million text tokens.
• Developing PodGPT, a model that integrates podcast dialogue and RAG on a vector database.
• Evaluating PodGPT on multiple benchmarks for performance improvement.

Main Results:

• PodGPT demonstrated an average improvement of 1.82 percentage points over standard open-source benchmarks.
• Augmentation with RAG pipeline evidence led to an additional 2.43 percentage point improvement.
• An average improvement of 1.18 percentage points in zero-shot multilingual transfer ability was observed.

Conclusions:

• PodGPT effectively leverages STEMM podcast content to advance LLM capabilities.
• The model shows enhanced performance in understanding complex scientific dialogue and knowledge.
• PodGPT offers improved natural language processing and conversational AI for STEMM research and education.