Two-Layer Retrieval-Augmented Generation Framework for Low-Resource Medical Question Answering Using Reddit Data:...

Area of Science:

• Natural Language Processing
• Medical Informatics
• Computational Linguistics

Background:

• Social media provides valuable data on substance use, including side effects and patterns of novel psychoactive substances.
• Analyzing this large volume of user-generated content for medical insights is challenging with traditional methods.
• Large language models (LLMs) offer potential but require efficient architectures for medical question answering.

Purpose of the Study:

• To develop a retrieval-augmented generation (RAG) architecture for medical question answering.
• To utilize user-generated social media data for clinicians' queries on emerging health topics.
• To create a system capable of extracting and summarizing relevant information from vast online discussions.

Main Methods:

• A two-layer RAG framework was proposed for query-focused answer generation.
• The framework was evaluated using a proof-of-concept for drug-related information from social media forums.
• User-generated data from Reddit concerning xylazine and ketamine use was analyzed to answer clinician queries, comparing a quantized LLM (Nous-Hermes-2-7B-DPO) with GPT-4.

Main Results:

• The RAG framework demonstrated comparable performance to GPT-4 in relevance, length, hallucination, coverage, and coherence.
• No statistically significant differences were found between GPT-4 and Nous-Hermes-2-7B-DPO for most evaluated metrics.
• A statistically significant difference was observed in the Coleman-Liau Index, indicating variations in readability.

Conclusions:

• The developed RAG framework effectively answers targeted medical questions using social media data.
• The architecture is suitable for deployment in resource-constrained environments.
• This approach offers an efficient method for extracting critical health information from large-scale online user-generated content.