Related Experiment Video
Updated: Jun 3, 2025
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Performance Assessment of Large Language Models in Medical Consultation: Comparative Study
Sujeong Seo1,2, Kyuli Kim1, Heyoung Yang1
1Future Technology Analysis Center, Korea Institute of Science and Technology Information, Seoul, Republic of Korea.
Background:
The recent introduction of generative artificial intelligence (AI) as an interactive consultant has sparked interest in evaluating its applicability in medical discussions and consultations, particularly within the domain of depression.
Objective:
This study evaluates the capability of large language models (LLMs) in AI to generate responses to depression-related queries.
Methods:
Using the PubMedQA and QuoraQA data sets, we compared various LLMs, including BioGPT, PMC-LLaMA, GPT-3.5, and Llama2, and measured the similarity between the generated and original answers.
Results:
The latest general LLMs, GPT-3.5 and Llama2, exhibited superior performance, particularly in generating responses to medical inquiries from the PubMedQA data set.
Conclusions:
Considering the rapid advancements in LLM development in recent years, it is hypothesized that version upgrades of general LLMs offer greater potential for enhancing their ability to generate "knowledge text" in the biomedical domain compared with fine-tuning for the biomedical field. These findings are expected to contribute significantly to the evolution of AI-based medical counseling systems.
