A pretrained biomedical large language model for Persian biomedical text mining.

Area of Science:

• Bioinformatics
• Natural Language Processing
• Artificial Intelligence

Background:

• Large language models (LLMs) are increasingly utilized in life sciences for complex data analysis.
• Their application in bioinformatics, particularly for medical question answering (QA), is an emerging area.
• Existing LLMs show limitations in handling nuanced, real-world bioinformatics queries.

Purpose of the Study:

• To introduce BioPars, a novel benchmark and QA dataset for evaluating LLMs in Persian medical contexts.
• To assess the knowledge acquisition, synthesis, and evidence-based reasoning abilities of LLMs.
• To compare the performance of prominent LLMs like ChatGPT, Llama, and Galactica on specialized medical QA tasks.

Main Methods:

• Development of the BIOPARS-BENCH dataset from diverse scientific and medical sources.
• Creation of the BioParsQA dataset with 5231 Persian medical questions and answers.
• Evaluation of LLMs using BioPars, focusing on knowledge retrieval, interpretation, and evidence demonstration.
• Comparative analysis using metrics such as ROUGE-L, BERTScore, MoverScore, and BLEURT.

Main Results:

• LLMs demonstrate strong knowledge recall but struggle with higher-level reasoning and fine-grained inferences in medical QA.
• The BioPars model achieved superior performance on BioParsQA, outperforming GPT-4 with a ROUGE-L score of 29.99.
• The model also achieved high scores in BERTScore (90.87), MoverScore (60.43), and BLEURT (50.78), indicating robust performance in Persian medical QA.

Conclusions:

• BioPars represents a significant contribution to evaluating LLMs in Persian medical QA, particularly for long-form answers.
• Further fine-tuning of LLMs is crucial to enhance their capabilities for complex bioinformatics tasks.
• The developed benchmark and model show potential for advancing AI applications in specialized medical domains.