PharmAI: domain-specific QA generation by adaptive learning and fine-tuning of small language models

Domain-specific medical question-answering (QA) models face bottlenecks due to scarce labeled data, specialized terminology, and constrained computational hardware.

The study aims to develop an efficient pharmaceutical question-answering (QA) system that can be trained and deployed on consumer-grade hardware using Small Language Models (SLMs).

The pipeline processes four pharmaceutical textbooks containing 3,977 total pages, producing 7,370 semantically similar text chunks and 2,144 synthetic QA pairs under four questioning categories: definition, mechanism, comparison, and clinical application. Three Small Language Models (SLMs), namely SmolLM2-1.7B-Instruct, Qwen2.5-1.5B-Instruct, and Llama-3.2-1B-Instruct, are fine-tuned under the same QLoRA (Quantized Low-Rank Adaptation) configurations and then evaluated with retrieval metrics (Recall@K, Mean Reciprocal Rank) and generation metrics (ROUGE-1/2/L, Semantic Similarity, BERTScore). Architectural decisions including NF4 4-bit quantization, CPU-based FAISS indexing, and completion-only loss masking are implemented due to hardware constraints and validated via ablation analysis.

SmolLM2-1.7B-Instruct, Qwen2.5-1.5B-Instruct, and Llama-3.2-1B-Instruct demonstrated identical retrieval performance, achieving a Recall@1 of 0.517, Recall@3 of 0.732, Recall@5 of 0.797, and an MRR of 0.628. Similarly, the generation performance evaluation revealed that Qwen2.5-1.5B FT outperformed the other fine-tuned SLMs, achieving the highest scores across all metrics (ROUGE-1 = 0.47, ROUGE-2 = 0.25, ROUGE-L = 0.38, and BERTScore = 0.72). In comparison, SmolLM2-1.7B FT and Llama-3.2-1B FT exhibited comparable performance, with ROUGE-1 scores of approximately 0.35, ROUGE-2 scores of 0.16-0.17, ROUGE-L scores of 0.25-0.26, and BERTScores of 0.66, indicating lower lexical and semantic alignment with the reference outputs than Qwen2.5-1.5B FT.

We evaluate retrieval and generation performance using comprehensive metric suites, incorporating a category-wise breakdown that reveals comparison questions to be the most challenging for single-passage retrieval architectures.