Evaluation of the Construction and Accuracy of a Local LLM Based on Clinical Engineering

This study aimed to develop a local large language model (LLM) specialized in clinical engineering and to evaluate its performance on the Japanese National Clinical Engineer Licensing Examination. Three local LLMs (Cogito-32b-think, GPT-4o-mini, and Qwen3-14B) and Gemini-2.5 Pro were tested. Among the local models, Qwen3-14B achieved the highest accuracy and was further improved through supervised fine-tuning (SFT) using reasoning data generated by Gemini-2.5 Pro. After SFT, the overall accuracy of Qwen3-14B increased from 77% to 82%, with improvements in image (39% to 52%) and calculation (58% to 68%) questions. Although Gemini-2.5 Pro outperformed Qwen3-14B with 94% accuracy, the results demonstrate that SFT effectively enhances local LLMs, offering a promising, secure, and cost-efficient approach for specialized clinical engineering applications.