VetLLM: Large Language Model for Predicting Diagnosis from Veterinary Notes.

Area of Science:

• Veterinary Informatics
• Natural Language Processing
• Machine Learning

Background:

• Accurate diagnosis coding from veterinary notes is crucial for medical and public health research.
• Existing methods using rule-based or supervised learning models are often tedious and lack transferability.

Purpose of the Study:

• To evaluate the effectiveness of open-source large language models (LLMs) for automated diagnosis coding in veterinary medicine.
• To demonstrate the performance improvements achievable through fine-tuning LLMs on veterinary clinical text.

Main Methods:

• Assessed zero-shot performance of Alpaca-7B on veterinary coding benchmarks (CSU and PP).
• Developed and fine-tuned a specialized LLM, VetLLM, on a subset of veterinary notes.
• Compared VetLLM's performance against state-of-the-art supervised models.

Main Results:

• Alpaca-7B achieved zero-shot F1 scores of 0.538 (CSU) and 0.389 (PP).
• VetLLM, fine-tuned on 5000 notes, reached F1 scores of 0.747 (CSU) and 0.637 (PP), surpassing supervised models.
• Data-efficient fine-tuning demonstrated superior performance with only 200 notes compared to models trained on over 100,000 notes.

Conclusions:

• Open-source LLMs offer a viable and efficient approach for diagnosis coding from veterinary notes.
• Fine-tuning LLMs presents a powerful, data-efficient paradigm for clinical text processing in veterinary medicine.
• This approach holds significant potential for advancing medical and public health research using clinical data.