PreLora: A fine-tuning approach with low-rank matrix decomposition and prefix tuning for pre-hospital emergency text...

With expanding applications of artificial intelligence technology in the medical field, Large Language Models (LLMs) have achieved substantial success in medical text processing. However, there remain a number of challenges in effectively adapting to specific tasks, such as pre-hospital emergency text classification.

We propose a novel fine-tuning method PreLora, which combines prefix tuning with matrix low-rank decomposition. First, this approach incorporates task-specific prompts based on multi-layer perceptron (MLP) encoder into the input. Then, it inject trainable rank-decomposed matrices into every layer of the transformer architecture to compress model parameters, reduce the number of parameters, and capture correlations among the input. To validate its efficacy, we carried out a comparative validation on a pre-hospital emergency text dataset.

Comparison results indicated that the model fine-tuned with PreLora outperformed the baseline models without fine-tuning, achieving a performance improvement of 45.4%-75.4%. Moreover, PreLora ranked first among all fine-tuning methods across each LLM evaluated. An in-depth performance analysis was further conducted on 21 ICD-10 categories with distinct semantic features. The results revealed a negative correlation between model performance and semantic similarity of ICD-10 categories: the low similarity groups performed better, while the high similarity groups performed worse. Notably, PreLora consistently maintained robust performance, with a smaller performance decline in high-similarity categories compared to other fine-tuning methods. In the classifying complex cases with high semantic similarity, PreLora still showed superior adaptability, improving by 68.6%-95.8% compared to the baseline model and 0.4%-8.4% compared to other fine-tuning methods.

This study demonstrates PreLora is an effective fine-tuning method to process pre-hospital emergency text classification. It has the potential to expand to other mainstream models for adapting specific tasks in the medical field.