Kernelized Sparse Fine-Tuning With Bi-Level Parameter Competition for Vision Models

Parameter-efficient fine-tuning (PEFT) aims to adapt pre-trained vision models to downstream tasks. Among PEFT paradigms, sparse tuning achieves remarkable performance by adjusting only the weights most relevant to downstream tasks, rather than densely tuning the entire weight matrix. Current sparse tuning methods follow a two-stage paradigm. First, it locates task-relevant weights by gradient information, which overlooks the parameter adjustments during fine-tuning and limits the performance. Second, it updates only the located weights by applying a sparse mask to the gradient of the weight matrix, which results in high memory usage due to the storage of all weight matrices in the optimizer. In this paper, we propose a one-stage method named SNELLA (Sparse tuning with kerNELized LoRA and Adaptive bi-level sparsity allocation) to overcome the above limitations. For memory usage, SNELLA selectively updates the weight matrix by adding it to another sparse matrix merged by two learnable low-rank matrices. We extend the low-rank decomposition by replacing the inner product with our proposed nonlinear kernel Mix-K, which largely increases the rank of the merged matrix while preserving training stability. The increase in rank prevents the interdependency among weight updates and enables better adaptation to downstream tasks. For locating task-relevant weights, we propose an adaptive bi-level sparsity allocation mechanism that encourages weights to compete across and inside layers based on their importance scores in an end-to-end manner under a predefined overall budget of weight updating, ensuring task-relevant weights to attain higher scores and be updated more likely. Extensive experiments are conducted on classification, segmentation, and generation tasks using pre-trained vision models with different parameter scales, architectures, and pre-training strategies. The results show that SNELLA achieves state-of-the-art performance with low memory usage. Notably, SNELLA obtains 1.8% (91.9% v.s. 90.1%) higher Top-1 accuracy on the FGVC benchmark compared to SPT-LoRA. Compared to previous sparse tuning methods, SNELLA achieves a memory reduction of 31.1%-39.9% across models with parameter scales from 86 M to 632 M.