Multiscale feature fusion for few-shot medical image learning with fisher information-driven layer selection

Few-shot medical image classification is a highly challenging problem in computer-aided diagnosis, with the central difficulty being enabling deep models to learn discriminative features conducive to classification from limited labeled samples. Vision transformers (ViTs) have recently demonstrated outstanding performance across various visual tasks. However, owing to their large parameter counts and dependence on massive pretraining data, ViTs are prone to overfitting in sample-scarce scenarios typical of few-shot learning. Parameter-efficient fine-tuning (PEFT) techniques, such as low-rank adaptation (LoRA), have alleviated some of these issues. However, conventional PEFT approaches still encounter difficulties in complex medical image classification tasks. To address this, this study proposes a general fine-tuning framework called a hierarchical probing and fusion network (HPF-Net), which integrates three core innovations to allow smarter and more efficient adaptation for few-shot medical image classification. First, a Fisher information-driven layer selection strategy strengthens the layer-selection robustness in few-shot settings. Subsequently, the attention-guided multiscale fusion module aligns and improves the features drawn from the selected critical layers. Subsequently, LoRA is incorporated into this efficient fine-tuning pipeline to reduce the parameter overhead while improving the accuracy. Extensive experiments on the public few-shot medical image benchmark, the medical imaging meta-dataset, demonstrated that HPF-Net significantly outperformed baseline methods, and ablation studies validated the necessity of each proposed component. The source code will be released upon acceptance.