Equivariant High-Resolution Hyperspectral Imaging via Mosaiced and PAN Image Fusion

Existing mosaic-based snapshot hyperspectral imaging systems struggle to capture high resolution (HR) hyperspectral image (HSI), limiting its application. Fusing a low resolution (LR) mosaiced image with an HR panchromatic (PAN) image serves as a feasible solution to obtain the HR HSI. Therefore, we propose a dual-sensor based HSI imaging system, combining a $4\times 4$ spectral filter array (SFA) mosaiced image sensor with a co-aligned PAN image sensor to provide complementary spatial-spectral information. To reconstruct HR HSI, we propose an unsupervised equivariant imaging (EI)-based training framework with a learnable degradation function, overcoming the inaccessibility of ground truth and spectral response function (SRF). Specifically, we formulate the degradation process as a combination of $8\times 8$ mosaicing and $2\times 2$ average downsampling for the LR mosaiced image, while modeling the PAN image as a linear projection of the HR HSI using SRF. Since parameters of SRF are inaccessible, we propose to make them learnable to have an accurate estimation. By enforcing transformation equivariance between the input-output pair of the fusion network, the proposed framework ensures the reconstructed HSI preserves spatial-spectral consistency without relying on paired supervision. Furthermore, we instantiate the proposed HSI imaging system and collect a real-world dataset of 60 paired mosaiced / PAN images. The mosaiced image exhibits 16 spectral bands ranging from 722 to 896 nm and $1020\times 1104$ spatial pixels while the PAN image exhibits $2040\times 2208$ spatial pixels. Comprehensive experiments demonstrate that the proposed method exhibits high spatial consistency and spectral fidelity while maintaining computational efficiency.