SIFT-CNN: When Convolutional Neural Networks Meet Dense SIFT Descriptors for Image and Sequence Classification.

Area of Science:

• Computer Vision
• Machine Learning
• Image Processing

Background:

• Deep convolutional neural networks (CNNs) have largely replaced hand-crafted features but require extensive data.
• Hand-crafted features like Scale-Invariant Feature Transform (SIFT) offer valuable properties such as local rotation invariance.
• Existing CNNs often struggle with variations in local image orientation.

Purpose of the Study:

• To propose a novel hybrid approach, SIFT-CNN, integrating SIFT descriptors with CNNs.
• To enhance CNN performance by incorporating local rotation invariance into the input representation.
• To demonstrate the effectiveness of SIFT-CNN across diverse image classification tasks.

Main Methods:

• Compute SIFT descriptors for each pixel in a single-channel image, generating an M-channel SIFT image.
• Train a CNN to process these multiscale SIFT images, preserving spatial information.
• Utilize the inherent spatial relations within SIFT histograms to guide CNN feature learning from local gradients.

Main Results:

• The SIFT-CNN model demonstrated improved performance on indirect immunofluorescence cell classification, all-sky image-cloud classification, and human lip-reading tasks.
• SIFT-CNN surpassed traditional CNNs trained directly on pixel values in these challenging datasets.
• The proposed method showed robustness against local rotations, a key factor in performance improvement.

Conclusions:

• The input image representation significantly impacts the efficiency of data-driven systems.
• Combining SIFT descriptors with CNNs offers a powerful approach for image classification, particularly when local rotation invariance is crucial.
• SIFT-CNN provides a robust alternative to purely data-driven methods, especially in scenarios with rotational variations.