Unified interest point detection and description for perspective and Fisheye images

Keypoint detection and descriptor matching are essential in tasks like feature matching, object tracking, and 3D reconstruction. While CNN-based methods have advanced these areas, most focus on perspective projection cameras, with limited consideration of fisheye cameras, which introduce significant distortion. Conventional keypoint methods have limitations on fisheye images, causing camera models to underperform in hybrid camera systems. This paper proposes a robust keypoint detection and description method under a hybrid camera model, addressing challenges in mixed camera systems. Since fisheye datasets are scarce, we used viewpoint and projection transformations to augment our training data. We propose a method for generating fisheye data, and inspired by SuperPoint, we modify the network architecture for feature extraction and descriptor generation. By employing nearest neighbor (NN) matching, our proposed dataset generation method improves the performance of the original SuperPoint network, while the network architecture introduced in this paper further improves overall performance.