VisualRNet: Lightweight Camera Rotation Estimation from Low-Resolution Optical Flow via Cross-Modal Supervision

Area of Science:

• Computer Vision
• Robotics
• Machine Learning

Background:

• Camera rotation estimation is crucial for video stabilization and motion analysis.
• Classical methods struggle with challenging conditions like blur and low illumination, and inertial sensors can be unreliable.
• Existing approaches often require high-resolution data or complex pipelines.

Purpose of the Study:

• To investigate the efficacy of substantially downsampled optical flow for accurate frame-to-frame rotation regression.
• To introduce VisualRNet, a lightweight, rotation-specific visual regression framework.
• To demonstrate the potential of low-resolution optical flow in practical, cost-sensitive applications.

Main Methods:

• Developed VisualRNet, a lightweight framework utilizing coordinate-aware feature encoding, depthwise separable convolutions, and lightweight attention.
• Employed a compact 6D rotation head for modeling rotational flow fields.
• Trained the network using cross-modal Inertial Measurement Unit (IMU) supervision.

Main Results:

• VisualRNet achieved a mean rotation error of 0.3151° on the Deep-FVS test set.
• The VisualRNet regression head is highly efficient (7.7K parameters, 0.002 GFLOPs, 729 FPS).
• The full pipeline runs at ~113 FPS, demonstrating real-time capabilities.
• Cross-camera adaptation on TUM VI showed successful alignment with limited calibration data.

Conclusions:

• Substantially downsampled optical flow retains sufficient structure for accurate visual rotation estimation.
• VisualRNet offers a practical and efficient solution for rotation estimation, especially in stabilization-oriented and cost-sensitive applications.
• The learned motion representation is adaptable to new camera systems, broadening its applicability.