Efficient Visual Computing With Camera RAW Snapshots.

Area of Science:

• Computer Vision
• Image Processing
• Machine Learning

Background:

• Conventional cameras convert RAW sensor data to RGB images via an image signal processor (ISP).
• This ISP conversion is often unnecessary for visual computing tasks, as RAW data contains complete information.
• Existing RAW image datasets are scarce, hindering direct training of models on RAW data.

Purpose of the Study:

• To propose the novel ρ-Vision framework for high-level semantic understanding and low-level compression directly from RAW images.
• To eliminate the reliance on the traditional image signal processor (ISP) subsystem.
• To address the challenge of limited RAW image datasets.

Main Methods:

• Developed an unpaired CycleR2R network using unsupervised CycleGAN to train modular unrolled ISP and inverse ISP (invISP) models.
• Generated simulated RAW images (simRAW) from existing RGB datasets for flexible model training.
• Fine-tuned RGB-domain models to process real-world camera RAW images using the ρ-Vision framework.
• Demonstrated object detection (RAW-domain YOLOv3) and image compression (RIC) on camera snapshots.

Main Results:

• RAW-domain task inference demonstrated superior object detection accuracy and image compression efficiency compared to RGB-domain methods.
• The ρ-Vision framework exhibited generalization across diverse camera sensors and task-specific models.
• Eliminating the ISP subsystem led to potential reductions in computations and processing times.

Conclusions:

• The ρ-Vision framework enables effective direct processing of RAW camera images for computer vision tasks.
• This approach offers improved performance and efficiency by bypassing the conventional ISP.
• The method provides a flexible and generalizable solution for RAW image analysis, applicable to various sensors and models.