On-sensor binarized CNN inference with dynamic model swapping in pixel processor arrays.

Area of Science:

• Computer Vision
• Machine Learning
• Embedded Systems Engineering

Background:

• Convolutional Neural Networks (CNNs) are computationally intensive, posing challenges for resource-limited embedded and edge devices.
• Pixel Processor Arrays (PPAs) offer a promising camera-sensor architecture for edge computing but require efficient model deployment strategies.
• Balancing model generality with computational efficiency is crucial for embedded CNN applications.

Purpose of the Study:

• To develop novel methods for training and deploying efficient Convolutional Neural Networks (CNNs) on embedded devices, specifically Pixel Processor Arrays (PPAs).
• To address the limitations of computation and memory in edge devices by optimizing CNN architectures and deployment.
• To demonstrate the effectiveness of proposed methods on various computer vision tasks using a resource-constrained PPA sensor-processor.

Main Methods:

• Training purely binarized CNNs incorporating batch normalization and adaptive thresholding for binary activations.
• Converting batch normalization and binary activations into a bias matrix for parallel add/sub operations, optimizing for PPAs.
• Implementing a dynamic model swapping paradigm by decomposing large applications into sub-tasks solvable by dynamically loadable tree networks.

Main Results:

• Successfully trained and implemented binarized CNNs with optimized batch normalization and activation handling suitable for PPA hardware.
• Demonstrated the feasibility of dynamic model swapping for deploying complex applications exceeding PPA capacity.
• Achieved effective performance on classification, localization, and coarse segmentation tasks on a highly resource-constrained PPA sensor-processor.

Conclusions:

• The proposed methods provide efficient solutions for deploying CNNs on resource-constrained embedded systems, particularly PPAs.
• Binarized CNNs with optimized components and dynamic model swapping are viable strategies for edge AI applications.
• These advancements enable sophisticated computer vision capabilities on low-power, memory-limited PPA sensor-processors.