ShaderNN: A Lightweight and Efficient Inference Engine for Real-time Applications on Mobile GPUs.

Area of Science:

• Computer Science
• Artificial Intelligence
• Mobile Computing

Background:

• Deep neural network inference on mobile devices faces challenges due to limited resources (computation, power, memory).
• Real-time applications require minimized data movement and increased data locality for efficient inference.
• Existing inference engines often involve costly data transfers between CPU and GPU.

Purpose of the Study:

• To propose Shader Neural Network (ShaderNN), a fast and power-efficient OpenGL-based inference framework for mobile devices.
• To address challenges of limited computational capacity, power budget, and data movement in mobile deep learning.
• To enable seamless integration with real-time graphics and image processing applications.

Main Methods:

• Developed ShaderNN using OpenGL, leveraging texture-based input/output for zero-copy integration.
• Utilized fragment shaders for neural network inference operators, particularly for smaller models.
• Implemented a hybrid compute and fragment shader approach with layer-level shader selection for performance optimization.
• Employed OpenGL features like normalization, interpolation, and texture padding to enhance performance.

Main Results:

• ShaderNN demonstrated superior performance compared to TensorFlow-Lite on mobile devices with Qualcomm and MediaTek chips.
• The framework achieved efficient, power-saving inference by minimizing data transfers between CPU and GPU.
• A case study confirmed ShaderNN's usability and seamless integration within an Android media processing application.

Conclusions:

• ShaderNN offers a novel and effective solution for on-device deep learning inference, optimizing for mobile constraints.
• The texture-based, OpenGL-centric approach provides significant performance and efficiency gains.
• ShaderNN is a viable and performant alternative for integrating deep learning into mobile applications.