Comprehensive Analysis of Neural Network Inference on Embedded Systems: Response Time, Calibration, and Model...

Area of Science:

• Computer Science
• Artificial Intelligence
• Embedded Systems Engineering

Background:

• Real-time inference speed of Artificial Neural Networks (ANNs) is critical for embedded systems processing sensor data at the edge.
• Applications like predictive maintenance require timely state change predictions, making ANN response time a key performance indicator.

Purpose of the Study:

• To enable estimation of ANN model response times on various embedded platforms.
• To analyze the impact of network parameters, activation functions, and threading on inference speed.
• To investigate the influence of network structure on model calibration and the effects of conversion to Tensorflow Lite (TFLite).

Main Methods:

• Benchmarking generic ANN applications on edge devices.
• Analysis of network parameters, activation functions, and single- versus multi-threading.
• Evaluation of hardware influences like clock rate variances.
• Quantification of model calibration using Expected Calibration Error (ECE).

Main Results:

• Response times are significantly influenced by network parameters, activation functions, and threading strategies.
• Task partitioning and scheduling complexity necessitate precise parameter coordination for cross-platform optimization.
• Cutting-edge frameworks may not automatically optimize all configurations, potentially impacting performance.
• Model conversion to TFLite requires careful consideration of both performance and calibration.

Conclusions:

• Optimizing ANN inference on embedded systems demands a holistic approach, considering hardware, software, and model-specific parameters.
• Accurate benchmarking and parameter coordination are vital for achieving desired performance and calibration levels.
• Developers must balance performance and calibration when deploying ANNs on resource-constrained edge devices.