Efficient and accurate compound scaling for convolutional neural networks

Designing efficient and accurate network architectures to support various workloads, from servers to edge devices, is a fundamental problem as the use of Convolutional Neural Networks (ConvNets) becomes increasingly widespread. One simple yet effective method is to scale ConvNets by systematically adjusting the dimensions of the baseline network, including width, depth, and resolution, enabling it to adapt to diverse workloads by varying its computational complexity and representation ability. However, current state-of-the-art (SOTA) scaling methods for neural network architectures overlook the inter-dimensional relationships within the network and the impact of scaling on inference speed, resulting in suboptimal trade-offs between accuracy and inference speed. To overcome those limitations, we propose a scaling method for ConvNets that utilizes dimension relationship and runtime proxy constraints to improve accuracy and inference speed. Specifically, our research notes that higher input resolutions in convolutional layers lead to redundant filters (convolutional width) due to increased similarity between information in different positions, suggesting a potential benefit in reducing filters while increasing input resolution. Based on this observation, the relationship between the width and resolution is empirically quantified in our work, enabling models with higher parametric efficiency to be prioritized through our scaling strategy. Furthermore, we introduce a novel runtime prediction model that focuses on fine-grained layer tasks with different computational properties for more accurate identification of efficient network configurations. Comprehensive experiments show that our method outperforms prior works in creating a set of models with a trade-off between accuracy and inference speed on the ImageNet datasets for various ConvNets.