VanillaNet: the Power of Minimalism in Deep Learning

Proposes VanillaNet: a simple, concise, and robust architecture with less sophisticated parts (contrary to ResNet that has residuals, ViT that has attention complexity, Swin Transformer that has custom CUDA modules). Simple 6 layer CNN (convolution neural network) architecture: a 4x4 conv (stride 4), then max-pooling with stride 2 (reduce height and width) and 1x1 conv/linear layer (double channels) for three modules/stages, then average pooling followed by FC layer (for classification settings). Deep training: combine activation functions with a residual-like alternative for non-linearity (linear weight - hyperparameter - increases linearly with epoch); merge batch norm and convolution layer, then 1x1 conv layers (merged weights and biases). Series Informed Activation function: Concurrently stack activation functions (with their own scale and bias) for each layer (increases non-linearity), can aggregate information from neighborhood; they have lesser computation complexity. Visualizes and contrasts (with ResNet) the attention learned (by VanillaNet) using GradCam++. Comparable (better than ViT) performance on ImageNet classification (SwinT comes close), but much lesser latency (high inference speed because of simple design). Higher FPS with slightly better AP on COCO detection and segmentation (with RetinaNet and Mask RCNN frameworks, compared to Swin-T backbone). Appendix has architecture and training details (LAMB optimizer). Code also has TensorRT ports. From Huawei, University of Sydney.

Links: arxiv, GitHub (Gitee)