ADAPTIVE ACTIVATIONS AND SHIFT INVARIANCE IN SHALLOW CONVOLUTIONAL NEURAL NETWORKS
View/ Open
Date
2021-08-13Author
Rane, Chinmay Appa
0000-0002-1373-819X
Metadata
Show full item recordAbstract
Deep learning training training algorithms are a huge success in recent years in many fields including speech, text,image video etc. Deeper and deeper layers are proposed with huge success with resnet structures having around 152 layers. Shallow convolution neural networks(CNN's) are still an active research, where some phenomena are still unexplanined. CNN's are assumed to be invariant to shift due to its architecture, but recent studies have shown other wise. Apart from shift invariance, activation functions used in the network are of utmost importance, as they provide non linearity to the networks. Relu's are the most commonly used activation function.
We show a shallow network which is specifically used for classifying images with shifted objects. Completed Tasks are shown for analyzing and improving shallow networks shift invariance in convolutional neural networks. We demonstrate commonly used downsampling technique and show if these downsampling techniques work for shallow CNN's. We also show a way to factorize the output weights in the feature layer. A traditional segmentation example is shown for the shifted objects and subsequent results are also given.
We also show a complex piece-wise linear(PWL) activation in the hidden layer. We show that these PWL activations work much better than relu activations in our networks for convolution neural networks and multilayer perceptrons. Result comparison in MATALB and PYTORCH for shallow and deep CNNs are given to further strengthen our case. A naive growing and pruning algorithm for these PWL activation is shown and compared with original results.