Multistep Second Order Training For The Multilayer Perceptron

Abstract

Training a feedforward multilayer perceptron (MLP) requires obtaining train- ing data and solving a non-convex optimization problem to calculate the network's weights. Various problems can arise during training that ultimately can limit a MLP's usefulness such as slow convergence and high computational complexity. Additionally, when training one needs to have confidence that the chosen algorithm is working optimally for the chosen coordinate system.We introduce novel second order training algorithms to overcome these difficulties. In the process, a piecewise affine model of the multilayer perceptron is introduced which shows that objective functions for training are poorly modeled by quadratic functions of network weights. One step and multistep second order training algorithms are derived which avoid the problems implied by the model.The new second order algorithms are shown to have a form of affine invariance which ensures that they are optimal in the sense that they cannot be improved by affine transformation.In simulation, their training and validation performance is comparable to Levenberg- Marquardt, yet they have the advantage of reduced computational complexity.