Optimal Output Gain Algorithm For Feed-forward Network Training

Abstract

A batch training algorithm for feed-forward networks is proposed which uses Newton's method to estimate a vector of optimal scaling factors for output errors in the network. Using this vector, backpropagation is used to modify weights feeding into the hidden units. Linear equations are then solved for the network's output weights. Elements of the new method's Gauss-Newton Hessian matrix are shown to be weighted sums of elements from the total network's Hessian. The effect of output transformation on training a feed-forward network is reviewed and explained, using the concept of equivalent networks. In several examples, the new method performs better than backpropagation and conjugate gradient, with similar numbers of required multiplies. The method performs about as well as Levenberg-Marquardt, with several orders of magnitude fewer multiplies due to the small size of its Hessian.