Nonlinear Phenomena in Complex Systems, 2008, Vol.11, No.4, pp.488-492

NONLINEAR PHENOMENA IN COMPLEX SYSTEMS
An Interdisciplinary Journal

2008, Vol.11, No.4, pp.488-492

Stochastic Dynamics of Nonlinear Optimization by Gradient Descent and the Learning of Neural Networks.
A. D. Linkevich

Minimization of nonlinear functions is treated as a dynamical process in the presence of noise. Path integration approach is applied to formulate a diagrammatic perturbation theory for calculation of correlation and response functions. The steepest descent method can be used to evaluate the path integrals approximately as well. As a specific area of application, the learning of neural networks is considered. Two exactly solvable examples are given.
Key words: nonlinear optimization, stochastic dynamics, gradient descent, neural network, learning, path integral

Full text: Acrobat PDF (218KB)