Poster
in
Workshop: Mathematics of Modern Machine Learning (M3L)
Improving the Gaussian Approximation in Neural Networks: Para-Gaussians and Edgeworth Expansions
Mihai Nica · Janosch Ortmann
Keywords: [ infinite neural networks ] [ wide networks ] [ Gaussian processes ] [ neural scaling laws ]
Gaussian approximations are often used for developing the theory of how neural networks scale as the number of neurons grows large. However, it is known that these approximations break down as depth increases due to the accumulation of approximation errors. To remedy this, we provide a new family of distributions that appear naturally in neural networks and provide more accurate approximations than the usual Gaussian approximation. We develop a method for obtaining the probability density function via Hermite polynomials and connect this to the classical Edgeworth expansion.