Séminaire de Probabilités et Statistique :

Le 10 mars 2025 à 13:45 - UM - Bât 09 - Salle de conférence (1er étage)

Présentée par Bensaid Bilel - Toulouse School of Economics

Neural Networks Optimizers from the point of view of numerical analysis: stability, acceleration and mini-batching.

These recent years, a great number of algorithms have been developed to optimize neural networks parameters (p-GD, clipping GD, Momentum, RMSProp, Adam, ...) but they need an accurate tuning to be stable and efficient. To get rid of the long and experimental step of GridSearch, we are looking for adaptive optimizers that come with guarantees. By analysing the stability of these algorithms, a general methodology to adapt the learning rate is suggested (generalization of the Armijo rule) for any deep learning optimizers, relating "robust" optimizers to preserving discretization schemes. Convergence and complexity of these methods are discussed leading to acceleration results, promoting the use of adaptive learning rate strategies for Analytic and Recurrent Neural Networks. Finally, this study is extended to the mini-batch setting, revealing the link between mini-batch optimization and splitting operator methods. In a nutshell, this work comes up with deep relations between neural network training and classical issues in the numerical analysis of differential equations.

Séminaire en salle 109, également retransmis sur zoom : https://umontpellier-fr.zoom.us/j/7156708132