Room 109, IMAG, Triolet campus

Machine Learning in Montpellier, Theory & Practice

We consider a gradient-based optimization method applied to a function [image: \mathcal{L}] of a vector of variables [image: {\theta}], in the case where [image: {\theta}] is represented as a tuple of tensors [image: (\mathbf{T}_1, \cdots, \mathbf{T}_S)]. This framework encompasses many common use-cases, such as training neural networks by gradient descent. First, we propose a computationally inexpensive technique providing higher-order information on [image: \mathcal{L}], especially about the interactions between the tensors [image: \mathbf{T}_s], based on automatic differentiation and computational tricks. Second, we use this technique at order 2 to build a second-order optimization method which is suitable, among other things, for training deep neural networks of various architectures. This second-order method leverages the partition structure of [image: \theta] into tensors [image: (\mathbf{T}_1, \cdots, \mathbf{T}_S)], in such a way that it requires neither the computation of the Hessian of [image: \mathcal{L}] according to [image: \theta], nor any approximation of it. The key part consists in computing a smaller matrix interpretable as a « Hessian according to the partition’’, which can be computed exactly and efficiently. In contrast to many existing practical second-order methods used in neural networks, which performs a diagonal or block-diagonal approximation of the Hessian or its inverse, the method we propose does not neglect interactions between layers. GitHub: \url{https://isdm.umontpellier.fr/github.com/p-wol/GroupedNewton%7D. arXiv https://isdm.umontpellier.fr/arxiv.org/abs/2312.03885

Online Code Code Visio Code Online Online Online Online Online