Deep learning under Lipschitz constraints
21 mars 2024 @ 14 h 00 min –
Room 03.124, Building 5, St Priest campus
Machine Learning in Montpellier, Theory & Practice
This talk is about Lipschitz constraints in the context of deep learning. Lipschitz constrained neural networks are of huge interest in the context of optimal transport (through Kantorovich-Rubinstein duality) and certifiability against adversarial attacks. First, I will explain how to implement these constraints in practice by performing a brief literature reviews of the dominant aproaches and methods. I will also present the DEEL-LIP library that contains some of these implementations in a Python package. Then, I will present some of my contributions to this field. I will show that these class of functions can be used to solve any classification task, with robustness guarantees, generalization guarantees (some of which being architecture-independant), and I will emphasize the importance of entropic regularization to chose an accuracy/robustness tradeoff on a Pareto front. After, I will present the explainability properties of these networks when they are trained with the Hinge-Kantorovich-Rubinstein (HKR) loss, inspired by optimal transport. Their saliency maps are more aligned with humans perception than any other network, including on challenging datasets like ImageNet. The HKR loss and Lipschitz neural networks can also be used to compute Signed Distance Function (SDF) in an unsupervised manner, relieving the need for a ground truth based on nearest neighbor. The resulting algorithm can be seen as a parametric K-nearest neighbor, or a robust one-class classifier. Furthermore, the SDF is also of huge interest in the computer graphics community since it allow for implicit surface parametrization with formal guarantees during raytracing. Finally, I will show how Lipschitz constraints w.r.t the input can be converted easily into Lipschitz constraints w.r.t the parameters, using a « backpropagation for bounds » algorithm, which opens path for deep learning with privacy guarantees without the need for the expensive gradient clipping operation of DP-SGD.
Machine Learning in Montpellier, Theory & Practice
Adapting Newton’s Method to Neural Networks through a Summary of Higher-Order Derivatives
21 mars 2024 @ 14 h 00 min –
Room 109, IMAG, Triolet campus
Machine Learning in Montpellier, Theory & Practice
We consider a gradient-based optimization method applied to a function [image: \mathcal{L}] of a vector of variables [image: {\theta}], in the case where [image: {\theta}] is represented as a tuple of tensors [image: (\mathbf{T}_1, \cdots, \mathbf{T}_S)]. This framework encompasses many common use-cases, such as training neural networks by gradient descent. First, we propose a computationally inexpensive technique providing higher-order information on [image: \mathcal{L}], especially about the interactions between the tensors [image: \mathbf{T}_s], based on automatic differentiation and computational tricks. Second, we use this technique at order 2 to build a second-order optimization method which is suitable, among other things, for training deep neural networks of various architectures. This second-order method leverages the partition structure of [image: \theta] into tensors [image: (\mathbf{T}_1, \cdots, \mathbf{T}_S)], in such a way that it requires neither the computation of the Hessian of [image: \mathcal{L}] according to [image: \theta], nor any approximation of it. The key part consists in computing a smaller matrix interpretable as a « Hessian according to the partition’’, which can be computed exactly and efficiently. In contrast to many existing practical second-order methods used in neural networks, which performs a diagonal or block-diagonal approximation of the Hessian or its inverse, the method we propose does not neglect interactions between layers. GitHub: \url{https://isdm.umontpellier.fr/github.com/p-wol/GroupedNewton%7D. arXiv https://isdm.umontpellier.fr/arxiv.org/abs/2312.03885
Machine Learning in Montpellier, Theory & Practice