Machine Learning in Montpellier, Theory & Practice
This talk is about Lipschitz constraints in the context of deep learning. Lipschitz constrained neural networks are of huge interest in the context of optimal transport (through Kantorovich-Rubinstein duality) and certifiability against adversarial attacks. First, I will explain how to implement these constraints in practice by performing a brief literature reviews of the dominant aproaches and methods. I will also present the DEEL-LIP library that contains some of these implementations in a Python package. Then, I will present some of my contributions to this field. I will show that these class of functions can be used to solve any classification task, with robustness guarantees, generalization guarantees (some of which being architecture-independant), and I will emphasize the importance of entropic regularization to chose an accuracy/robustness tradeoff on a Pareto front. After, I will present the explainability properties of these networks when they are trained with the Hinge-Kantorovich-Rubinstein (HKR) loss, inspired by optimal transport. Their saliency maps are more aligned with humans perception than any other network, including on challenging datasets like ImageNet. The HKR loss and Lipschitz neural networks can also be used to compute Signed Distance Function (SDF) in an unsupervised manner, relieving the need for a ground truth based on nearest neighbor. The resulting algorithm can be seen as a parametric K-nearest neighbor, or a robust one-class classifier. Furthermore, the SDF is also of huge interest in the computer graphics community since it allow for implicit surface parametrization with formal guarantees during raytracing. Finally, I will show how Lipschitz constraints w.r.t the input can be converted easily into Lipschitz constraints w.r.t the parameters, using a « backpropagation for bounds » algorithm, which opens path for deep learning with privacy guarantees without the need for the expensive gradient clipping operation of DP-SGD.