Cours/Séminaire
Notice
Lieu de réalisation
Centre Inria d'Université Côte d'Azur
Langue :
Anglais
Crédits
Iñaki Esnaola (Intervention)
Conditions d'utilisation
Droit commun de la propriété intellectuelle
DOI : 10.60527/3q3b-vh95
Citer cette ressource :
Iñaki Esnaola. DALI. (2026, 25 février). Machine Learning Regularization via Information Measures. [Vidéo]. Canal-U. https://doi.org/10.60527/3q3b-vh95. (Consultée le 9 novembre 2026)
logo DALI
Decisions, Algorithms, Learning, and Information
S'abonner
Contacter

Machine Learning Regularization via Information Measures

Réalisation : 25 février 2026 - Mise en ligne : 19 mars 2026
  • document 1 document 2 document 3
  • niveau 1 niveau 2 niveau 3
Descriptif

The effect of relative entropy asymmetry is analyzed in the context of empirical risk minimization (ERM) with relative entropy regularization (ERM-RER). We show that regularization by relative entropy forces the support of the solution to collapse into the support of the reference measure, introducing a strong inductive bias that negates the evidence provided by the training data. Motivated by this insight, we extend the results to f-divergence regularization by obtaining a closed form expression for the solution under mild assumptions on the structure of the regularizer. The dual formulation of empirical risk minimization with f-divergence regularization (ERM-fDR) is introduced. The solution of the dual optimization problem to the ERM-fDR is connected to the notion of normalization function introduced as an implicit function. This dual approach leverages the Legendre-Fenchel transform and the implicit function theorem to provide an ODE expression for the normalization function. Furthermore, the ODE expression and its properties provide a computationally efficient method to calculate the normalization function of the ERM-fDR solution under a mild condition.

Intervention / Responsable scientifique
Thème
Disciplines :
Documentation

Sur le même thème