Date: 17 January 2024
Time: 12.45 Hours
Room: RA1501 & online
Speaker: Prof.dr. Mark (A.) Peletier
Title: “Singular-limit analysis of training with noise injection"
Abstract:
Many training algorithms inject some form of noise in the training. The classical example is the mini-batch noise in Stochastic Gradient Descent, but other examples are dropout, data augmentation, 'noise nodes', 'label noise', and input-data noise.
While the additional noise is generally believed to improve generalisation performance, there is little mathematical understanding of how this is achieved. In this talk I will describe recent work, together with Anna Shalova (TU/e) and André Schlichting (Münster), in which we analyse a fairly general class of iterative training schemes with noise injection. In the limit of small noise, we prove convergence of the appropriately rescaled time courses to solutions of an auxiliary evolution equation. This auxiliary equation is a gradient flow driven by a functional for which we obtain an explicit expression, thus opening the door to understanding the different types of regularisation generated by different types of noise injection.