On Monday, 25 April 2022, our group had the pleasure of hosting Utku Evci (🐦utkuevci,🎓scholar), a Research Engineer in the Google Brain team in Montreal, to give an in-person invited talk about sparse neural networks. The title of his talk was “Beyond Static Network Architectures”. The presentation was not recorded, but a very similar inspiring talk, given by Utku at Mila - Quebec AI Institute, can be found here.
Abstract: Going beyond static architectures and using dynamically (1) trained, (2) executed or (3) adapted architectures has been shown to provide faster optimization, better scaling and more effective generalization. In this talk I will give a short overview of these results and share some of our recent work on dynamic training and adaptation of neural networks. On the dynamic training front, I plan to discuss our work on (a) training sparse neural networks and (b) growing neural networks, both of which use gradients as the guiding signal to update architectures during training. I will conclude with our recent work on (c) transfer learning, in which we propose to utilize a pretrained network head2toe by selecting features from all intermediate activations and show that this approach matches fine tuning performance.