Parameter-free Machine Learning through Coin Betting

08 Mar
Thursday, 03/08/2018 4:00pm to 5:00pm
Computer Science Building, Room 151

Abstract: Machine Learning (ML) has been described as the fuel of the next industrial revolution. Yet, despite their name, the majority of the ML algorithms still heavily rely on having humans in the loop in order to set their "parameters". For example, when using a regularizer, the regularization weight is critical to obtain theoretical and practical optimal performance. Moreover, the minimization itself, usually done through stochastic gradient descent procedures, requires to set "learning rates" in order to achieve good performance. Are these parameters strictly necessary? Is it possible to have parameter-free ML algorithms?

In this talk, Francesco will show that both the problems of ML and stochastic optimization with quasi-convex losses can be reduced to a game of betting on a non stochastic coin. Betting on a non-stochastic coin is a well-known problem that can be solved using tools from information theory. Moreover, the optimal coin betting algorithm is parameter-free, giving rise to parameter-free ML and stochastic optimization algorithms.

This approach is very general, i.e. it works for any norm, and it gives optimal results in a number of settings, i.e. reproducing kernel Hilbert spaces, without any parameter to tune. Beside the theoretical results, we will show that this approach can also be used in modern deep learning architectures, showing how to achieve state-of-the-art performance using the first stochastic gradient procedure without a learning rate.

Bio: Francesco Orabona is an Assistant Professor at Stony Brook University. His background covers both theoretical and practical aspects of machine learning and optimization. His current research interests lie in online learning, and more generally the problem of designing and analyzing adaptive and parameter-free learning algorithms. He received the PhD degree in Electrical Engineering at the University of Genoa, in 2007. He is (co)author of more than 60 peer reviewed papers.

A reception for attendees will be held at 3:30 p.m. in CS 150

Faculty Host