Abstract: Reinforcement learning (RL) is a promising approach for learning decision-making policies and more broadly, training generative models to be controllable and goal-oriented. Nonetheless, it has been challenging to use reinforcement learning approaches with modern day function approximators such as transformers, which often consist of a huge number of parameters, even in scenarios where we start off with good base models and policies (e.g., strong large language models). Why is this the case? In this talk, I would argue that this is due to algorithmic design choices that make existing deep RL methods sort of "incompatible" with modern function approximators of today. I will then present a roadmap towards making deep RL scalable and present our progress in two cases: (1) training large language models to be good at "agent" tasks, and (2) developing novel objective functions and network architectures that significantly improve scalability of deep RL algorithms across the board in NLP, robotics, and games. I will conclude by discussing some open questions in this area.
Bio: Aviral Kumar a research scientist at Google DeepMind, based in Mountain View. He finished Ph.D. from UC Berkeley in September 2023, and will start as an Assistant Professor in the Computer Science (CSD) and Machine Learning (MLD) departments at Carnegie Mellon University (CMU) in Fall 2024. His research focuses on developing effective and reliable approaches for (sequential) decision-making. Towards this goal, he focuses on designing reinforcement learning techniques and on understanding and applying these methods in practice. Before his Ph.D., Aviral obtained his B.Tech. in Computer Science from IIT Bombay in India. He is a recipient of the C.V. & Daulat Ramamoorthy Distinguished Research Award, given to 1 PhD student in EECS at Berkeley for outstanding contributions to a new area of research in computer science, Facebook Ph.D. Fellowship in Machine Learning and Apple Scholars in AI/ML Ph.D. Fellowship.