Machine Learning and Friends Lunch (Online)

Title: Learning-based Dynamics Modeling for Physical Inference and Model-based Control

Abstract: Humans have a strong intuitive understanding of physics. From visual observations, we can predict how the environment would change if we apply a specific action. This ability applies to objects of different materials, including rigid bodies, deformable objects, and fluids, which enables a tremendous amount of manipulation skills that are far beyond the reach of the current robot. It is desirable to help the robot learn from its interactions and better understand the dynamics. In this talk, I will present our attempt to model the dynamics of the environment from observational data by combining learning with different state representations and model class, aiming to capture the compositional nature of the objects and introduce the desired inductive bias. Using the learned simulators, robots can infer the physical properties and the relational structure within a dynamic environment and have achieved success in complex manipulation tasks, such as manipulating a pile of boxes, a cup of water, and a deformable foam. I will also present our work on building a scalable tactile glove to learn the patterns in human-object interaction and discuss how it can contribute to a multi-modal perception system that can facilitate human activities learning and dynamics understanding.

Bio: Yunzhu Li is a Ph.D. student at MIT, advised by Prof. Antonio Torralba and Prof. Russ Tedrake. He works on the intersection of computer vision, machine learning, and robotics. He is particularly interested in enabling robots to better perceive and interact with the world via learning-based dynamics modeling and multimodal perception. Yunzhu is a recipient of the Adobe Research Fellowship. His research was published in top journals and conferences, including Nature, NeurIPS, CVPR, ICRA, etc., and has been featured by CNN, BBC, Forbes, The Economist, MIT Technology Review, WIRED, and other media outlets. Before coming to MIT, he received a B.S. degree from Peking University. He has also spent time at Stanford AI Lab and NVIDIA Robotics Research Lab.

To obtain the Zoom link for this event, please see the event announcements from MLFL on the college email lists or contact Kalpesh Krishna.