Abstract: Deep neural networks significantly power the success of machine learning. Over the past decade, the community keeps designing architectures of deep layers and complicated connections. However, the gap between deep learning theory and application is growingly large. This talk will center around this challenge and tries to bridge the gap between the two worlds. By theoretically analyzing a network's Jacobian, NNGP, and NTK, we find an intrinsic trade-off in network architectures. Given a space of architectures, a network cannot be optimal in its expressivity, trainability, and generalization at the same time, and it has to keep a balance between its depth and width. In other words, separately optimizing expressivity, trainability, and generalization will give us different network architectures. This analysis has further practical implications. Automated machine learning (AutoML) is a powerful tool to address design problems, yet, at the price of heavy computation costs during model training. Our theory serves as accurate and efficient guidance for the architecture design. We propose to significantly accelerate AutoML with our theory-grounded, training-free metrics. Without any training cost, our TE-NAS framework can automatically design novel and accurate network architectures on ImageNet in only four GPU hours.
Bio: Wuyang Chen is a Ph.D. candidate in Electrical and Computer Engineering at University of Texas at Austin. Wuyang's research focuses on theoretical understandings of deep network architectures and AutoML applications. Wuyang also worked on domain adaptation/generalization and self-supervised learning. His work is published on ICLR, ICML, CVPR, ICCV, etc. Wuyang completed his research internship in NVIDIA and Google Brain. Wuyang chaired the 4th and the 5th version of UG2+ workshop and challenge in CVPR 2021 and 2022. Wuyang is also a board member of the One World Seminar Series on the Mathematics of Machine Learning.