Seminar: Local and Adaptive Visual Recognition

Abstract: The visual world is vast and varied, so learning is needed to reliably recognize what is where in images. In this talk, I will show how to reconcile learning with visual structure, to make the most of the data during training, and how to extend learning by adapting the model on every input, to cope with changing data during testing. Fully convolutional networks harness locality structure for image-to-image learning and inference in order to efficiently learn from more data. Nevertheless, data varies and even shifts: variation from image to image or shifts from training to testing can degrade the accuracy of a static model that stays the same on all inputs. To mitigate variations in size across images, we can factor the model into structured and free-form parameters, and learn both, to optimize the size and shape of filters along with their contents. To mitigate shifts across training and testing, we can equip a model with optimization during inference to update itself as the data changes, in particular to optimize for its own confidence by entropy minimization. These choices take steps toward turning machine perception into more of a process, rather than a fixed computation, for more robust recognition.