Faculty Recruiting Support CICS

Language as a Scaffold for Grounded Intelligence

21 Mar
Thursday, 03/21/2019 4:00pm to 5:00pm
Computer Science Building, Room 151
Special Event
Speaker: Mark Yatskar

Abstract: Natural language can be used to construct rich, compositional descriptions of the world, highlighting for example entities (nouns), events (verbs), and the interactions between them (simple sentences). In this talk, I show how compositional structure around verbs and nouns can be repurposed to build computer vision systems that scale to recognize hundreds of thousands of visual concepts in images. I introduce the task of situation recognition, where the goal is to map an image to a language-inspired structured representation of the main activity it depicts. The problem is challenging because it requires recognition systems to identify not only what entities are present, but also how they are participating within an event (e.g. not only that there are scissors but they are they are being used to cut). I also describe new deep learning models that better capture compositionality in situation recognition and leverage the close connection to language 'to know what we don't know' and cheaply mine new training data. Although these methods work well, I show that they have a tendency to amplify underlying societal biases in the training data (including over predicting stereotypical activities based on gender), and introduce a new dual decomposition method that significantly reduces this amplification without sacrificing classification accuracy. Finally, I propose new directions for expanding what visual recognition systems can see and ways to minimize the encoding of negative social biases in our learned models.

Bio: Mark Yatskar is a post-doc at the Allen Institute for Artificial Intelligence and recipient of their Young Investigator Award. His primary research is in the intersection of natural language processing and computer vision and fairness in machine learning.  He received his Ph.D. from the University of Washington with Luke Zettlemoyer and Ali Farhadi, received the EMNLP best paper award in 2017, and his work has been featured in Wired and the New York Times.

A reception for attendees will be held in CS 150 at 3:30 p.m.


Faculty Host