CIIR Talk Series

Join the Center for Intelligent Information Retrieval for this talk by Fernando Diaz, Research Scientist at Google Brain Montreal.

Title: Evaluating Stochastic Rankings with Expected Exposure

Abstract: We introduce the concept of expected exposure as the average attention ranked items receive from users over repeated samples of the same query. Furthermore, we advocate for the adoption of the principle of equal expected exposure: given a fixed information need, no item receives more or less expected exposure compared to any other item of the same relevance grade. We argue that this principle is desirable for many retrieval objectives and scenarios, including topical diversity and fair ranking. Leveraging user models from existing retrieval metrics, we propose a general evaluation methodology based on expected exposure and draw connections to related metrics in information retrieval evaluation. Importantly, this methodology relaxes classic information retrieval assumptions, allowing a system, in response to a query, to produce a distribution over rankings instead of a single fixed ranking. We study the behavior of the expected exposure metric and stochastic rankers across a variety of information access conditions, including ad hoc retrieval and recommendation. We believe that measuring and optimizing expected exposure metrics using randomization opens a new area for retrieval algorithm development and progress.

Joint work with Bhaskar Mitra, Michael Ekstrand, Asia Biega, and Ben Carterette. CIKM 2020 best paper award nominee. Research conducted while at Microsoft Research Montreal.

 

Biography: Fernando Diaz is a research scientist at Google Brain Montreal. His research focuses on the design of information access systems, including search engines, music recommendation services and crisis response platforms.  He is particularly interested in understanding and addressing the societal implications of artificial intelligence more generally. Previously, Fernando was the assistant managing director of Microsoft Research Montreal and a director of research at Spotify, where he helped establish its research organization on recommendation, search, and personalization.  Fernando's work has received awards at SIGIR, WSDM, ISCRAM, and ECIR. He is the recipient of the 2017 British Computer Society Karen Sparck Jones Award. Fernando has co-organized workshops and tutorials at SIGIR, WSDM, and WWW. He has also co-organized several NIST TREC initiatives, WSDM (2013), Strategic Workshop on Information Retrieval (2018), FAT* (2019), SIGIR (2021), and the CIFAR Workshop on Artificial Intelligence and the Curation of Culture (2019).

 

To attend this virtual meeting via Zoom, click here. Participants will need to enter a passcode to attend in this event. If you need the passcode for this series, please see the event advertisement on the seminars email list or reach out to Alex Taubman. For additional questions about the event, please contact Hamed Zamani.