Canceled: CIIR Talk Series: Evaluation by Maximum Similarity to an Ideal Ranking
This event has been canceled due to illness.
Title: Evaluation by Maximum Similarity to an Ideal Ranking
Abstract: In this talk, I propose a radical simplification of offline evaluation for search, recommendation, and question-answering systems. Traditional information retrieval measures, notably NDCG, require definitions for two or more relevance levels, which human assessors then apply to judge individual documents. Due to this dependence on a definition of relevance, it can be difficult to extend these measures to account for factors beyond relevance and to align these measures with online metrics. Instead, for each query, we define a set of ideal rankings and compute the maximum rank similarity between members of this set and an actual ranking generated by a system. This maximum similarity to an ideal ranking becomes our effectiveness measure, replacing NDCG and similar measures. To illustrate the measure, I discuss the use of preference labels as a method for determining ideal rankings. These preference labels can be obtained from human judgments, inferred from online interactions, or generated by large language models. Preference judgments can also recognize distinctions between items that appear equivalent under graded judgments, accounting for factors such as readability and recency. I report experiments on crowdsourcing preference labels for the MS Marco dataset, TREC 2019 Conversational Assistance Track, and the TREC 2021 Deep Learning Track, along with experiments illustrating the measure on diversity and health misinformation tasks.
Bio: Charles Clarke is a Professor of Computer Science at the University of Waterloo. His research focuses on data-intensive tasks and efficiency, including search, ranking, question answering, and other problems involving human language data. He has supervised to completion over 30 graduate students and published over 200 refereed contributions on a wide range of topics, including search, metrics, user interfaces, filesystem search, natural language processing, machine learning, and databases. He has worked on search engine technology for both Microsoft Bing and Facebook Search. Clarke is an ACM Distinguished Scientist, an elected member of the SIGIR Academy, and leading member of the search and information retrieval community, serving as the Chair of the Executive Committee for the ACM Special Interest Group from 2013 to 2016 and as the Co-Editor-in-Chief of the Information Retrieval Journal from 2010 to 2018.
To attend this talk via Zoom, click here. To obtain the passcode for this series, please see the event advertisement on the seminars email list or reach out to Alex Taubman. For any questions about this event with the Center for Intelligent Information Retrieval, please contact Jean Joyce.