Expert assignment is an important process at the heart of knowledge production, including scientific peer review and community question answering websites like StackExchange. However, experts can only perform these tasks effectively if they have the proper expertise, interest, and availability. We develop multiple novel approaches to assign experts to knowledge-specific tasks, addressing questions of fairness, scalability, assignment quality, and robustness to uncertainty.
In the domain of reviewer assignment, our contributions are as follows:
1) We present two algorithms based on the classic picking sequence mechanism, that assign reviewers to papers in such a way that no paper "envies" another paper's assigned reviewers. This approach is simple and fast, and ensures that reviewer expertise is fairly distributed amongst the papers. 14 venues have used the assignments produced by our FairSequence algorithm on OpenReview in the last 1.5 years.
2) Existing methods for calculating reviewer-paper fit can be noisy and ineffective. We present the Robust Reviewer Assignment (RRA) framework that robustly optimizes over a region containing the true fit scores with high probability, directly addressing the inherent uncertainty in the assignment process.
We also propose the completion of 2 ongoing projects:
1) We are currently extending the RRA algorithm to incorporate group fairness objectives under adversarial uncertainty, and to cover distribution-based robustness concepts besides adversarial robustness (for both utilitarian and egalitarian objectives).
2) We also propose a study on predictive expert assignment in community question answering forums like StackExchange. We find that predicting future task performance, and assigning for predicted performance, can boost outcomes across a wide range of metrics. We also show that historical performance data is a powerful predictor of future performance, and should be further investigated in other expert assignment tasks.
Advisor: Yair Zick