PhD Dissertation Proposal: Aaron Sun, Dealing with Difficult Data: Learning in Unorthodox Domains with Missing, Noisy, or Limited Labels
Content
Speaker:
Abstract:
In many scientific and ecological applications of computer vision, data and label quality and quantity present a significant challenge due to lack of data for long-tailed classes and expensive or infeasible scaling of expert labeling. Furthermore, foundation models lack generalization capabilities for these specific contexts, meaning each problem requires a different and unique solution depending on the setting. This thesis explores the best ways to deal with data from these unconventional and specialized domains with incomplete or unreliable labels.
First, the thesis examines fine-grained bird sound identification in a single positive, multi-label context, meaning each recording has a single label while other species may be present. Then, the thesis finds that state-of-the-art identity-preserving image generation methods falter in fine-grained contexts which lack data, but find that training noisy proxy identity labels lead to strong improvements. Finally, this thesis benchmarks image self-supervised learning methods on a class of nanoporous materials known as zeolites which we represent as 3D volumes, and discover that some methods outperform state-of-the-art supervised training while others fall below ImageNet pretrained networks adapted to 3D.
As an extension, I propose to focus more on the concept of capturing and understanding identity itself through multiple views. While existing identity encoders are restricted to a single image, I aim to focus on a multi-view image identity embedding which can capture information about geometry and multi-sided details which cannot be seen with a single view.
Advisor:
Subhransu Maji