PhD Dissertation Proposal Defense: Sandeep Polisetty, Abstractions to Eliminate Redundancy in Training Graph Neural Networks on GPUs
Content
Speaker
Abstract
Graph machine learning has achieved significant success in extending deep learning to graph-structured data, removing the need for hand-engineered features. These models typically use message passing across edges—expressed as sparse computations—to capture the relational structure inherent in graphs. However, current abstractions for mapping such computations onto highly parallel processors like GPUs are often inefficient, introducing redundancy due to the dense and irregular connectivity of graph data.
In the first part of my thesis, I introduce split parallelism, a novel
abstraction that addresses the limitations of traditional data parallelism on GPUs. In this approach, each node is assigned to a specific GPU, and information from remote neighbors is communicated across GPUs at every hidden layer. We also present an API that allows various graph machine learning models to be expressed with minimal modification to the standard training loop. To further optimize this approach, we propose a graph partitioning strategy that minimizing inter-GPU communication for split-parallel training. Our implementation of split parallelism achieves up to 1.7× speedup over state-of-the-art methods.
We find that one drawback of split-parallelism as we attempt to train
larger/deeper models is that communication and synchronization costs can become substantive. For future work, I aim to reduce these costs by exploring hybrid forms of parallelism:
- Pipeline + Split Parallelism: This approach introduces controlled
redundancy across multiple batches, overlapping computation of one batch’s split with the communication of another, thereby effectively hiding communication costs. - Data + Split Parallelism: Here, split parallelism is selectively applied
only to certain layers, while other layers use traditional data parallelism, reducing overhead of communication and synchronization.
Advisors
Hui Guan and Marco Serafini
Host Name: Sandeep Polisetty
Host URL: https://umass-amherst.zoom.us/j/99105166451