https://github.com/vita-group/gradientgcn
[NeurIPS 2022] Old can be Gold: Better Gradient Flow can Make Vanilla-GCNs Great Again by Ajay Jaiswal*, Peihao Wang*, Tianlong Chen, Justin F Rousseau, Ying Ding, Zhangyang Wang
https://github.com/vita-group/gradientgcn
deep-gcns gradient-flow graph-neural-networks initialization
Last synced: about 1 month ago
JSON representation
[NeurIPS 2022] Old can be Gold: Better Gradient Flow can Make Vanilla-GCNs Great Again by Ajay Jaiswal*, Peihao Wang*, Tianlong Chen, Justin F Rousseau, Ying Ding, Zhangyang Wang
- Host: GitHub
- URL: https://github.com/vita-group/gradientgcn
- Owner: VITA-Group
- Created: 2022-09-29T22:01:57.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2022-11-25T03:46:00.000Z (over 2 years ago)
- Last Synced: 2025-03-29T09:42:01.572Z (2 months ago)
- Topics: deep-gcns, gradient-flow, graph-neural-networks, initialization
- Language: Python
- Homepage:
- Size: 81.1 KB
- Stars: 8
- Watchers: 11
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Old can be Gold: Better Gradient Flow can make Vanilla-GCNs Great Again
[](https://opensource.org/licenses/MIT)
https://arxiv.org/abs/2210.08122
## Abstract
Despite the enormous success of Graph Convolutional Networks (GCNs) in mod-
elling graph-structured data, most of the current GCNs are shallow due to the
notoriously challenging problems of over-smoothening and information squashing
along with conventional difficulty caused by vanishing gradients and over-fitting.
Previous works have been primarily focused on the study of over-smoothening and
over-squashing phenomenon in training deep GCNs. Surprisingly, in comparison
with CNNs/RNNs, very limited attention has been given towards understanding
how healthy gradient flow can benefit the trainability of deep GCNs. In this paper,
firstly, we provide a new perspective of gradient flow to understand the substandard
performance of deep GCNs and hypothesize that by facilitating healthy gradient
flow, we can significantly improve their trainability, as well as achieve state-of-the-
art (SOTA) level performance from vanilla-GCNs [1]. Next, we argue that blindly
adopting the Glorot initialization for GCNs is not optimal, and derive a topology-
aware isometric initialization scheme for vanilla-GCNs based on the principles
of isometry. Additionally, contrary to ad-hoc addition of skip-connections, we
propose to use gradient-guided dynamic rewiring of vanilla-GCNs with skip-
connections. Our dynamic rewiring method uses the gradient flow within each
layer during training to introduce skip-connections on-demand basis. We provide
extensive empirical evidence across multiple datasets that our methods improves
gradient flow in deep vanilla-GCNs and significantly boost their performance to
comfortably compete and outperform many fancy state-of-the-art methods.
## Benefits of our proposed techniques
If you find our work helpful in your research, please cite our paper
## Citation
If you find our code implementation helpful for your own resarch or work, please cite our paper.
```
@inproceedings{Jaiswal22GradientGCN,
title={Old can be Gold: Better Gradient Flow can make Vanilla-GCNs Great Again},
author={Ajay Jaiswal, Peihao Wang, Tianlong Chen, Justin F Rousseau, Ying Ding, Zhangyang Wang},
booktitle={NeurIPS 2022},
year={2022}
}
```