{"id":20663729,"url":"https://github.com/vita-group/deep_gcn_benchmarking","last_synced_at":"2025-04-19T15:56:00.166Z","repository":{"id":39815450,"uuid":"395146671","full_name":"VITA-Group/Deep_GCN_Benchmarking","owner":"VITA-Group","description":"[TPAMI 2022] \"Bag of Tricks for Training Deeper Graph Neural Networks A Comprehensive Benchmark Study\" by Tianlong Chen*, Kaixiong Zhou*, Keyu Duan, Wenqing Zheng, Peihao Wang, Xia Hu, Zhangyang Wang","archived":false,"fork":false,"pushed_at":"2021-12-29T10:50:58.000Z","size":824,"stargazers_count":126,"open_issues_count":3,"forks_count":21,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-29T09:42:01.666Z","etag":null,"topics":["benchmark","deep-graph-library","deep-graph-networks","dropout","graph-neural-networks","normalization","ogb","skip-connections","training-tricks"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VITA-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-08-12T00:08:39.000Z","updated_at":"2024-12-01T22:07:42.000Z","dependencies_parsed_at":"2022-08-27T16:51:48.603Z","dependency_job_id":null,"html_url":"https://github.com/VITA-Group/Deep_GCN_Benchmarking","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FDeep_GCN_Benchmarking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FDeep_GCN_Benchmarking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FDeep_GCN_Benchmarking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FDeep_GCN_Benchmarking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VITA-Group","download_url":"https://codeload.github.com/VITA-Group/Deep_GCN_Benchmarking/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249731218,"owners_count":21317341,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","deep-graph-library","deep-graph-networks","dropout","graph-neural-networks","normalization","ogb","skip-connections","training-tricks"],"created_at":"2024-11-16T19:19:32.195Z","updated_at":"2025-04-19T15:56:00.145Z","avatar_url":"https://github.com/VITA-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n\nCodes for [Preprint] [Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study](https://arxiv.org/pdf/2108.10521.pdf)\n\nTianlong Chen\\*, Kaixiong Zhou\\*, Keyu Duan, Wenqing Zheng, Peihao Wang, Xia Hu, Zhangyang Wang\n\n## Introduction\n\nThis is the first fair and reproducible benchmark dedicated to assessing the \"tricks\" of training deep GNNs. We categorize existing approaches, investigate their hyperparameter sensitivity, and unify the basic configuration. Comprehensive evaluations are then conducted on tens of representative graph datasets including the recent large-scale Open Graph Benchmark (OGB), with diverse deep GNN backbones. Based on synergistic studies, we discover the transferable combo of superior training tricks, that lead us to attain the new state-of-the-art results for deep GCNs, across multiple representative graph datasets.\n\n## Requirements\n#### Installation with Conda\n```bash\nconda create -n deep_gcn_benchmark\nconda activate deep_gcn_benchmark\npip install -r requirement.txt\n```\n\n#### Our Installation Notes for PyTorch Geometric.\n\nWhat env configs that we tried that have succeeded: Mac/Linux + cuda driver 11.2 + Torch with cuda 11.1 + torch_geometric/torch sparse/etc with cuda 11.1.\n\nWhat env configs that we tried but didn't work: Linux+Cuda 11.1/11.0/10.2 + whatever version of Torch.\n\nIn the above case when it did work, we adopted the following installation commands, and it automatically downloaded built wheels, and the installation completes within seconds.\n\nIn the case when it did not work, the installation appears to be very slow (ten minutes level for torch sparse/torch scatter). Then the installation did not produce any error, while when import torch_geometric in python code, it reports errors of different types.\n\nInstallation codes that we adopted on Linux cuda 11.2 that did work:\n\n```bash\npip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html\npip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.0+cu111.html\npip install torch-sparse -f https://pytorch-geometric.com/whl/torch-1.9.0+cu111.html\npip install torch-geometric\n```\n\n## Project Structure\n\n```\n.\n├── Dataloader.py\n├── main.py\n├── trainer.py\n├── models\n│   ├── *.py\n├── options\n│   ├── base_options.py\n│   └── configs\n│       ├── *.yml\n├── tricks\n│   ├── tricks\n│   │   ├── dropouts.py\n│   │   ├── norms.py\n│   │   ├── others.py\n│   │   └── skipConnections.py\n│   └── tricks_comb.py\n└── utils.py\n```\n\n## How to Use the Benchmark\n\n#### Train Deep GCN models as your baselines\n\nTo train a deep GCN model `\u003cmodel\u003e` on dataset `\u003cdataset\u003e` as your baseline, run:\n\n```bash\npython main.py --compare_model=1 --cuda_num=0 --type_model=\u003cmodel\u003e --dataset=\u003cdataset\u003e\n# \u003cmodel\u003e   in  [APPNP, DAGNN, GAT, GCN, GCNII, GPRGNN, JKNet, SGC]\n# \u003cdataset\u003e in  [Cora, Citeseer, Pubmed, ogbn-arixv, CoauthorCS, CoauthorPhysics, AmazonComputers, AmazonPhoto, TEXAS, WISCONSIN, CORNELL, ACTOR]\n```\n\nwe comprehensively explored the optimal hyperparameters for all models we implemented and train the models under the\nwell-studied hyperparameter settings. For model-specific hyperparameter configs, please refer to `options/configs/*.yml`\n\n#### Explore different trick combinations\n\nTo explore different trick combinations, we provide a `tricks_comb` model, which integrates different types of tricks as\nfollows:\n\n```\ndropouts:        DropEdge, DropNode, FastGCN, LADIES\nnorms:           BatchNorm, PairNorm, NodeNorm, MeanNorm, GroupNorm, CombNorm\nskipConnections: Residual, Initial, Jumping, Dense\nothers:          IdentityMapping\n```\n\nTo train a `tricks_comb` model with specific tricks, run:\n\n```bash\npython main.py --compare_model=0 --cuda_num=0 --type_trick=\u003ctrick_1\u003e+\u003ctrick_2\u003e+...+\u003ctrick_n\u003e --dataset=\u003cdataset\u003e\n```\n\n, where you can assign `type_trick` with any number of tricks. For instance, to train a `trick_comb` model\nwith `Initial`, `EdgeDrop`, `BatchNorm` and `IdentityMapping` on Cora, run:\n\n```bash\npython main.py --compare_model=0 --cuda_num=0 --type_trick=Initial+EdgeDrop+BatchNorm+IdentityMapping --dataset=Cora\n```\n\nWe provide two backbones `--type_model=GCN` and `--type_tricks=SGC` for trick combinations. Specifically,\nwhen `--type_model=SGC` and `--type_trick=IdenityMapping` co-occur, `IdentityMapping` has higher priority.\n\n## How to Contribute\nYou are welcome to make any type of contributions. Here we provide a brief guidance to add your own deep GCN models and tricks.\n\n#### Add your own model\nSeveral simple steps to add your own deep GCN model `\u003cDeepGCN\u003e`.\n\n1. Create a python file named `\u003cDeepGCN\u003e.py`\n2. Implement your own model as a `torch.nn.Module`, where the class name is recommended to be consistent with your filename `\u003cDeepGCN\u003e`\n3. Make sure the commonly-used hyperparameters is consistent with ours (listed as follows). To create any new hyperparameter, add it in `options/base_options.py`.\n```\n --dim_hidden        # hidden dimension\n --num_layers        # number of GCN layers\n --dropout           # rate of dropout for GCN layers\n --lr:               # learning rate\n --weight_decay      # rate of l2 regularization\n```\n4. Register your model in `models/__init__.py` by add the following codes:\n```python\nfrom \u003cDeepGCN\u003e import \u003cDeepGCN\u003e\n__all__.append('\u003cDeepGCN\u003e')\n```\n5. You are recommend to use `YAML` to store your dataset-specific hyperparameter configuration. Create a `YAML` file `\u003cDeepGCN\u003e.yml` in `options/configs` and add the hyperparameters as the following style:\n```yaml\n\u003cdataset_1\u003e\n  \u003chyperparameter_1\u003e : value_1\n  \u003chyperparameter_2\u003e : value_2\n```\nNow your own model `\u003cDeepGCN\u003e` should be added successfully into our benchmark framework. To test the performance of `\u003cDeepGCN\u003e` on `\u003cdataset\u003e`, run:\n```bash\npython main.py --compare_model=1 --type_model=\u003cDeepGCN\u003e --dataset=\u003cdataset\u003e\n```\n\n\n#### Add your own trick\nAs all implemented tricks are coupled in `tricks_comb.py` tightly, we do not recommend integrating your own trick to `trick_comb` to avoid unexpected errors. However, you can use the interfaces we provided in `tricks/tricks/` to combine your own trick with ours.\n\n## Main Results and Leaderboard\n\n- Superior performance of our best combo with 32 layers deep GCNs\n\n| Model Ranking on Cora | Test Accuracy |\n| :-------------------: | :-----------: |\n|         Ours          |     85.48     |\n|         GCNII         |     85.29     |\n|         APPNP         |     83.68     |\n|         DAGNN         |     83.39     |\n|        GPRGNN         |     83.13     |\n|         JKNet         |     73.23     |\n|          SGC          |     68.45     |\n\n| Model Ranking on Citeseer | Test Accuracy |\n| :-----------------------: | :-----------: |\n|           Ours            |     73.35     |\n|           GCNII           |     73.24     |\n|           DAGNN           |     72.59     |\n|           APPNP           |     72.13     |\n|          GPRGNN           |     71.01     |\n|            SGC            |     61.92     |\n|           JKNet           |     50.68     |\n\n| Model Ranking on PubMed | Test Accuracy |\n| :---------------------: | :-----------: |\n|          Ours           |     80.76     |\n|          DAGNN          |     80.58     |\n|          APPNP          |     80.24     |\n|          GCNII          |     79.91     |\n|         GPRGNN          |     78.46     |\n|           SGC           |     66.61     |\n|          JKNet          |     63.77     |\n\n| Model Ranking on OGBN-ArXiv | Test Accuracy |\n| :-------------------------: | :-----------: |\n|            Ours             |     72.70     |\n|            GCNII            |     72.60     |\n|            DAGNN            |     71.46     |\n|           GPRGNN            |     70.18     |\n|            APPNP            |     66.94     |\n|            JKNet            |     66.31     |\n|             SGC             |     34.22     |\n\n- Transferability of our best combo with 32 layers deep GCNs\n\n| Models | Average Ranking on (CS, Physics, Computers, Photo, Texas, Wisconsin, Cornell, Actor) |\n| :----: | :----------------------------------------------------------: |\n|  Ours  |                            1.500                             |\n|  SGC   |                            6.250                             |\n| DAGNN  |                            4.375                         |\n| GCNII  |                            3.875                           |\n| JKNet  |                            4.875                           |\n| APPNP  |                            4.000                         |\n| GPRGNN |                            3.125                           |\n\n- Takeaways of the best combo\n\n![](figs/combo.png)\n\n## Citation\nif you find this repo is helpful, please cite\n```\n@misc{chen2021bag,\n      title={Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study}, \n      author={Tianlong Chen and Kaixiong Zhou and Keyu Duan and Wenqing Zheng and Peihao Wang and Xia Hu and Zhangyang Wang},\n      year={2021},\n      eprint={2108.10521},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fdeep_gcn_benchmarking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvita-group%2Fdeep_gcn_benchmarking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fdeep_gcn_benchmarking/lists"}