{"id":20515460,"url":"https://github.com/snap-research/mlpinit-for-gnns","last_synced_at":"2025-04-14T00:23:54.602Z","repository":{"id":151008424,"uuid":"524230027","full_name":"snap-research/MLPInit-for-GNNs","owner":"snap-research","description":"[ICLR 2023] MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initialization","archived":false,"fork":false,"pushed_at":"2023-04-05T02:15:42.000Z","size":756,"stargazers_count":77,"open_issues_count":0,"forks_count":4,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-27T14:46:42.666Z","etag":null,"topics":["deep-learning","efficient-training","feedforward-neural-network","gnn","graph-algorithm","graph-neural-networks","multi-layer-perceptron","pytorch","scalability"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2210.00102","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snap-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-12T21:34:50.000Z","updated_at":"2024-12-10T12:13:16.000Z","dependencies_parsed_at":"2023-07-17T11:45:25.432Z","dependency_job_id":null,"html_url":"https://github.com/snap-research/MLPInit-for-GNNs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FMLPInit-for-GNNs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FMLPInit-for-GNNs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FMLPInit-for-GNNs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FMLPInit-for-GNNs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snap-research","download_url":"https://codeload.github.com/snap-research/MLPInit-for-GNNs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248800080,"owners_count":21163404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","efficient-training","feedforward-neural-network","gnn","graph-algorithm","graph-neural-networks","multi-layer-perceptron","pytorch","scalability"],"created_at":"2024-11-15T21:21:53.226Z","updated_at":"2025-04-14T00:23:54.587Z","avatar_url":"https://github.com/snap-research.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initialization\n\nImplementation for the ICLR2023 paper, _MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initialization_,  [[openreview]](https://openreview.net/forum?id=P8YIphWNEGO) [[arxiv]](https://arxiv.org/abs/2210.00102) [[slides]](https://ahxt.github.io/files/mlpinit_slides.pdf), by [Xiaotian Han](https://ahxt.github.io/), [Tong Zhao](https://tzhao.io/), [Yozen Liu](https://scholar.google.com/citations?user=i3U2JjEAAAAJ\u0026hl), [Xia Hu](https://cs.rice.edu/~xh37/index.html), and [Neil Shah](http://nshah.net/).\n\n## 1. Introduction\n\nTraining graph neural networks (GNNs) on large graphs is complex and extremely time consuming. This is attributed to overheads caused by sparse matrix multiplication, which are sidestepped when training multi-layer perceptrons (MLPs) with only node features. MLPs, by ignoring graph context, are simple and faster for graph data, however they usually sacrifice prediction accuracy, limiting their applications for graph data. We observe that for most message passing-based GNNs, we can trivially derive an analog MLP (we call this a PeerMLP) with an equivalent weight space, by setting the trainable parameters with the same shapes, making us curious about how do GNNs using weights from a fully trained PeerMLP perform? Surprisingly, we find that GNNs initialized with such weights significantly outperform their PeerMLPs, motivating us to use PeerMLP training as a precursor, initialization step to GNN training. To this end, we propose an embarrassingly simple, yet hugely effective initialization method for GNN training acceleration, called MLPInit. Our extensive experiments on multiple large-scale graph datasets with diverse GNN architectures validate that MLPInit can accelerate the training of GNNs (up to 33× speedup on OGBN-Products) and often improve prediction performance (e.g., up to 7.97% improvement for GraphSAGE across 7 datasets for node classification, and up to 17.81% improvement across 4 datasets for link prediction on Hits@10).\n### The training speed comparison of the GNNs with Random initialization and MLPInit.\n\u003cimg src=\"img/res.png\" width='800' /\u003e\n\n\n\n## 2. Minimal Example\n\n### 2.1 We provide a self-contained [Jupyter Notebook](minimal_example_of_mlpinit.ipynb) to show the results on ogb-products dataset.\n\n\u003cimg src=\"img/output.png\" width='500' /\u003e\n\n### 2.2 We provide the demo for MLPInit.\n\n```bash\n## cmd for ogbn-products dataset\npython demo/ogbn_sage.py --init_method random --dataset ogbn-products\npython demo/ogbn_sage.py --init_method mlp    --dataset ogbn-products\n```\n\n### 2.3 We also provide the [Jupyter Notebook](gcn_peermlp_comparison.ipynb) to examine the performance of GCN with the weight of PeerMLP (Table 7).\n\n\u003cimg src=\"img/gcn_peermlp.png\" width='500' /\u003e\n\n\n\n\n## 3. Running Environments\n\n### 3.1 Main python packages and their version\n```\ntorch                   1.9.0\ntorch-geometric         2.0.4\nogb                     1.3.3\n```\n\n### 3.2 Environment Setup\nWe use conda to setup the runing environment. GPU installation is required to meet our requirements.\n\n```bash\nconda create --name mlpinit  python=3.7\nconda activate mlpinit\npip install torch==1.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html\npip install -r requirements.txt\n```\n\n### 3.3 Datasets\nAll the datasets will be automatically download by torch-geometric packages.\n\n\n\n\n\n## 4. MLPInit\n\nYou can use the following command to reproduce the results of ogbn-arxiv on GraphSAGE in Table 4. We also provide a shell script [run.sh](run.sh) for other datasets.\n```bash\n## cmd for PeerMLP training\npython -u src/main.py --batch_size 1000 --dataset ogbn-arxiv --dim_hidden 512 --dropout 0.5 --epochs 50 --eval_steps 1 --lr 0.001 --num_layers 4 --random_seed 31415 --save_dir . --train_model_type mlp --gnn_model GraphSAGE --weight_decay 0\n## cmd for GraphSAGE training with MLPInit\npython -u src/main.py --batch_size 1000 --dataset ogbn-arxiv --dim_hidden 512 --dropout 0.5 --epochs 50 --eval_steps 1 --lr 0.001 --num_layers 4 --random_seed 31415 --save_dir . --train_model_type gnn --gnn_model GraphSAGE --weight_decay 0 --pretrained_checkpoint ./ogbn-arxiv_GraphSAGE_mlp_512_4_31415.pt \n```\n\n\n\n\n\n\n\n## 5. Cite Our Paper\n\nIf you find our paper or code is useful for you research, please cite our paper.\n\n```BiBTeX\n@inproceedings{han2023mlpinit,\n        title={{MLPI}nit: Embarrassingly Simple {GNN} Training Acceleration with {MLP} Initialization},\n        author={Xiaotian Han and Tong Zhao and Yozen Liu and Xia Hu and Neil Shah},\n        booktitle={International Conference on Learning Representations},\n        year={2023},\n        url={https://openreview.net/forum?id=P8YIphWNEGO}\n}\n```\n\n## 6. Credits\nThe GNN architectures and training code are based on https://github.com/VITA-Group/Large_Scale_GCN_Benchmarking\n\n## 7. Contact\nPlease contact [Xiaotian Han](https://ahxt.github.io/) if you have any question about this code.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnap-research%2Fmlpinit-for-gnns","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnap-research%2Fmlpinit-for-gnns","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnap-research%2Fmlpinit-for-gnns/lists"}