{"id":13958728,"url":"https://github.com/chaitjo/efficient-gnns","last_synced_at":"2025-04-05T06:06:13.768Z","repository":{"id":43397002,"uuid":"390534061","full_name":"chaitjo/efficient-gnns","owner":"chaitjo","description":"Code and resources on scalable and efficient Graph Neural Networks","archived":false,"fork":false,"pushed_at":"2023-02-04T07:43:14.000Z","size":2436,"stargazers_count":526,"open_issues_count":0,"forks_count":64,"subscribers_count":20,"default_branch":"main","last_synced_at":"2024-09-29T21:02:58.332Z","etag":null,"topics":["contrastive-learning","efficient-deep-learning","efficient-neural-networks","gnn","graph-convolutional-networks","graph-neural-networks","graph-representation-learning","knowledge-distillation","scalable-deep-learning"],"latest_commit_sha":null,"homepage":"https://www.chaitjo.com/post/efficient-gnns/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chaitjo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-28T23:09:11.000Z","updated_at":"2024-09-23T09:56:49.000Z","dependencies_parsed_at":"2023-02-18T14:45:44.892Z","dependency_job_id":null,"html_url":"https://github.com/chaitjo/efficient-gnns","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaitjo%2Fefficient-gnns","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaitjo%2Fefficient-gnns/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaitjo%2Fefficient-gnns/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaitjo%2Fefficient-gnns/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chaitjo","download_url":"https://codeload.github.com/chaitjo/efficient-gnns/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247294536,"owners_count":20915340,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["contrastive-learning","efficient-deep-learning","efficient-neural-networks","gnn","graph-convolutional-networks","graph-neural-networks","graph-representation-learning","knowledge-distillation","scalable-deep-learning"],"created_at":"2024-08-08T13:01:49.706Z","updated_at":"2025-04-05T06:06:13.748Z","avatar_url":"https://github.com/chaitjo.png","language":"Python","funding_links":[],"categories":["图机器学习库"],"sub_categories":["网络服务_其他"],"readme":"# Knowledge Distillation for Resource-efficient Graph Neural Networks\n\nThis repository provides resources on Graph Neural Network efficiency and scalability, as well as implementations of knowledge distillation techniques for developing resource-efficient GNNs.\n\n![Knowledge distillation pipeline for GNNs](img/pipeline.png)\n\nCheck out the accompanying paper ['On Representation Knowledge Distillation for Graph Neural Networks'](https://arxiv.org/abs/2111.04964), which introduces new GNN distillation techniques using contrastive learning to preserve the global topology of teacher and student embeddings.\n\n\u003e Chaitanya K. Joshi, Fayao Liu, Xu Xun, Jie Lin, and Chuan Sheng Foo. On Representation Knowledge Distillation for Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), *Special Issue on Deep Neural Networks for Graphs: Theory, Models, Algorithms and Applications*.\n\u003e\n\u003e [PDF](https://arxiv.org/pdf/2111.04964.pdf) | [Blog](https://www.chaitjo.com/post/efficient-gnns/)\n\n❓New to GNN scalability: See [`awesome-efficient-gnns.md`](awesome-efficient-gnns.md) and the [accompanying blogpost](https://www.chaitjo.com/post/efficient-gnns/) for a currated overview of papers on efficient and scalable Graph Representation Learning for real-world applications.\n\n## Distillation Techniques\n\n![Representation distillation techniques for GNNs](img/techniques.png)\n\nWe benchmark the following knowledge distillation techniques for GNNs:\n- **Local Structure Preserving loss**, [Yang et al., CVPR 2020](https://arxiv.org/abs/2003.10477): preserve pairwise relationships over graph edges, but may not preserve global topology due to latent interactions. \n- **Global Structure Preserving loss**, [Joshi et al., TNNLS 2022](https://arxiv.org/abs/2111.04964): preserve all pairwise global relationships, but computationally more cumbersome.\n- 🌟 **Graph Contrastive Representation Distillation**, [Joshi et al., TNNLS 2022](https://arxiv.org/abs/2111.04964): contrastive learning among positive/negative pairwise relations across the teacher and student embedding spaces.\n- 🔥 Your new GNN distillation technique?\n\nWe also include baselines: **Logit-based KD**, [Hinton et al., 2015](https://arxiv.org/abs/1503.02531); and feature mimicking baselines for computer vision: **FitNet**, [Romero et al., 2014](https://arxiv.org/abs/1412.6550), **Attention Transfer**, [Zagoruyko and Komodakis, 2016](https://arxiv.org/abs/1612.03928).\n\n## Datasets and Architectures\n\nWe conduct benchmarks on large-scale and real-world graph datasets, where the performance gap between expressive+cumbersome teacher and resource-efficient student GNNs is non-negligible:\n- **Graph classification** on `MOLHIV` from Open Graph Benchmark/MoleculeNet -- GIN-E/PNA teachers, GCN/GIN students.\n- **Node classification** on `ARXIV` and `MAG` from Open Graph Benchmark and Microsoft Academic Graph -- GAT/R-GCN teachers, GCN/GraphSage/SIGN students.\n- **3D point cloud segmentation** on `S3DIS` -- not released publicly yet.\n- **Node classification** on `PPI` -- provided to reproduce results from [Yang et al.](https://arxiv.org/abs/2003.10477)\n\n## Installation and Usage\n\nOur results are reported with Python 3.7, PyTorch, 1.7.1, and CUDA 11.0.\nWe used the following GPUs: RTX3090 for ARXIV/MAG, V100 for MOLHIV/S3DIS.\n\nUsage instructions for each dataset are provided within the corresponding directory.\n\n```sh\n# Create new conda environment\nconda create -n ogb python=3.7\nconda activate ogb\n\n# Install PyTorch (Check CUDA version!)\nconda install pytorch=1.7.1 cudatoolkit=11.0 -c pytorch\n\n# Install DGL\nconda install -c dglteam dgl-cuda11.0\n\n# Install PyG\nCUDA=cu110\npip3 install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.7.1+${CUDA}.html\npip3 install torch-sparse -f https://pytorch-geometric.com/whl/torch-1.7.1+${CUDA}.html\npip3 install torch-cluster -f https://pytorch-geometric.com/whl/torch-1.7.1+${CUDA}.html\npip3 install torch-spline-conv -f https://pytorch-geometric.com/whl/torch-1.7.1+${CUDA}.html\npip3 install torch-geometric\n\n# Install other dependencies\nconda install tqdm scikit-learn pandas urllib3 tensorboard\npip3 install ipdb, nvidia-ml-py3\n\n# Install OGB\npip3 install -U ogb\n```\n\n## Citation\n\n```\n@article{joshi2022representation,\n  title={On Representation Knowledge Distillation for Graph Neural Networks},\n  author={Chaitanya K. Joshi and Fayao Liu and Xu Xun and Jie Lin and Chuan-Sheng Foo},\n  journal={IEEE Transactions on Neural Networks and Learning Systems},\n  year={2022}\n}\n\n@article{joshi2022efficientgnns,\n  author = {Joshi, Chaitanya K.},\n  title = {Recent Advances in Efficient and Scalable Graph Neural Networks},\n  year = {2022},\n  howpublished = {\\url{https://www.chaitjo.com/post/efficient-gnns/}},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchaitjo%2Fefficient-gnns","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchaitjo%2Fefficient-gnns","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchaitjo%2Fefficient-gnns/lists"}