Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/snap-research/graphless-neural-networks
[ICLR 2022] Code for Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation (GLNN)
https://github.com/snap-research/graphless-neural-networks
deep-learning distillation efficient-inference gnn graph-algorithm graph-neural-networks knowledge-distillation pytorch scalability
Last synced: 1 day ago
JSON representation
[ICLR 2022] Code for Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation (GLNN)
- Host: GitHub
- URL: https://github.com/snap-research/graphless-neural-networks
- Owner: snap-research
- License: mit
- Created: 2021-10-27T23:25:11.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-01-13T02:55:47.000Z (almost 2 years ago)
- Last Synced: 2023-07-31T14:18:40.000Z (over 1 year ago)
- Topics: deep-learning, distillation, efficient-inference, gnn, graph-algorithm, graph-neural-networks, knowledge-distillation, pytorch, scalability
- Language: Python
- Homepage:
- Size: 643 KB
- Stars: 65
- Watchers: 7
- Forks: 17
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Graph-less Neural Networks (GLNN)
Code for [Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation](https://arxiv.org/pdf/2110.08727.pdf) by [Shichang Zhang](https://shichangzh.github.io/), [Yozen Liu](https://research.snap.com/team/yozen-liu/), [Yizhou Sun](http://web.cs.ucla.edu/~yzsun/), and [Neil Shah](http://nshah.net/).
## Overview
### Distillation framework
### Accuracy vs. inference time on the `ogbn-products` dataset
## Getting Started
### Setup Environment
We use conda for environment setup. You can use
`bash ./prepare_env.sh`
which will create a conda environment named `glnn` and install relevant requirements (from `requirements.txt`). For simplicity, we use CPU-based `torch` and `dgl` versions in this guide, as specified in requirements. To run experiments with CUDA, please install `torch` and `dgl` with proper CUDA support, remove them from `requirements.txt`, and properly set the `--device` argument in the scripts. See https://pytorch.org/ and https://www.dgl.ai/pages/start.html for more installation details.
Be sure to activate the environment with
`conda activate glnn`
before running experiments as described below.
### Preparing datasets
To run experiments for dataset used in the paper, please download from the following links and put them under `data/` (see below for instructions on organizing the datasets).
- *CPF data* (`cora`, `citeseer`, `pubmed`, `a-computer`, and `a-photo`): Download the '.npz' files from [here](https://github.com/BUPT-GAMMA/CPF/tree/master/data/npz). Rename `amazon_electronics_computers.npz` and `amazon_electronics_photo.npz` to `a-computer.npz` and `a-photo.npz` respectively.
- *OGB data* (`ogbn-arxiv` and `ogbn-products`): Datasets will be automatically downloaded when running the `load_data` function in `dataloader.py`. More details [here](https://ogb.stanford.edu/).
- *BGNN data* (`house_class` and `vk_class`): Follow the instructions [here](https://github.com/dmlc/dgl/tree/473d5e0a4c4e4735f1c9dc9d783e0374328cca9a/examples/pytorch/bgnn) and download dataset pre-processed in DGL format from [here](https://www.dropbox.com/s/verx1evkykzli88/datasets.zip).
- *NonHom data* (`penn94` and `pokec`): Follow the instructions [here](https://github.com/CUAI/Non-Homophily-Benchmarks) to download the `penn94` dataset and its splits. The `pokec` dataset will be automatically downloaded when running the `load_data` function in `dataloader.py`.
- Your favourite datasets: download and add to the `load_data` function in `dataloader.py`.
### Usage
To quickly train a teacher model you can run `train_teacher.py` by specifying the experiment setting, i.e. transductive (`tran`) or inductive (`ind`), teacher model, e.g. `GCN`, and dataset, e.g. `cora`, as per the example below.
```
python train_teacher.py --exp_setting tran --teacher GCN --dataset cora
```
To quickly train a student model with a pretrained teacher you can run `train_student.py` by specifying the experiment setting, teacher model, student model, and dataset like the example below. Make sure you train the teacher using the `train_teacher.py` first and have its result stored in the correct path specified by `--out_t_path`.
```
python train_student.py --exp_setting ind --teacher SAGE --student MLP --dataset citeseer --out_t_path outputs
```
For more examples, and to reproduce results in the paper, please refer to scripts in `experiments/` as below.
```
bash experiments/sage_cpf.sh
```
To extend GLNN to your own model, you may do one of the following.
- Add your favourite model architectures to the `Model` class in `model.py`. Then follow the examples above.
- Train teacher model and store its output (log-probabilities). Then train the student by `train_student.py` with the correct `--out_t_path`.
## Results
GraphSAGE vs. MLP vs. GLNN under the production setting described in the paper (transductive and inductive combined). Delta_MLP (Delta_GNN) represents difference between the GLNN and the MLP (GNN). Results show classification accuracy (higher is better); Delta_GNN > 0 indicates GLNN outperforms GNN. We observe that GLNNs always improve from MLPs by large margins and achieve competitive results as GNN on 6/7 datasets. Please see Table 3 in the paper for more details.
| Datasets | GNN(SAGE) | MLP | GLNN | Delta_MLP | Delta_GNN |
|------------|----------------|--------------|----------------|-----------------|-------------------|
| Cora | **79.29** | 58.98 | 78.28 | 19.30 (32.72\%) | -1.01 (-1.28\%) |
| Citseer | 68.38 | 59.81 | **69.27** | 9.46 (15.82\%) | 0.89 (1.30\%) |
| Pubmed | **74.88** | 66.80 | 74.71 | 7.91 (11.83\%) | -0.17 (-0.22\%) |
| A-computer | 82.14 | 67.38 | **82.29** | 14.90 (22.12\%) | 0.15 (0.19\%) |
| A-photo | 91.08 | 79.25 | **92.38** | 13.13 (16.57\%) | 1.30 (1.42\%) |
| Arxiv | **70.73** | 55.30 | 65.09 | 9.79 (17.70\%) | -5.64 (-7.97\%) |
| Products | **76.60** | 63.72 | 75.77 | 12.05 (18.91\%) | -0.83 (-1.09\%) |
## Citation
If you find our work useful, please cite the following:
```BibTeX
@inproceedings{zhang2021graphless,
title={Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation},
author={Shichang Zhang and Yozen Liu and Yizhou Sun and Neil Shah},
booktitle={International Conference on Learning Representations}
year={2022},
url={https://arxiv.org/abs/2110.08727}
}
```
## Contact Us
Please open an issue or contact `[email protected]` if you have any questions.