{"id":13478619,"url":"https://github.com/jw9730/tokengt","last_synced_at":"2025-03-27T07:31:25.140Z","repository":{"id":43509118,"uuid":"508545365","full_name":"jw9730/tokengt","owner":"jw9730","description":"[NeurIPS'22] Tokenized Graph Transformer (TokenGT), in PyTorch","archived":false,"fork":false,"pushed_at":"2023-04-11T05:25:29.000Z","size":1287,"stargazers_count":316,"open_issues_count":11,"forks_count":45,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-10-30T11:42:22.582Z","etag":null,"topics":["equivariance","gnn","graph","hypergraph","pytorch","self-attention","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jw9730.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-06-29T04:30:26.000Z","updated_at":"2024-10-25T07:15:00.000Z","dependencies_parsed_at":"2024-01-16T07:22:16.365Z","dependency_job_id":"99505a29-e8b8-4afa-b026-201722db67ae","html_url":"https://github.com/jw9730/tokengt","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jw9730%2Ftokengt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jw9730%2Ftokengt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jw9730%2Ftokengt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jw9730%2Ftokengt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jw9730","download_url":"https://codeload.github.com/jw9730/tokengt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245802592,"owners_count":20674702,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["equivariance","gnn","graph","hypergraph","pytorch","self-attention","transformer"],"created_at":"2024-07-31T16:01:59.482Z","updated_at":"2025-03-27T07:31:22.821Z","avatar_url":"https://github.com/jw9730.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Tokenized Graph Transformer (PyTorch)\n\n[**Pure Transformers are Powerful Graph Learners**](https://arxiv.org/abs/2207.02505) \\\n[Jinwoo Kim](https://bit.ly/jinwoo-kim), [Tien Dat Nguyen](https://github.com/tiendatnguyen-vision), [Seonwoo Min](https://scholar.google.co.kr/citations?user=dWKk68wAAAAJ\u0026hl=en), [Sungjun Cho](https://scholar.google.com/citations?user=bEilQPMAAAAJ\u0026hl=en), [Moontae Lee](https://moontae.people.uic.edu/), [Honglak Lee](https://web.eecs.umich.edu/~honglak/), [Seunghoon Hong](https://maga33.github.io/) \\\nNeurIPS 2022\n\n![image-tokengt](./tokengt.png)\n\n## Setting up experiments\nUsing the provided Docker image (recommended)\n```bash\ndocker pull jw9730/tokengt:latest\ndocker run -it --gpus=all --ipc=host --name=tokengt -v /home:/home jw9730/tokengt:latest bash\n# upon completion, you should be at /tokengt inside the container\n```\n\nUsing the provided ```Dockerfile```\n```bash\ngit clone --recursive https://github.com/jw9730/tokengt.git /tokengt\ncd tokengt\ndocker build --no-cache --tag tokengt:latest .\ndocker run -it --gpus all --ipc=host --name=tokengt -v /home:/home tokengt:latest bash\n# upon completion, you should be at /tokengt inside the container\n```\n\nUsing ```pip```\n```bash\nsudo apt-get update\nsudo apt-get install python3.9\ngit clone --recursive https://github.com/jw9730/tokengt.git tokengt\ncd tokengt\nbash install.sh\n```\n\n## Running experiments\n\nSynthetic second-order equivariant basis approximation\n```bash\ncd equivariant-basis-approximation/scripts\n\n# Train and save logs, ckpts, and attention maps (--save_display)\nbash [INPUT]-[NODE_IDENTIFIER]-[TYPE_IDENTIFIER].sh\n\n# Test and save attention maps (--save_display)\nbash [INPUT]-[NODE_IDENTIFIER]-[TYPE_IDENTIFIER]-test.sh\n\n# For the visualization of saved attention maps, please see viz_multi.ipynb\n```\n\nPCQM4Mv2 large-scale graph regression\n```bash\ncd large-scale-regression/scripts\n\n# TokenGT (ORF)\nbash pcqv2-orf.sh\n\n# TokenGT (Lap)\nbash pcqv2-lap.sh\n\n# TokenGT (Lap) + Performer\nbash pcqv2-lap-performer-finetune.sh\n\n# TokenGT (ablated)\nbash pcqv2-ablated.sh\n\n# Attention distance plot for TokenGT (ORF)\nbash visualize-pcqv2-orf.sh\n\n# Attention distance plot for TokenGT (Lap)\nbash visualize-pcqv2-lap.sh\n```\n\n## Pre-Trained Models\nWe provide checkpoints of ```TokenGT (ORF)``` and ```TokenGT (Lap)```, both trained with PCQM4Mv2.\nPlease download ```ckpts.zip``` from [this link](https://drive.google.com/drive/folders/1mo0dV-aLxGFWbPF8xfE8phWTmOtIV1HG?usp=sharing).\nThen, unzip ```ckpts``` and place it in the ```large-scale-regression/scripts``` directory, so that each trained checkpoint is located at ```large-scale-regression/scripts/ckpts/pcqv2-tokengt-[NODE_IDENTIFIER]-trained/checkpoint_best.pt```.\nAfter that, you can resume the training from these checkpoints by adding the option ```--pretrained-model-name pcqv2-tokengt-[NODE_IDENTIFIER]-trained``` to the training scripts.\n\n## References\nOur implementation uses code from the following repositories:\n- [Performer](https://github.com/lucidrains/performer-pytorch.git) for FAVOR+ attention kernel\n- [Graph Transformer](https://github.com/graphdeeplearning/graphtransformer.git), [SignNet](https://github.com/cptq/SignNet-BasisNet.git), and [SAN](https://github.com/DevinKreuzer/SAN/blob/main/data/molecules.py) for Laplacian eigenvectors\n- [Graphormer](https://github.com/microsoft/Graphormer.git) for PCQM4Mv2 experiment pipeline\n- [timm](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/drop.py) for stochastic depth regularization\n\n## Citation\nIf you find our work useful, please consider citing it:\n\n```bib\n@article{kim2022pure,\n  author    = {Jinwoo Kim and Tien Dat Nguyen and Seonwoo Min and Sungjun Cho and Moontae Lee and Honglak Lee and Seunghoon Hong},\n  title     = {Pure Transformers are Powerful Graph Learners},\n  journal   = {arXiv},\n  volume    = {abs/2207.02505},\n  year      = {2022},\n  url       = {https://arxiv.org/abs/2207.02505}\n}\n```\n\n## Acknowledgements\nThe development of this open-sourced code was supported in part by the National Research Foundation of Korea (NRF) (No. 2021R1A4A3032834).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjw9730%2Ftokengt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjw9730%2Ftokengt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjw9730%2Ftokengt/lists"}