{"id":13478565,"url":"https://github.com/jw9730/hot","last_synced_at":"2025-03-27T07:31:22.883Z","repository":{"id":44146009,"uuid":"421772270","full_name":"jw9730/hot","owner":"jw9730","description":"[NeurIPS'21] Higher-order Transformers for sets, graphs, and hypergraphs, in PyTorch","archived":false,"fork":false,"pushed_at":"2022-11-28T21:30:23.000Z","size":2048,"stargazers_count":60,"open_issues_count":1,"forks_count":8,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-30T11:42:21.708Z","etag":null,"topics":["equivariance","gnn","graph","hypergraph","pytorch","self-attention","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jw9730.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-27T10:24:27.000Z","updated_at":"2024-08-01T03:16:17.000Z","dependencies_parsed_at":"2023-01-22T12:01:06.008Z","dependency_job_id":null,"html_url":"https://github.com/jw9730/hot","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jw9730%2Fhot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jw9730%2Fhot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jw9730%2Fhot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jw9730%2Fhot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jw9730","download_url":"https://codeload.github.com/jw9730/hot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245802563,"owners_count":20674695,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["equivariance","gnn","graph","hypergraph","pytorch","self-attention","transformer"],"created_at":"2024-07-31T16:01:58.779Z","updated_at":"2025-03-27T07:31:19.873Z","avatar_url":"https://github.com/jw9730.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Higher-order Transformer (PyTorch)\n\n[**Transformers Generalize DeepSets and Can be Extended to Graphs and Hypergraphs**](https://arxiv.org/abs/2110.14416) \\\n[Jinwoo Kim](https://bit.ly/jinwoo-kim), [Saeyoon Oh](https://github.com/saeyoon17), [Seunghoon Hong](https://maga33.github.io/) @ [KAIST](https://www.kaist.ac.kr/en/) \\\nNeurIPS 2021\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/transformers-generalize-deepsets-and-can-be/graph-regression-on-pcqm4m-lsc)](https://paperswithcode.com/sota/graph-regression-on-pcqm4m-lsc?p=transformers-generalize-deepsets-and-can-be)\n\n***We present a generalization of Transformers to sets, graphs, and hypergraphs, and reduce its computational cost to linear to input size.***\n- Powerful operations, involving both local and global interactions over input elements\n- Translation between different-order graphs (e.g., set-to-graph, graph-to-set, graph-to-vector)\n- Theoretically and empirically stronger than MPNNs, even with the same linear complexity\n- Works well on large datasets!\n\n![image-second-order](./docs/second-order-transformer.png)\n\nIn this repository, we provide the PyTorch implementation of:\n- Up-to-second-order Transformers: {set, graph} input × {set, graph, vector} output\n- A space of their variants: {dense, sparse} data × {softmax, kernel} attention\n- A special extension for higher-order hyperedge prediction: set input, *k*-hyperedges output\n- Corresponding equivariant linear layers as baselines\n\nIn practice, we recommend to use the ***sparse kernel variant*** as it scales linearly to data size while still powerful.\n\n## How do I use Higher-order Transformers for my project?\nGo to your repository and execute the following.\n```bash\n# this is enough\npip install torch\u003e=1.8.1\n\n# add this repository as a submodule\ngit submodule add https://github.com/jw9730/hot hot\ngit submodule init\n\n# run basic tests\ncd hot\npython3 run_tests.py\ncd ..\n```\nFor basic use cases, please see `run_perf_tests.py` and `run_tests.py`.\n\nTo update the code, execute the following:\n```bash\ngit submodule update --remote --merge\n```\n\n## Setting up experiments\nUsing Docker\n```bash\ngit clone https://github.com/jw9730/hot.git hot\ncd hot\ndocker build --no-cache --tag hot:latest .\ndocker run -it --gpus all --ipc=host --name=hot -v /home:/home hot:latest bash\n# upon completion, you should be at /hot inside the container\n```\n\nUsing pip\n```bash\ngit clone https://github.com/jw9730/hot.git hot\ncd hot\npip3 install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html\npip install -r requirements.txt\npip install torch-sparse==0.6.9 torch-scatter==2.0.6 -f https://data.pyg.org/whl/torch-1.8.0+cu111.html\npip install torch-geometric==1.6.3\n```\n\n## Running experiments\nRuntime and memory analysis\n```bash\npython3 run_perf_tests.py\n```\n\nPCQM4M-LSC graph regression\n```bash\ncd regression/examples/pcqm4m-lsc\n\n# Second-order Transformers (sparse kernel)\nbash enc.sh\nbash enc-small.sh\n\n# Comparison with more baselines\nbash enc-short.sh\nbash enc-small-short.sh\n\n# Second-order MLP\nbash mlp-short.sh\n\n# Vanilla Transformer + Laplacian PE\nbash laplacian-short.sh\n```\nSet-to-graph prediction\n```bash\ncd set-to-graph/main\npython download_jets_data.py\ncd ../examples\n\n# Delaunay (50)\ncd delaunay-a\nbash enc-kernel.sh\nbash enc-softmax.sh\nbash s2g.sh\n# visualize\nbash visualize-kernel.sh\nbash visualize-softmax.sh\n\n# Delaunay (20-80)\ncd ../delaunay-b\nbash enc-kernel.sh\nbash enc-softmax.sh\nbash s2g.sh\n# visualize\nbash visualize-kernel.sh\nbash visualize-softmax.sh\n\n# Jets\ncd ../jets\nbash enc-kernel.sh\nbash enc-softmax.sh\n# test\nbash test-enc-kernel.sh\nbash test-enc-softmax.sh\n```\n*k*-uniform hyperedge prediction\n```bash\ncd k-uniform-hyperedge/examples\n\n# GPS\ncd gps\nbash enc.sh\nbash s2g+.sh\nbash hyper-sagnn-e.sh\nbash hyper-sagnn-w.sh\n\n# MovieLens\ncd ../movielens\nbash enc.sh\nbash s2g+.sh\nbash hyper-sagnn-e.sh\nbash hyper-sagnn-w.sh\n\n# Drug\ncd ../drug\nbash enc.sh\nbash s2g+.sh\nbash hyper-sagnn-e.sh\nbash hyper-sagnn-w.sh\n```\n\n## References\nThis implementation uses code from the following repositories:\n- [Performer](https://github.com/lucidrains/performer-pytorch.git) for FAVOR+ attention kernel\n- [Graph Transformer](https://github.com/graphdeeplearning/graphtransformer.git) for Laplacian PE of baseline model\n- [Graphormer](https://github.com/microsoft/Graphormer.git) for graph regression experiment\n- [Set2Graph](https://github.com/hadarser/SetToGraphPaper.git) for set-to-graph prediction experiment\n- [Hyper-SAGNN](https://github.com/ma-compbio/Hyper-SAGNN.git) for k-uniform hyperedge prediction experiment\n\n## Citation\nIf you find our work useful, please consider citing it:\n```bib\n@article{kim2021transformers,\n  author    = {Jinwoo Kim and Saeyoon Oh and Seunghoon Hong},\n  title     = {Transformers Generalize DeepSets and Can be Extended to Graphs and Hypergraphs},\n  booktitle = {2021 Conference on Neural Information Processing Systems (NeurIPS)},\n  year      = {2021}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjw9730%2Fhot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjw9730%2Fhot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjw9730%2Fhot/lists"}