{"id":16515429,"url":"https://github.com/njmarko/graph-transformer-psiml","last_synced_at":"2025-10-28T04:33:06.110Z","repository":{"id":53880823,"uuid":"520968040","full_name":"njmarko/graph-transformer-psiml","owner":"njmarko","description":"Transformer implemented with graph attention network (GAT) layers from PyTorch Geometric","archived":false,"fork":false,"pushed_at":"2022-08-14T17:47:37.000Z","size":43018,"stargazers_count":17,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-01T11:41:26.619Z","etag":null,"topics":["attention","gat","gnn","graph-neural-networks","pytorch-geometric","transformer","vision-transformer","vit"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/njmarko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-08-03T17:11:50.000Z","updated_at":"2024-11-06T14:12:29.000Z","dependencies_parsed_at":"2022-08-13T03:11:21.360Z","dependency_job_id":null,"html_url":"https://github.com/njmarko/graph-transformer-psiml","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njmarko%2Fgraph-transformer-psiml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njmarko%2Fgraph-transformer-psiml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njmarko%2Fgraph-transformer-psiml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njmarko%2Fgraph-transformer-psiml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/njmarko","download_url":"https://codeload.github.com/njmarko/graph-transformer-psiml/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238597388,"owners_count":19498396,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","gat","gnn","graph-neural-networks","pytorch-geometric","transformer","vision-transformer","vit"],"created_at":"2024-10-11T16:17:07.217Z","updated_at":"2025-10-28T04:32:55.798Z","avatar_url":"https://github.com/njmarko.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# graph-transformer-psiml\nTransformer implemented with graph neural network attention layer from Pytorch Geometric. This was a project for [PSIML](https://psiml.petlja.org/), Practical Seminar for Machine Learning organized by PFE, Petlja, Everseen, and Microsoft in Belgrade 2022.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/34657562/184308361-554b6ce6-5cac-4f99-94c0-66bb48864d69.png\" align=\"center\" width=\"50%\"\u003e\n\u003c/div\u003e\n\n## Authors\n\n- Marina Debogović (ETF)\n- Marko Njegomir (FTN)\n\n## Mentors\n- Anđela Donević (Everseen)\n- Nikola Popović (ETH Zurich)\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/34657562/184306183-802cb780-29ce-4fed-95b6-82023b199354.png\"\u003e\n  \u003cp align=\"center\"\u003eIllustration 1 - Transformer with graph attention network (DALLE-2).\u003c/p\u003e\n\u003c/div\u003e\n\n# Architecture\n\n- The attention layer in ViT Encoder is replaced with GATv2 (Graph Attention network).\n  - Inputs for the GATv2 must be a single graph and an adjacency list.\n      - To support batches, a disjoint union of graphs in the batch is created, so we get a single graph.\n  - Output dim from the GATv2 is multiplied by the number of heads\n      - A new layer is added that reduces the output dim to the input dimensions so the layers can be stacked.\n- GATv2 layers can easily be replaced with any other GNN layer in Pytorch Geometric.\n  - For some specific layers that take more than just vertices and edges some tweaks to the inputs and outputs might be necessary.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"images/graph_transformer_encoder.png\"\u003e\n  \u003cp align=\"center\"\u003eIllustration 2 - Attention layer in Vision Transformer's Encoder is replaced with Graph Attention Network.\u003c/p\u003e\n\u003c/div\u003e\n\n# Results\n\n- Trained and tested on VM with a single V100 GPU\n- Due to time and hardware constraints, models were compared on MNIST and CIFAR10\n- There were no pre-trained models on Imagenet with this architecture available, so no transfer learning was possible.\n  - Training the model on Imagenet first and then finetuning to some other specific task might improve performance.\n\n## MNIST\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"images/mnist-train-loss.png\"\u003e\n  \u003cp align=\"center\"\u003eIllustration 3 - MNIST train loss for Classic ViT and our Graph Transformer.\u003c/p\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"images/mnist-train-acc.png\"\u003e\n  \u003cp align=\"center\"\u003eIllustration 4 - MNIST train accuracy for Classic ViT and our Graph Transformer.\u003c/p\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"images/mnist-val-acc.png\"\u003e\n  \u003cp align=\"center\"\u003eIllustration 5 - MNIST validation accuracy for Classic ViT and our Graph Transformer.\u003c/p\u003e\n\u003c/div\u003e\n\n## CIFAR10\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"images/cifar10-train-loss.png\"\u003e\n  \u003cp align=\"center\"\u003eIllustration 6 - CIFAR10 train loss for Classic ViT and our Graph Transformer.\u003c/p\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"images/cifar10-train-acc.png\"\u003e\n  \u003cp align=\"center\"\u003eIllustration 7 - CIFAR10 train accuracy for Classic ViT and our Graph Transformer.\u003c/p\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"images/cifar10-val-acc.png\"\u003e\n  \u003cp align=\"center\"\u003eIllustration 8 - CIFAR10 validation accuracy for Classic ViT and our Graph Transformer.\u003c/p\u003e\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjmarko%2Fgraph-transformer-psiml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnjmarko%2Fgraph-transformer-psiml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjmarko%2Fgraph-transformer-psiml/lists"}