{"id":18600788,"url":"https://github.com/autonomousvision/gta","last_synced_at":"2025-08-15T17:31:56.541Z","repository":{"id":200498412,"uuid":"705656076","full_name":"autonomousvision/gta","owner":"autonomousvision","description":"[ICLR'24] GTA: A Geometry-Aware Attention Mechanism for Multi-view Transformers","archived":false,"fork":false,"pushed_at":"2024-06-14T06:36:13.000Z","size":64217,"stargazers_count":120,"open_issues_count":1,"forks_count":2,"subscribers_count":14,"default_branch":"main","last_synced_at":"2024-12-08T01:30:39.486Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/autonomousvision.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-16T12:49:17.000Z","updated_at":"2024-11-06T18:45:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"250cf04e-a839-4d55-af16-a8a6b6391b74","html_url":"https://github.com/autonomousvision/gta","commit_stats":null,"previous_names":["autonomousvision/gta"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fgta","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fgta/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fgta/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fgta/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/autonomousvision","download_url":"https://codeload.github.com/autonomousvision/gta/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229934971,"owners_count":18147144,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T02:05:29.873Z","updated_at":"2024-12-16T09:20:15.926Z","avatar_url":"https://github.com/autonomousvision.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\n  \u003ch1 align=\"center\"\u003eGeometric Transform Attention\u003c/h1\u003e\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://takerum.github.io/\"\u003eTakeru Miyato\u003c/a\u003e\n    ·\n    \u003ca href=\"https://kait0.github.io/\"\u003eBernhard Jaeger\u003c/a\u003e\n    ·\n    \u003ca href=\"https://staff.fnwi.uva.nl/m.welling/\"\u003eMax Welling\u003c/a\u003e\n    ·\n    \u003ca href=\"https://www.cvlibs.net/\"\u003eAndreas Geiger\u003c/a\u003e\n\n  \u003c/p\u003e\n  \u003ch3 align=\"center\"\u003e\u003ca href=\"https://openreview.net/forum?id=uJVHygNeSZ\"\u003eOpenReview\u003c/a\u003e | \u003ca href=\"https://arxiv.org/abs/2310.10375\"\u003earXiv\u003c/a\u003e  | \u003ca href=\"https://takerum.github.io/gta/\"\u003eProject Page\u003c/a\u003e \u003c/h3\u003e\n\n\n  \u003cp align=\"center\"\u003e\n  \u003cimg width=\"180\" alt=\"gta_mech\" src=\"https://github.com/autonomousvision/gta/assets/11573649/939bc44b-3fde-48b8-96ce-4f68c6c55ccd\"\u003e\n  \u003c/p\u003e\n\u003c/p\u003e\n\nOfficial reproducing code of our ICLR2024 work: \"GTA: A Geometry-Aware Attention Mechanism for Multi-view Transformers\", a simple way to make your multi-view transformer more expressive! \n\n(3/15/2024): The GTA mechanism is also effective for image generation, which is a purely 2D task. You can find the experimental details in our camera-ready paper and the implementation at this [branch](https://github.com/autonomousvision/gta/tree/DiT).\n\n\n## Contents\n\nThis repository contains the following different codebases, each of which can be accessed by switching to the corresponding branch:\n- NVS experiments on CLEVR-TR and MSN-Hard (this branch)\n- NVS experiments on ACID and RealEstate ([link](https://github.com/autonomousvision/gta/tree/crsrndr))\n- ImageNet generation with Diffusion transformers (DiT) ([link](https://github.com/autonomousvision/gta/tree/DiT))\n\n\nYou can find the code of GTA for multi-view ViTs\n[here](https://github.com/autonomousvision/gta/blob/main/source/utils/gta.py) and for image ViTs [here](https://github.com/autonomousvision/gta/blob/DiT/gta.py#L69). \n\nPlease feel free to reach out to us if you have any questions!\n\n# Setup\n\n## 1. Create env and install python libraries\n\n```\nconda create -n gta python=3.9\nconda activate gta\npip3 install -r requirements.txt\n```\n\n## 2. Download dataset\n\n```\nexport DATADIR=\u003cpath_to_datadir\u003e\nmkdir -p $DATADIR\n```\n\n### CLEVR-TR \n\nDownload the dataset from this [link](https://drive.google.com/file/d/1iT3LjOPm1etcLKs7nVoHhYWU6qR1PdRG/view?usp=drive_link) and place it under `$DATADIR`\n\u003cp align=\"center\"\u003e\n\u003cimg width=\"400\" alt=\"clevr1\" src=\"https://github.com/autonomousvision/gta/assets/11573649/0952c2f7-47e4-41c1-93a4-bad57a8df12e\"\u003e\n\u003cimg width=\"400\" alt=\"clevr2\" src=\"https://github.com/autonomousvision/gta/assets/11573649/1aca1f94-8b7e-42c8-9703-13f2dd60de38\"\u003e\n\u003c/p\u003e\n\n### MultiShapeNet Hard (MSN-Hard)\n```\ngsutil -m cp -r gs://kubric-public/tfds/kubric_frames/multi_shapenet_conditional/2.8.0/ ${DATADIR}/multi_shapenet_frames/\n```\n\u003cp align=\"center\"\u003e\n\u003cimg width=\"800\" alt=\"gta_mech\" src=\"https://github.com/autonomousvision/gta/assets/11573649/de09e2d9-1eb1-4833-981b-3ef11c1c5fa3\"\u003e\n\u003c/p\u003e\n\n## *Pretrained models (MSN-Hard pre-trained models will be uploaded soon)\n- CLEVR-TR: [link](https://drive.google.com/drive/folders/1YPhMpvrVFOgJhCMuyUQqrUx2YC25oLlh)\n- MSN-Hard: [link](https://drive.google.com/drive/folders/1HJUfXPnslRvDo2hez2GJ8tFh_O7Lk5iL) \n\n# Training\n\n### CLEVR-TR\n```\ntorchrun --standalone --nnodes 1 --nproc_per_node 4 train.py runs/clevrtr/GTA/gta/config.yaml  ${DATADIR}/clevrtr --seed=0 \n```\n\n### MSN-Hard\n```\ntorchrun --standalone --nnodes 1 --nproc_per_node 4 train.py runs/msn/GTA/gta_so3/config.yaml  ${DATADIR} --seed=0 \n```\n\n\n# Evaluation of PSNR, SSIM and LPIPS\n```\npython evaluate.py runs/clevrtr/GTA/gta/config.yaml ${DATADIR}/clevrtr $PATH_TO_CHECKPOINT # CLEVR-TR\npython evaluate.py runs/msn/GTA/gta_so3/config.yaml ${DATADIR} $PATH_TO_CHECKPOINT # MSN-Hard\n```\n\n# Acknowledgements\nThis repository is built on top of [SRT](https://github.com/stelzner/srt) and [OSRT](https://github.com/stelzner/osrt) created by @stelzner. We would like to thank him for his open-source contribution of the SRT models.\nWe also thank @lucidrains for providing the values of J matrices, which are needed to compute the irreps of SO(3) efficiently.\n \n\n\n# Citation\n```bibtex\n@inproceedings{Miyato2024GTA,\n    title={GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers},\n    author={Miyato,Takeru and Jaeger, Bernhard and Welling, Max and Geiger, Andreas},\n    booktitle={International Conference on Learning Representations (ICLR)},\n    year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fgta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautonomousvision%2Fgta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fgta/lists"}