https://github.com/autonomousvision/gta
[ICLR'24] GTA: A Geometry-Aware Attention Mechanism for Multi-view Transformers
https://github.com/autonomousvision/gta
Last synced: 12 months ago
JSON representation
[ICLR'24] GTA: A Geometry-Aware Attention Mechanism for Multi-view Transformers
- Host: GitHub
- URL: https://github.com/autonomousvision/gta
- Owner: autonomousvision
- License: mit
- Created: 2023-10-16T12:49:17.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-06-14T06:36:13.000Z (about 2 years ago)
- Last Synced: 2024-12-08T01:30:39.486Z (over 1 year ago)
- Language: Python
- Homepage:
- Size: 61.2 MB
- Stars: 120
- Watchers: 14
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Geometric Transform Attention
Takeru Miyato
·
Bernhard Jaeger
·
Max Welling
·
Andreas Geiger
OpenReview | arXiv | Project Page
Official reproducing code of our ICLR2024 work: "GTA: A Geometry-Aware Attention Mechanism for Multi-view Transformers", a simple way to make your multi-view transformer more expressive!
(3/15/2024): The GTA mechanism is also effective for image generation, which is a purely 2D task. You can find the experimental details in our camera-ready paper and the implementation at this [branch](https://github.com/autonomousvision/gta/tree/DiT).
## Contents
This repository contains the following different codebases, each of which can be accessed by switching to the corresponding branch:
- NVS experiments on CLEVR-TR and MSN-Hard (this branch)
- NVS experiments on ACID and RealEstate ([link](https://github.com/autonomousvision/gta/tree/crsrndr))
- ImageNet generation with Diffusion transformers (DiT) ([link](https://github.com/autonomousvision/gta/tree/DiT))
You can find the code of GTA for multi-view ViTs
[here](https://github.com/autonomousvision/gta/blob/main/source/utils/gta.py) and for image ViTs [here](https://github.com/autonomousvision/gta/blob/DiT/gta.py#L69).
Please feel free to reach out to us if you have any questions!
# Setup
## 1. Create env and install python libraries
```
conda create -n gta python=3.9
conda activate gta
pip3 install -r requirements.txt
```
## 2. Download dataset
```
export DATADIR=
mkdir -p $DATADIR
```
### CLEVR-TR
Download the dataset from this [link](https://drive.google.com/file/d/1iT3LjOPm1etcLKs7nVoHhYWU6qR1PdRG/view?usp=drive_link) and place it under `$DATADIR`
### MultiShapeNet Hard (MSN-Hard)
```
gsutil -m cp -r gs://kubric-public/tfds/kubric_frames/multi_shapenet_conditional/2.8.0/ ${DATADIR}/multi_shapenet_frames/
```
## *Pretrained models (MSN-Hard pre-trained models will be uploaded soon)
- CLEVR-TR: [link](https://drive.google.com/drive/folders/1YPhMpvrVFOgJhCMuyUQqrUx2YC25oLlh)
- MSN-Hard: [link](https://drive.google.com/drive/folders/1HJUfXPnslRvDo2hez2GJ8tFh_O7Lk5iL)
# Training
### CLEVR-TR
```
torchrun --standalone --nnodes 1 --nproc_per_node 4 train.py runs/clevrtr/GTA/gta/config.yaml ${DATADIR}/clevrtr --seed=0
```
### MSN-Hard
```
torchrun --standalone --nnodes 1 --nproc_per_node 4 train.py runs/msn/GTA/gta_so3/config.yaml ${DATADIR} --seed=0
```
# Evaluation of PSNR, SSIM and LPIPS
```
python evaluate.py runs/clevrtr/GTA/gta/config.yaml ${DATADIR}/clevrtr $PATH_TO_CHECKPOINT # CLEVR-TR
python evaluate.py runs/msn/GTA/gta_so3/config.yaml ${DATADIR} $PATH_TO_CHECKPOINT # MSN-Hard
```
# Acknowledgements
This repository is built on top of [SRT](https://github.com/stelzner/srt) and [OSRT](https://github.com/stelzner/osrt) created by @stelzner. We would like to thank him for his open-source contribution of the SRT models.
We also thank @lucidrains for providing the values of J matrices, which are needed to compute the irreps of SO(3) efficiently.
# Citation
```bibtex
@inproceedings{Miyato2024GTA,
title={GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers},
author={Miyato,Takeru and Jaeger, Bernhard and Welling, Max and Geiger, Andreas},
booktitle={International Conference on Learning Representations (ICLR)},
year={2024}
}
```