{"id":13442894,"url":"https://github.com/elicassion/3DTRL","last_synced_at":"2025-03-20T15:31:29.573Z","repository":{"id":38837472,"uuid":"496874259","full_name":"elicassion/3DTRL","owner":"elicassion","description":"Code for NeurIPS 2022 paper \"Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space\"","archived":false,"fork":false,"pushed_at":"2023-04-20T14:56:19.000Z","size":25437,"stargazers_count":18,"open_issues_count":2,"forks_count":0,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-08-01T03:42:19.984Z","etag":null,"topics":["3d-models","action-recognition","deep-learning","image-classification","pytorch","video-alignment"],"latest_commit_sha":null,"homepage":"https://elicassion.github.io/3dtrl/3dtrl.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elicassion.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-27T05:38:00.000Z","updated_at":"2024-06-26T17:04:12.000Z","dependencies_parsed_at":"2022-09-11T19:50:14.298Z","dependency_job_id":null,"html_url":"https://github.com/elicassion/3DTRL","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elicassion%2F3DTRL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elicassion%2F3DTRL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elicassion%2F3DTRL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elicassion%2F3DTRL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elicassion","download_url":"https://codeload.github.com/elicassion/3DTRL/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221772594,"owners_count":16878134,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-models","action-recognition","deep-learning","image-classification","pytorch","video-alignment"],"created_at":"2024-07-31T03:01:52.837Z","updated_at":"2025-03-20T15:31:29.561Z","avatar_url":"https://github.com/elicassion.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space\nby [Jinghuan Shang](https://www3.cs.stonybrook.edu/~jishang/), [Srijan Das](https://srijandas07.github.io/) and [Michael S. Ryoo](http://michaelryoo.com/) at NeurIPS 2022\n\nWe present 3DTRL, a plug-and play layer in Transformer using 3D camera transformations to recover tokens in 3D that learns viewpoint-agnostic representations.\nCheck our [paper](https://arxiv.org/abs/2206.11895) and [project page](https://www3.cs.stonybrook.edu/~jishang/3dtrl/3dtrl.html) for more details.\n\nQuick link: [[Usage]](#usage) [[Dataset]](#ftpv-dataset) [[Image Classification]](#image-classification) [[Action Recognition]](#action-recognition) [[Video Alignment]](#video-alignment)\n\nBy 3DTRL, we can align videos from multiple viewpoints, even including ego-centric view and third-person view videos.\n| \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; Third-person view \u0026nbsp;\t\u0026nbsp; \u0026nbsp; \u0026nbsp;| \u0026nbsp; \u0026nbsp; First-person view GT \u0026nbsp; \u0026nbsp; |   \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;   Ours   \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;  |   \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; DeiT+TCN  \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; |\n| ----------------- | -------------------- | ----------- | ----------- |\n\n![Multi-view Video Alignment Results](_doc/3dtrl_can_mh.gif)\n\n3DTRL recovers pseudo-depth of images -- getting semantically meaningful results.\n![Pseudo-depth](_doc/pseudo_depth_demo2.gif)\n\nOverview of 3DTRL\n![3DTRL](_doc/overview_white.png)\n\n## Usage\n\n### Directory Structure\n\n```\n├── _doc                            # images, gifs, etc for readme\n├── action_recognition              # all files related to action recognition go here, this can work stand alone\n    ├── configs                     # config files for TimeSformer and +3DTRL\n    ├── timesformer\n        ├── datasets                # data pipeline for action recognition\n        ├── models                  # definitions of TimeSformer and +3DTRL\n    ├── script.sh                   # launch script for action recognition\n├\n├── backbone                        # modules used by 3DTRL (depth and camera estimators)\n├── model                           # Transformer models with 3DTRL plug-in (ViT, Swin, TnT)\n├── data_pipeline                   # dataset class for video alignment\n├── i1k_configs                     # Configuration files for ImageNet-1K training\n├\n├── 3dtrl_env.yml                   # conda env for image classification and video alignment\n├── i1k.sh                          # launch script for ImageNet-1K jobs\n├── imagenet_train.py               # entry point of ImageNet-1K training\n├── imagenet_val.py                 # entry point of ImageNet-1K evaluation\n├── multiview_video_alignment.py    # entry point of video alignment\n├── utils.py                        # some utility functions\n```\n\n### Image Classification\nEnvironment:\n```\nconda env create -f 3dtrl_env.yml\n```\n\nRun:\n```\nconda activate 3dtrl\nbash i1k.sh num_gpu your_imagenet_dir\n```\n\nCredit: We build our code for image classification on top of [timm](https://github.com/rwightman/pytorch-image-models).\n\n### Video Alignment\n#### FTPV Dataset\nWe release the First-Third Person View (FTPV) dataset (including MC, Panda, Lift, and Can used in our paper) at [Google Drive](https://drive.google.com/file/d/14chFXCi74rmd086-QPoAbOcRA-sGcwXn/view?usp=share_link). Download and unzip it. Please consider [cite](#cite-3dtrl) our paper if you use the datasets. Note: I also include Pouring dataset introduced by [TCN paper](https://arxiv.org/pdf/1704.06888.pdf) in the drive. The reason is that I got a hard time to find a valid source to download it when doing my research. I'm re-sharing it for your convenience. Please cite them if you use Pouring.\n\nEnvironment:\n```\nconda env create -f 3dtrl_env.yml\n```\n\nRun:\n```\nconda activate 3dtrl\npython multiview_video_alignment.py --data dataset_name [--model vit_3dtrl] [--train_videos num_video_used]\n```\n\n\n### Action Recognition\nEnvironment: we follow [TimeSformer](https://github.com/facebookresearch/TimeSformer) to set up the virtual environment. Then,\n```\ncd action_recognition\nbash script.sh your_config_file data_location log_location\n```\n\n\n## Cite 3DTRL\nIf you find our research useful, please consider cite:\n```\n@inproceedings{\n    3dtrl,\n    title={Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space},\n    author={Jinghuan Shang and Srijan Das and Michael S Ryoo},\n    booktitle={Advances in Neural Information Processing Systems},\n    year={2022},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felicassion%2F3DTRL","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felicassion%2F3DTRL","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felicassion%2F3DTRL/lists"}