{"id":21564829,"url":"https://github.com/zinengtang/perceiver_vl","last_synced_at":"2025-07-15T01:05:25.207Z","repository":{"id":63606168,"uuid":"469553799","full_name":"zinengtang/Perceiver_VL","owner":"zinengtang","description":"PyTorch code for \"Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention\" (WACV 2023)","archived":false,"fork":false,"pushed_at":"2023-02-05T12:44:38.000Z","size":1138,"stargazers_count":33,"open_issues_count":2,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-10T13:24:05.159Z","etag":null,"topics":["efficiency","retrieval","scalability","video-language","vision-and-language"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zinengtang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-14T02:25:39.000Z","updated_at":"2024-11-27T05:10:42.000Z","dependencies_parsed_at":"2023-02-18T23:31:14.986Z","dependency_job_id":null,"html_url":"https://github.com/zinengtang/Perceiver_VL","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zinengtang/Perceiver_VL","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zinengtang%2FPerceiver_VL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zinengtang%2FPerceiver_VL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zinengtang%2FPerceiver_VL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zinengtang%2FPerceiver_VL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zinengtang","download_url":"https://codeload.github.com/zinengtang/Perceiver_VL/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zinengtang%2FPerceiver_VL/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265382880,"owners_count":23756423,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["efficiency","retrieval","scalability","video-language","vision-and-language"],"created_at":"2024-11-24T10:17:21.050Z","updated_at":"2025-07-15T01:05:25.158Z","avatar_url":"https://github.com/zinengtang.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Perceiver-VL\n\n### **[Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention](https://arxiv.org/abs/2211.11701) [WACV 2023 [bib](https://github.com/zinengtang/Perceiver_VL#citation)]**  \n[Zineng Tang*](https://zinengtang.github.io/), [Jaemin Cho*](https://j-min.io/), [Jie Lei](https://jayleicn.github.io/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/)   \n\nLearning vision-language representation by iterative latent attention that scales with long inputs linearly.\n\n## Introduction\n\u003c!-- \u003cp align=\"center\"\u003e\n  \u003cbig\u003e\u003cb\u003ePerceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention (WACV 2023)\u003c/b\u003e\u003c/big\u003e\n\u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e\n  \u003cbig\u003e\u003cb\u003eZineng Tang*, Jaemin Cho*, Jie Lei, Mohit Bansal\u003c/b\u003e\u003c/big\u003e\n\u003c/p\u003e --\u003e\n\nPerceiver-VL Architecture Overview\n\n\u003cp align=\"center\"\u003e\n  \u003cimg align=\"middle\" width=\"800\" src=\"assets/architecture.png\"/\u003e\n\u003c/p\u003e\n\n\n## Install\n### Setup `python` environment\n```\nconda create -n Perceiver-VL python=3.8   # You can also use other environment.\n```\n\n### Install other dependencies\n```\npip install -r requirements.txt\n```\n\n\n## Training\n\nTODO: Finish datasets/tasks instructions and scripts\n\n### Pretraining (scripts)\n\n```\n# Pretrain on Webvid + GCC\nbash scripts/co_pretrain.sh\n```\n\n```\n# Pretrain on Webvid\nbash scripts/webvid_pretrain.sh\n```\n\n```\n# Pretrain on GCC\nbash scripts/gcc_pretrain.sh\n```\n\n```\n# Pretrain on ImageNet\nbash scripts/imagenet_pretrain.sh\n```\n\n### Pretrained Checkpoint\nDownload Checkpoint [[link]](https://huggingface.co/murgelab/PerceiverVL/resolve/main/perceivervl_mlm_itm_vtm.ckpt)\n\n### Finetuning on Downstream (scripts)\n\n```\n# Fintune on MSRVTT Retrieval\nbash scripts/msrvtt_vrtr_finetune.sh\n```\n\n```\n# Fintune on VQA\nbash scripts/vqa_finetune.sh\n```\n\n\n## Code Structure\n\n```\nPerceiver_VL\n│\n├── assets                          # illustrations                          \n│   └── architecture.png\n│\n├── model                           # main source       \n│   ├── datamodules                 # pytorch-lightning wrap\n│   │   ├── datamodule_base.py\n│   │   └── ...          \n│   └── datasets                    # Datasets\n│   │   ├── vqa_dataset.py     \n│   │   └── ...    \n│   ├── gadgets     \n│   │   └── my_metrics.py           # metric utils\n│   ├── modules                     \n│   │   ├── heads.py                # model heads\n│   │   ├── model_module.py         # pytorch-lightning wrap for model\n│   │   ├── model_utils.py          # pytorch-lightning wrap for training metrics\n│   │   ├── objectives.py           # pretraining/finetuning objectives\n│   │   └── perceiver_vl.py         # main model\n│   ├── transforms                  # image transformation utils\n│   │   └── ... \n│   └── config.py                   # all configurations\n│\n├── scripts                         # all scripts\n│   ├── vqa_finetune.sh \n│   ├── co_pretrain.sh\n│   └── ... \n│\n├── run.py                          # main\n└── requirements.txt                \n```\n\n\n## Citation\n```\n@inproceedings{tang2023wacv,\n  title     = {Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention},\n  author    = {Zineng Tang and Jaemin Cho and Jie Lei and Mohit Bansal},\n  booktitle = {WACV},\n  year      = {2023}\n}\n```\n\n## Acknowledgement\n\nOur codebase is based on [ViLT](https://github.com/dandelin/ViLT). \nWe thank the authors for their open-source contributions.\n\n## Contact\n\nZineng Tang (zn.tang.terran@gmail.com)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzinengtang%2Fperceiver_vl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzinengtang%2Fperceiver_vl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzinengtang%2Fperceiver_vl/lists"}