{"id":13958452,"url":"https://github.com/xyzforever/BEVT","last_synced_at":"2025-07-21T00:30:55.975Z","repository":{"id":37831695,"uuid":"465185914","full_name":"xyzforever/BEVT","owner":"xyzforever","description":"PyTorch implementation of BEVT (CVPR 2022) https://arxiv.org/abs/2112.01529","archived":false,"fork":false,"pushed_at":"2022-07-19T06:38:26.000Z","size":20132,"stargazers_count":158,"open_issues_count":3,"forks_count":19,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-11-28T02:34:43.941Z","etag":null,"topics":["action-recognition","bert","deep-learning","foundation-models","masked-autoencoder","pytorch","self-supervised-learning","video-representation-learning","video-understanding"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xyzforever.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-02T06:34:08.000Z","updated_at":"2024-11-05T14:39:34.000Z","dependencies_parsed_at":"2022-07-20T00:48:30.534Z","dependency_job_id":null,"html_url":"https://github.com/xyzforever/BEVT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/xyzforever/BEVT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xyzforever%2FBEVT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xyzforever%2FBEVT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xyzforever%2FBEVT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xyzforever%2FBEVT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xyzforever","download_url":"https://codeload.github.com/xyzforever/BEVT/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xyzforever%2FBEVT/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266221247,"owners_count":23894964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["action-recognition","bert","deep-learning","foundation-models","masked-autoencoder","pytorch","self-supervised-learning","video-representation-learning","video-understanding"],"created_at":"2024-08-08T13:01:36.286Z","updated_at":"2025-07-21T00:30:51.525Z","avatar_url":"https://github.com/xyzforever.png","language":"Python","funding_links":[],"categories":["其他_机器视觉","MIM for CV Downstream Tasks"],"sub_categories":["网络服务_其他","Video Rrepresentation"],"readme":"# BEVT: BERT Pretraining of Video Transformers\r\n\r\nRui Wang\u003csup\u003e1\u003c/sup\u003e, Dongdong Chen\u003csup\u003e2\u003c/sup\u003e, Zuxuan Wu\u003csup\u003e1\u003c/sup\u003e, Yinpeng Chen\u003csup\u003e2\u003c/sup\u003e, Xiyang Dai\u003csup\u003e2\u003c/sup\u003e, Mengchen Liu\u003csup\u003e2\u003c/sup\u003e, Yu-Gang Jiang\u003csup\u003e1\u003c/sup\u003e, Luowei Zhou\u003csup\u003e2\u003c/sup\u003e, Lu Yuan\u003csup\u003e2\u003c/sup\u003e \u003cbr\u003e\r\n\u003csup\u003e1\u003c/sup\u003eShanghai Key Lab of Intelligent Info. Processing, School of Computer Science, Fudan University, \u003csup\u003e2\u003c/sup\u003eMicrosoft Cloud + AI\r\n\r\n\u003e This repository hosts the official PyTorch implementation of the paper: \"[**BEVT: BERT Pretraining of Video Transformers**](https://arxiv.org/abs/2112.01529)\".\r\n\r\n## Abstract\r\n\r\nThis paper studies the BERT pretraining of video transformers. It is a straightforward but worth-studying extension given the recent success from BERT pretraining of image transformers. We introduce BEVT which decouples video representation learning into spatial representation learning and temporal dynamics learning. In particular, BEVT first performs masked image modeling on image data, and then conducts masked image modeling jointly with masked video modeling on video data. This design is motivated by two observations: 1) transformers learned on image datasets provide decent spatial priors that can ease the learning of video transformers, which are often times computationally-intensive if trained from scratch; 2) discriminative clues, i.e., spatial and temporal information, needed to make correct predictions vary among different videos  due to large intra-class and inter-class variations. We conduct extensive experiments on three challenging video benchmarks where BEVT achieves very promising results. On Kinetics 400, for which recognition mostly relies on discriminative spatial representations, BEVT achieves comparable results to strong supervised baselines. On Something-Something-V2 and Diving 48, which contain videos relying on temporal dynamics, BEVT outperforms by clear margins all alternative baselines and achieves state-of-the-art performance with a 71.4% and 87.2% Top-1 accuracy respectively.\r\n\r\n\u003cimg src=\"assets/bevt_framework.png\"\u003e\r\n\r\n\r\n\r\n## Main Results on Downstream Tasks\r\n\r\n**Something-Something V2**\r\n\r\n| Backbone |  Pretrain   | Tokenizer | acc@1 | #params | FLOPs | Views | config | model |\r\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\r\n|  Swin-B  | ImageNet-1K + K400 |  DALL-E   |  70.6  |   89M   |  321G  |  1x3  |  [config](configs/recognition/swin/swin_base_patch244_window1677_bevt_finetune_ssv2.py)  | ToDo |\r\n|  Swin-B  | ImageNet-1K + K400 |  PeCo     |  71.4  |   89M   |  321G  |  1x3  |  [config](configs/recognition/swin/swin_base_patch244_window1677_bevt_finetune_ssv2.py)  | ToDo |\r\n\r\n\r\n**Kinetics-400**\r\n\r\n| Backbone |  Pretrain   | Tokenizer | acc@1 | #params | FLOPs | Views | config | model |\r\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\r\n|  Swin-B  | ImageNet-1K + K400 |  DALL-E   |  80.6  |   88M   |  282G  |  4x3  |  [config](configs/recognition/swin/swin_base_patch244_window877_bevt_finetune_k400.py)  | ToDo |\r\n|  Swin-B  | ImageNet-1K + K400 |  PeCo     |  81.1\u003csup\u003e*\u003c/sup\u003e  |   88M   |  282G  |  4x3  |  [config](configs/recognition/swin/swin_base_patch244_window877_bevt_finetune_k400.py)  | ToDo |\r\n\r\n**Note**:\r\n\r\n- BEVT uses the visual tokenizer of pretrained VQ-VAE from [DALL-E](https://arxiv.org/abs/2102.12092) or [PeCo](https://arxiv.org/abs/2111.12710).\r\n- PeCo is only pretrained on ImageNet1K and uses the same codebook size as in DALL-E.\r\n- BEVT does not need labels during pretraining.\r\n- \u003csup\u003e*\u003c/sup\u003e BEVT can achieve **81.5%** Top-1 accuracy on Kinetics-400 when using PeCo tokenizer for pretraining and finetuning for 100 epochs.  \r\n\r\n\r\n## Usage\r\n\r\n###  Installation\r\n\r\nPlease refer to [install.md](docs/install.md) for installation.\r\n\r\nWe use [apex](https://github.com/NVIDIA/apex.git) for mixed precision training by default.\r\n\r\n###  Data Preparation\r\n\r\nPlease refer to [data_preparation.md](docs/data_preparation.md) for a general knowledge of data preparation.\r\n\r\nWe use Kinetics-400 annotation files [k400_val](https://github.com/SwinTransformer/storage/releases/download/v1.0.6/k400_val.txt), [k400_train](https://github.com/SwinTransformer/storage/releases/download/v1.0.6/k400_train.txt) from [Video Swin Transformer](https://github.com/SwinTransformer/Video-Swin-Transformer).\r\n\r\n\r\n### BEVT Pretraining\r\n\r\nInstall DALL-E package before training:\r\n```\r\npip install DALL-E\r\n```\r\n\r\nDownload DALL-E tokenizer weight before training:\r\n```shell\r\nTOKENIZER_PATH=/path/to/save/dall_e_tokenizer_weight\r\nmkdir -p $TOKENIZER_PATH\r\nwget -O $TOKENIZER_PATH/encoder.pkl https://cdn.openai.com/dall-e/encoder.pkl\r\nwget -O $TOKENIZER_PATH/decoder.pkl https://cdn.openai.com/dall-e/decoder.pkl\r\n```\r\n\r\nSet `tokenizer_path` in the config file. For example, `configs/recognition/swin/swin_base_patch244_window877_bevt_in1k_k400.py`:\r\n```shell\r\ntokenizer_path = '/path/to/save/dall_e_tokenizer_weight'\r\n```\r\n\r\nFirst, pretrain the image stream of BEVT (Swin-base) on ImageNet-1K (800 epochs). The pretrained model of image stream could be downloaded at [google drive](https://drive.google.com/file/d/1VHKAH9YA_VD8M8bfGp2Svreqv0iuikB6/view?usp=sharing).\r\n\r\nThen pretrain two stream of BEVT on ImageNet-1K and K400 (initialized from swin transformer pretrained with the image stream) with 32 GPUs (150 epochs):\r\n\r\n```shell\r\nbash tools/dist_train.sh configs/recognition/swin/swin_base_patch244_window877_bevt_in1k_k400.py --work-dir OUTPUT/swin_base_bevt_twostream --cfg-options total_epochs=150 model.backbone.pretrained='/path/to/save/swin_base_image_stream_pretrain.pth' --seed 0 --deterministic\r\n```\r\n\r\nThe pretrained model of BEVT could be downloaded at [google drive](https://drive.google.com/file/d/13pMnG1ylyyKmhx5MKNZvqe7jHtPMY9cy/view?usp=sharing).\r\n\r\n### BEVT Finetuning\r\n\r\nFinetune BEVT model on K400 with 8 GPUs:\r\n\r\n```shell\r\nbash tools/dist_train.sh configs/recognition/swin/swin_base_patch244_window877_bevt_finetune_k400.py --work-dir OUTPUT/bevt_finetune/swin_base_bevt_finetune_k400 --cfg-options model.backbone.pretrained='OUTPUT/swin_base_bevt_twostream/latest.pth' --seed 0 --deterministic --validate --test-best --test-last\r\n```\r\n\r\nFinetune BEVT model on SSv2 with 8 GPUs:\r\n\r\n```shell\r\nbash tools/dist_train.sh configs/recognition/swin/swin_base_patch244_window1677_bevt_finetune_ssv2.py --work-dir OUTPUT/bevt_finetune/swin_base_bevt_finetune_ssv2 --cfg-options model.backbone.pretrained='OUTPUT/swin_base_bevt_twostream/latest.pth' --seed 0 --deterministic --validate --test-best --test-last\r\n```\r\n\r\n## To Do\r\n- [x] Release joint pretraining code\r\n- [x] Release fine-tuning code  \r\n- [x] Release pretrained model\r\n- [ ] Release finetuned model\r\n- [ ] Release image stream pretraining code\r\n\r\n## Acknowledgements\r\n\r\nThis code is based on [mmaction2](https://github.com/open-mmlab/mmaction2) and [Video Swin Transformer](https://github.com/SwinTransformer/Video-Swin-Transformer).\r\n\r\n## Citation\r\n\r\n```\r\n@inproceedings{wang2021bevt,\r\n  title={BEVT: BERT Pretraining of Video Transformers},\r\n  author={Wang, Rui and Chen, Dongdong and Wu, Zuxuan and Chen, Yinpeng and Dai, Xiyang and Liu, Mengchen and Jiang, Yu-Gang and Zhou, Luowei and Yuan, Lu},\r\n  booktitle={CVPR},\r\n  year={2022}\r\n}\r\n\r\n@article{dong2021peco,\r\n  title={PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers},\r\n  author={Dong, Xiaoyi and Bao, Jianmin and Zhang, Ting and Chen, Dongdong and Zhang, Weiming and Yuan, Lu and Chen, Dong and Wen, Fang and Yu, Nenghai},\r\n  journal={arXiv preprint arXiv:2111.12710},\r\n  year={2021}\r\n}\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxyzforever%2FBEVT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxyzforever%2FBEVT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxyzforever%2FBEVT/lists"}