{"id":13787846,"url":"https://github.com/jayleicn/ClipBERT","last_synced_at":"2025-05-12T02:30:43.633Z","repository":{"id":41112698,"uuid":"337849864","full_name":"jayleicn/ClipBERT","owner":"jayleicn","description":"[CVPR 2021 Best Student Paper Honorable Mention, Oral] Official PyTorch code for ClipBERT, an efficient framework for end-to-end learning on image-text and video-text tasks. ","archived":false,"fork":false,"pushed_at":"2023-08-08T22:18:55.000Z","size":75,"stargazers_count":693,"open_issues_count":12,"forks_count":84,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-07-24T23:01:50.553Z","etag":null,"topics":["cvpr2021","pytorch","video-question-answering","video-retrieval","vision-and-language","vqa"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2102.06183","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jayleicn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-02-10T20:48:30.000Z","updated_at":"2024-07-04T08:18:42.000Z","dependencies_parsed_at":"2024-01-18T23:52:29.246Z","dependency_job_id":null,"html_url":"https://github.com/jayleicn/ClipBERT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayleicn%2FClipBERT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayleicn%2FClipBERT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayleicn%2FClipBERT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayleicn%2FClipBERT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jayleicn","download_url":"https://codeload.github.com/jayleicn/ClipBERT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":213856345,"owners_count":15648199,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cvpr2021","pytorch","video-question-answering","video-retrieval","vision-and-language","vqa"],"created_at":"2024-08-03T21:00:32.269Z","updated_at":"2024-08-03T21:01:17.791Z","avatar_url":"https://github.com/jayleicn.png","language":"Python","funding_links":[],"categories":["Implementations","Python"],"sub_categories":[],"readme":"# ClipBERT\n\n[Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://arxiv.org/abs/2102.06183) \n\n[CVPR 2021](http://cvpr2021.thecvf.com/), Oral, [Best Student Paper Honorable Mention](http://cvpr2021.thecvf.com/node/329).\n\n[Jie Lei](http://www.cs.unc.edu/~jielei/)\\*, [Linjie Li](https://www.linkedin.com/in/linjie-li/)\\*,\n[Luowei Zhou](https://luoweizhou.github.io/), [Zhe Gan](https://zhegan27.github.io/), \n[Tamara L. Berg](http://tamaraberg.com/), [Mohit Bansal](http://www.cs.unc.edu/~mbansal/),\n[Jingjing Liu](https://www.linkedin.com/in/jingjing-liu-65703431/)\n\nOfficial PyTorch code for ClipBERT, an efficient framework for \nend-to-end learning for image-text and video-text tasks. \nIt takes raw videos/images + text as inputs, and outputs task predictions.\nClipBERT is designed based on 2D CNNs and transformers, and uses a sparse sampling strategy \nto enable efficient end-to-end video-and-language learning. In this repository, \nwe support end-to-end pretraining and finetuning for the following tasks:\n\n- Image-text pretraining on COCO and VG captions.\n- Text-to-video retrieval finetuning on MSRVTT, DiDeMo, and ActivityNet Captions.\n- Video-QA finetuning on TGIF-QA and MSRVTT-QA.\n- Image-QA finetuning on VQA 2.0.\n\nIt is also feasible and easy to add other image-text or video-text tasks for pretraining and finetuning. \n\n\n**[NEW]** If you are interested in ClipBERT, you might also be interested in our recent work, **Singularity** ([paper](https://arxiv.org/abs/2206.03428), [code](https://github.com/jayleicn/singularity)), with a single-frame trained model, it achieves state-of-the-art results a set of video-language tasks. \n\n## Requirements \nWe provide a Docker image for easier reproduction. Please install the following:\n  - [nvidia driver](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#package-manager-installation) (418+), \n  - [Docker](https://docs.docker.com/install/linux/docker-ce/ubuntu/) (19.03+), \n  - [nvidia-container-toolkit](https://github.com/NVIDIA/nvidia-docker#quickstart).\n\nOur scripts require the user to have the [docker group membership](https://docs.docker.com/install/linux/linux-postinstall/)\nso that docker commands can be run without sudo.\nWe only support Linux with NVIDIA GPUs. We test on Ubuntu 18.04 and V100 cards.\nWe use mixed-precision training hence GPUs with Tensor Cores are recommended.\n\n\n## Getting Started\n\n### General\n\n1. Create a folder that stores pretrained models, all the data, and results.\n    ```bash\n    PATH_TO_STORAGE=/path/to/your/data/\n    mkdir -p $PATH_TO_STORAGE/txt_db  # annotations\n    mkdir -p $PATH_TO_STORAGE/vis_db  # image and video \n    mkdir -p $PATH_TO_STORAGE/finetune  # finetuning results\n    mkdir -p $PATH_TO_STORAGE/pretrained  # pretrained models\n    ```\n\n2. Download pretrained models.\n\n    Our e2e pretrained ClipBERT model (849MB), can be downloaded with the following command.\n    ```bash\n    bash scripts/download_pretrained.sh $PATH_TO_STORAGE\n    ```\n    This pretrained model can be used for finetuning on video-text tasks and image-text tasks.\n    For your convenience, this script will also download `bert-base-uncased` and `grid-feat-vqa` \n    model weights, which are used as initialization for pretraining.  \n\n3. Launch the Docker container for running the experiments.\n    ```bash\n    # docker image should be automatically pulled\n    source launch_container.sh $PATH_TO_STORAGE/txt_db $PATH_TO_STORAGE/vis_db \\\n        $PATH_TO_STORAGE/finetune $PATH_TO_STORAGE/pretrained\n    ```\n    The launch script respects $CUDA_VISIBLE_DEVICES environment variable.\n    Note that the source code is mounted into the container under `/clipbert` instead \n    of built into the image so that user modification will be reflected without\n    re-building the image. (Data folders are mounted into the container separately\n    for flexibility on folder structures.)\n\n### Downstream Task Finetuning\n\n#### Text-to-Video Retrieval\n\nTasks: MSRVTT retrieval, DiDeMo and ActivityNet Captions paragprah-to-video retrieval, MSRVTT MC Test.\n\n1. Download data.\n    ```bash\n    # outside the container  \n    # download videos + annotations for $DSET\n    bash scripts/download_$DSET.sh $PATH_TO_STORAGE\n    ```\n    `$DSET` can be one of `msrvtt`, `didemo`, `anet`.\n\n2. Finetuning. \n    ```bash\n    # inside the container\n    horovodrun -np 4 python src/tasks/run_video_retrieval.py \\\n        --config $CONFIG_PATH \\\n        --output_dir $OUTPUT_DIR\n   \n    # for single GPU\n    python src/tasks/run_video_retrieval.py \\\n        --config $CONFIG_PATH \\\n        --output_dir $OUTPUT_DIR\n    ```\n    `$CONFIG_PATH` should be set to one of the .json config files available at [src/configs](src/configs) \n    prefixed with `_ret`. For example, you can use `src/configs/msrvtt_ret_base_resnet50.json` \n    for MSRVTT retrieval.\n    \n3. Run inference.\n    ```bash\n    # inside the container\n    horovodrun -np 4 python src/tasks/run_video_retrieval.py \\\n      --do_inference 1 --output_dir $OUTPUT_DIR \\\n      --inference_split val --inference_model_step $STEP \\\n      --inference_txt_db $TXT_DB \\\n      --inference_img_db $IMG_DB --inference_batch_size 64 \\\n      --inference_n_clips $INFERENCE_N_CLIPS\n    ```\n   `$STEP` is an integer, it tells the script to use the checkpoint \n   `$OUTPUT_DIR/ckpt/model_step_$STEP.pt` for inference.\n   `$TXT_DB` and `$IMG_DB` are path to annotation file and video data. You can use\n   `TXT_DB=/txt/downstream/msrvtt_retrieval/msrvtt_retrieval_val.jsonl` and \n   `IMG_DB=/img/msrvtt` for inference on MSRVTT retrieval val split.\n    The results will be written under `$OUTPUT_DIR`. You can use different `$INFERENCE_N_CLIPS` \n    for inference, such as 1 or 16. Using more clips will have a large impact \n    on inference speed and memory usage. You may want to use smaller batch sizes if larger \n    values are set.\n    \n    After MSRVTT retrieval model is trained, you can use it for inference \n    for the MSRVTT MC Test task as well, which is essentially a retrieval \n    task in a multiple-choice task setup. \n    ```bash\n    # inside the container\n    horovodrun -np 4 python src/tasks/run_msrvtt_mc.py \\\n      --do_inference 1 --output_dir $OUTPUT_DIR \\\n      --inference_split val --inference_model_step $STEP \\\n      --inference_txt_db /txt/downstream/msrvtt_retrieval_mc/msrvtt_retrieval_mc_test.jsonl \\\n      --inference_img_db /img/msrvtt --inference_batch_size 64 \\\n      --inference_n_clips $INFERENCE_N_CLIPS\n    ```    \n   \n   \n#### Video Question Answering\n\nTasks: TGIF-QA action, transition, and frameQA tasks; MSRVTT-QA. \n\n1. Download data.\n    ```bash\n    # outside the container  \n    # download MSRVTT videos, and QA + retrieval annotations\n    bash scripts/download_msrvtt.sh $PATH_TO_STORAGE  \n    # download TGIF-QA videos and annotations\n    bash scripts/download_tgif_qa.sh $PATH_TO_STORAGE  \n    ```\n\n2. Finetuning. \n    ```bash\n    # inside the container\n    horovodrun -np 4 python src/tasks/run_video_qa.py \\\n        --config $CONFIG_PATH \\\n        --output_dir $OUTPUT_DIR\n    ```\n    `$CONFIG_PATH` should be set to one of the .json config files available at [src/configs](src/configs) \n    contains the substring `_qa`. For example, you can use `src/configs/msrvtt_qa_base_resnet50.json` \n    for MSRVTT-QA.\n\n3. Run inference.\n    ```bash\n    # inside the container\n    horovodrun -np 4 python src/tasks/run_video_qa.py \\\n      --do_inference 1 --output_dir $OUTPUT_DIR \\\n      --inference_split val --inference_model_step $STEP \\\n      --inference_txt_db $TXT_DB \\\n      --inference_img_db $IMG_DB --inference_batch_size 64 \\\n      --inference_n_clips $INFERENCE_N_CLIPS\n    ```\n   `$STEP` is an integer, which tells the script to use the checkpoint \n   `$OUTPUT_DIR/ckpt/model_step_$STEP.pt` for inference.\n   `$TXT_DB` and `$IMG_DB` are path to annotation file and video data. You can use\n   `TXT_DB=/txt/downstream/msrvtt_retrieval/msrvtt_qa_val.jsonl` and \n   `IMG_DB=/img/msrvtt` for inference on MSRVTT QA val split.\n   \n    The results will be written under `$OUTPUT_DIR`. You can use different `$INFERENCE_N_CLIPS` \n    for inference, such as 1 or 16. Using more clips will have a large impact \n    on inference speed and memory usage. You may want to use smaller batch sizes if larger \n    values are set.\n\n\n#### Image Question Answering (VQA)\n1. Download data\n    ```bash\n    # outside the container\n    # download COCO and VG data\n    bash scripts/download_coco_vg.sh $PATH_TO_STORAGE\n    # download VQA annotations\n    bash scripts/download_vqa.sh $PATH_TO_STORAGE\n    ```\n\n2. Finetuning\n    ```bash\n    # inside the container\n    horovodrun -np 4 python src/tasks/run_vqa.py \\\n        --config src/configs/vqa_base_resnet50.json \\\n        --output_dir $OUTPUT_DIR\n    ```\n\n3. Inference\n    ```bash\n    # inside the container\n    horovodrun -np 4 python src/tasks/run_vqa.py \\\n      --do_inference 1 --output_dir $OUTPUT_DIR \\\n      --inference_split val --inference_model_step $STEP \\\n      --inference_txt_db $TXT_DB \\\n      --inference_img_db $IMG_DB \\\n      --inference_batch_size 64\n    ```\n\n\n## Pretraining\n1. Download data\n    ```bash\n    # outside the container\n    bash scripts/download_coco_vg.sh $PATH_TO_STORAGE\n    ```\n\n2. Pretraining\n    ```bash\n    #inside the container\n    horovodrun -np 8 python src/pretrain/run_pretrain.py \\\n        --config src/configs/pretrain_image_text_base_resnet50_mlm_itm.json \\\n        --output_dir $OUTPUT_DIR \n    ``` \n\n## Data Preprocessing\nClipBERT takes raw video and text as inputs, there is no need to do feature extraction. \nHowever, to improve data loading speed, we use LMDB to store the raw image and video files. \nYou can use the following script to convert a list of videos with file extensions `mp4` and `avi` into LMDB:\n    \n```bash\n# outside the container\npython src/preprocessing/file2lmdb.py \\\n    --data_root /path/to/videos \\\n    --lmdb_save_dir /path/to/save/lmdb \\\n    --ext avi mp4 \\\n    --file_type video \n```\n\nFor images, use appropriate file extensions for `--ext` and `--file_type image`. \nText annotation files are reorganized into `jsonl` files, \nsee example preprocessed files downloaded by the scripts in [scripts/](scripts).   \n\n\n## Citation\n\nIf you find this code useful for your research, please consider citing:\n```\n@inproceedings{lei2021less,\n  title={Less is More: ClipBERT for Video-and-Language Learningvia Sparse Sampling},\n  author={Lei, Jie and Li, Linjie and Zhou, Luowei and Gan, Zhe and Berg, Tamara L. and Bansal, Mohit and Liu, Jingjing},\n  booktitle={CVPR},\n  year={2021}\n}\n```\n\n## Acknowledgement\nWe thank [Yen-Chun Chen](https://scholar.google.com/citations?user=Gptgy4YAAAAJ\u0026hl=en), \n[Ruotian Luo](https://ttic.uchicago.edu/~rluo/), and other members and interns at \n[Microsoft Multimodal AI](https://multimodalai.azurewebsites.net/people/members) \nfor their helpful discussions.\nWe also thank the anonymous reviewers for their constructive feedback.\n\nThis code used resources from [transformers](https://github.com/huggingface/transformers), \n[UNITER](https://github.com/ChenRocks/UNITER), [HERO](https://github.com/linjieli222/HERO), \n[grid-feats-vqa](https://github.com/facebookresearch/grid-feats-vqa), \n[SlowFast](https://github.com/facebookresearch/SlowFast), \n[Detectron2](https://github.com/facebookresearch/detectron2). \nThe code is implemented using [PyTorch](https://github.com/pytorch/pytorch), \nwith multi-GPU support from [Horovod](https://github.com/horovod/horovod) \nand mixed precision support from [apex](https://github.com/NVIDIA/apex).  We thank the authors for open-sourcing their awesome projects.\n\n\n\n## License\n\nMIT\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjayleicn%2FClipBERT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjayleicn%2FClipBERT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjayleicn%2FClipBERT/lists"}