{"id":13958477,"url":"https://github.com/zhegan27/VILLA","last_synced_at":"2025-07-21T00:30:52.424Z","repository":{"id":89305839,"uuid":"305626176","full_name":"zhegan27/VILLA","owner":"zhegan27","description":"Research Code for NeurIPS 2020 Spotlight paper \"Large-Scale Adversarial Training for Vision-and-Language Representation Learning\": UNITER adversarial training part","archived":false,"fork":false,"pushed_at":"2021-01-13T06:08:41.000Z","size":869,"stargazers_count":119,"open_issues_count":7,"forks_count":14,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-11-28T02:34:45.030Z","etag":null,"topics":["adversarial-training","neurips-2020","pretraining","vision-and-language","visual-question-answering"],"latest_commit_sha":null,"homepage":"https://arxiv.org/pdf/2006.06195.pdf","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zhegan27.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-20T07:32:19.000Z","updated_at":"2024-09-05T08:28:06.000Z","dependencies_parsed_at":"2023-05-01T18:34:43.035Z","dependency_job_id":null,"html_url":"https://github.com/zhegan27/VILLA","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zhegan27/VILLA","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhegan27%2FVILLA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhegan27%2FVILLA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhegan27%2FVILLA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhegan27%2FVILLA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zhegan27","download_url":"https://codeload.github.com/zhegan27/VILLA/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhegan27%2FVILLA/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266221247,"owners_count":23894964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-training","neurips-2020","pretraining","vision-and-language","visual-question-answering"],"created_at":"2024-08-08T13:01:37.580Z","updated_at":"2025-07-21T00:30:52.419Z","avatar_url":"https://github.com/zhegan27.png","language":"Python","funding_links":[],"categories":["其他_机器视觉"],"sub_categories":["网络服务_其他"],"readme":"# VILLA: Vision-and-Language Adversarial Training\nThis is the official repository of [VILLA](https://arxiv.org/pdf/2006.06195.pdf) (NeurIPS 2020 Spotlight).\nThis repository currently supports adversarial finetuning of UNITER on [VQA](https://visualqa.org/), [VCR](https://visualcommonsense.com/), [NLVR2](http://lil.nlp.cornell.edu/nlvr/), and \n[SNLI-VE](https://github.com/necla-ml/SNLI-VE).\nAdversarial pre-training with in-domain data will be available soon.\nBoth VILLA-base and VILLA-large pre-trained checkpoints are released. \n\n![Overview of VILLA](villa_framework.png)\n\nMost of the code in this repo are copied/modified from [UNITER](https://github.com/ChenRocks/UNITER).\n\n\n## Requirements\nWe provide Docker image for easier reproduction. Please install the following:\n  - [nvidia driver](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#package-manager-installation) (418+), \n  - [Docker](https://docs.docker.com/install/linux/docker-ce/ubuntu/) (19.03+), \n  - [nvidia-container-toolkit](https://github.com/NVIDIA/nvidia-docker#quickstart).\n\nOur scripts require the user to have the [docker group membership](https://docs.docker.com/install/linux/linux-postinstall/)\nso that docker commands can be run without sudo.\nWe only support Linux with NVIDIA GPUs. We test on Ubuntu 18.04 and V100 cards.\nWe use mixed-precision training hence GPUs with Tensor Cores are recommended.\n\n## Quick Start\n*NOTE*: Please run `bash scripts/download_pretrained.sh $PATH_TO_STORAGE` to get our latest pretrained VILLA\ncheckpoints. This will download both the base and large models.\n\nWe use VQA as an end-to-end example for using this code base.\n\n1. Download processed data and pretrained models with the following command.\n    ```bash\n    bash scripts/download_vqa.sh $PATH_TO_STORAGE\n    ```\n    After downloading you should see the following folder structure:\n    ```\n    ├── finetune \n    ├── img_db\n    │   ├── coco_test2015\n    │   ├── coco_test2015.tar\n    │   ├── coco_train2014\n    │   ├── coco_train2014.tar\n    │   ├── coco_val2014\n    │   ├── coco_val2014.tar\n    │   ├── vg\n    │   └── vg.tar\n    ├── pretrained\n        ├── uniter-base.pt\n    │   └── villa-base.pt\n    └── txt_db\n        ├── vqa_devval.db\n        ├── vqa_devval.db.tar\n        ├── vqa_test.db\n        ├── vqa_test.db.tar\n        ├── vqa_train.db\n        ├── vqa_train.db.tar\n        ├── vqa_trainval.db\n        ├── vqa_trainval.db.tar\n        ├── vqa_vg.db\n        └── vqa_vg.db.tar\n\n    ```\n    You can put different pre-trained checkpoints inside the /pretrained folder based on your need. \n\n2. Launch the Docker container for running the experiments.\n    ```bash\n    # docker image should be automatically pulled\n    source launch_container.sh $PATH_TO_STORAGE/txt_db $PATH_TO_STORAGE/img_db \\\n        $PATH_TO_STORAGE/finetune $PATH_TO_STORAGE/pretrained\n    ```\n    The launch script respects $CUDA_VISIBLE_DEVICES environment variable.\n    Note that the source code is mounted into the container under `/src` instead \n    of built into the image so that user modification will be reflected without\n    re-building the image. (Data folders are mounted into the container separately\n    for flexibility on folder structures.)\n\n\n3. Run finetuning for the VQA task.\n    ```bash\n    # inside the container\n    horovodrun -np $N_GPU python train_vqa_adv.py --config $YOUR_CONFIG_JSON\n\n    # specific example\n    horovodrun -np 4 python train_vqa_adv.py --config config/train-vqa-base-4gpu-adv.json\n    ```\n\n4. Run inference for the VQA task and then evaluate.\n    ```bash\n    # inference\n    python inf_vqa.py --txt_db /txt/vqa_test.db --img_db /img/coco_test2015 \\\n    --output_dir $VQA_EXP --checkpoint 6000 --pin_mem --fp16\n    ```\n    The result file will be written at `$VQA_EXP/results_test/results_6000_all.json`, which can be\n    submitted to the evaluation server\n\n\n5. Customization\n    ```bash\n    # training options\n    python train_vqa_adv.py --help\n    ```\n    - command-line argument overwrites JSON config files\n    - JSON config overwrites `argparse` default value.\n    - use horovodrun to run multi-GPU training\n    - `--gradient_accumulation_steps` emulates multi-gpu training\n    - `--checkpoint` selects UNITER or VILLA pre-trained checkpoints\n    - `--adv_training` decides using adv. training or not\n    - `--adv_modality` takes values from ['text'], ['image'], ['text','image'], and ['text','image','alter'], the last two correspond to adding perturbations on two modalities simultaneously or alternatively\n\n## Downstream Tasks Finetuning\n\n### VCR\nNOTE: train and inference should be ran inside the docker container\n1. download data\n    ```\n    bash scripts/download_vcr.sh $PATH_TO_STORAGE\n    ```\n2. train\n    ```\n    horovodrun -np 4 python train_vcr_adv.py --config config/train-vcr-base-4gpu-adv.json \\\n        --output_dir $VCR_EXP\n    ```\n3. inference\n    ```\n    horovodrun -np 4 python inf_vcr.py --txt_db /txt/vcr_test.db \\\n        --img_db \"/img/vcr_gt_test/;/img/vcr_test/\" \\\n        --split test --output_dir $VCR_EXP --checkpoint 8000 \\\n        --pin_mem --fp16\n    ```\n    The result file will be written at `$VCR_EXP/results_test/results_8000_all.csv`, which can be\n    submitted to VCR leaderboard for evaluation.\n\n### NLVR2\nNOTE: train and inference should be ran inside the docker container\n1. download data\n    ```\n    bash scripts/download_nlvr2.sh $PATH_TO_STORAGE\n    ```\n2. train\n    ```\n    horovodrun -np 4 python train_nlvr2_adv.py --config config/train-nlvr2-base-1gpu-adv.json \\\n        --output_dir $NLVR2_EXP\n    ```\n3. inference\n    ```\n    python inf_nlvr2.py --txt_db /txt/nlvr2_test1.db/ --img_db /img/nlvr2_test/ \\\n    --train_dir /storage/nlvr-base/ --ckpt 6500 --output_dir . --fp16\n    ```\n\n### Visual Entailment (SNLI-VE)\nNOTE: train should be ran inside the docker container\n1. download data\n    ```\n    bash scripts/download_ve.sh $PATH_TO_STORAGE\n    ```\n2. train\n    ```\n    horovodrun -np 2 python train_ve_adv.py --config config/train-ve-base-2gpu-adv.json \\\n        --output_dir $VE_EXP\n    ```\n\n## Adversarial Training of LXMERT\n\nTo keep things simple, we provide [another separate repo](https://github.com/zhegan27/LXMERT-AdvTrain) that can be used to reproduce our results on adversarial finetuning of [LXMERT](https://arxiv.org/pdf/1908.07490.pdf) on [VQA](https://visualqa.org/), [GQA](https://cs.stanford.edu/people/dorarad/gqa/about.html), and [NLVR2](http://lil.nlp.cornell.edu/nlvr/).\n\n## Citation\n\nIf you find this code useful for your research, please consider citing:\n```\n@inproceedings{gan2020large,\n  title={Large-Scale Adversarial Training for Vision-and-Language Representation Learning},\n  author={Gan, Zhe and Chen, Yen-Chun and Li, Linjie and Zhu, Chen and Cheng, Yu and Liu, Jingjing},\n  booktitle={NeurIPS},\n  year={2020}\n}\n\n@inproceedings{chen2020uniter,\n  title={Uniter: Universal image-text representation learning},\n  author={Chen, Yen-Chun and Li, Linjie and Yu, Licheng and Kholy, Ahmed El and Ahmed, Faisal and Gan, Zhe and Cheng, Yu and Liu, Jingjing},\n  booktitle={ECCV},\n  year={2020}\n}\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhegan27%2FVILLA","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhegan27%2FVILLA","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhegan27%2FVILLA/lists"}