{"id":13958460,"url":"https://github.com/facebookresearch/vilbert-multi-task","last_synced_at":"2025-07-21T00:30:53.727Z","repository":{"id":41096662,"uuid":"224054803","full_name":"facebookresearch/vilbert-multi-task","owner":"facebookresearch","description":"Multi Task Vision and Language","archived":true,"fork":false,"pushed_at":"2022-02-16T04:47:38.000Z","size":119980,"stargazers_count":804,"open_issues_count":68,"forks_count":179,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-02-23T00:14:17.613Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/facebookresearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-25T22:37:44.000Z","updated_at":"2025-02-11T11:22:38.000Z","dependencies_parsed_at":"2022-07-09T19:00:51.996Z","dependency_job_id":null,"html_url":"https://github.com/facebookresearch/vilbert-multi-task","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/facebookresearch/vilbert-multi-task","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fvilbert-multi-task","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fvilbert-multi-task/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fvilbert-multi-task/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fvilbert-multi-task/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/facebookresearch","download_url":"https://codeload.github.com/facebookresearch/vilbert-multi-task/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fvilbert-multi-task/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266221247,"owners_count":23894964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-08T13:01:36.796Z","updated_at":"2025-07-21T00:30:52.830Z","avatar_url":"https://github.com/facebookresearch.png","language":"Jupyter Notebook","readme":"# 12-in-1: Multi-Task Vision and Language Representation Learning\n\nPlease cite the following if you use this code. Code and pre-trained models for [12-in-1: Multi-Task Vision and Language Representation Learning](http://openaccess.thecvf.com/content_CVPR_2020/html/Lu_12-in-1_Multi-Task_Vision_and_Language_Representation_Learning_CVPR_2020_paper.html):\n\n```\n@InProceedings{Lu_2020_CVPR,\nauthor = {Lu, Jiasen and Goswami, Vedanuj and Rohrbach, Marcus and Parikh, Devi and Lee, Stefan},\ntitle = {12-in-1: Multi-Task Vision and Language Representation Learning},\nbooktitle = {The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\nmonth = {June},\nyear = {2020}\n}\n```\n\nand [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://arxiv.org/abs/1908.02265):\n\n```\n@inproceedings{lu2019vilbert,\n  title={Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks},\n  author={Lu, Jiasen and Batra, Dhruv and Parikh, Devi and Lee, Stefan},\n  booktitle={Advances in Neural Information Processing Systems},\n  pages={13--23},\n  year={2019}\n}\n```\n\n## Repository Setup\n\n1. Create a fresh conda environment, and install all dependencies.\n\n```text\nconda create -n vilbert-mt python=3.6\nconda activate vilbert-mt\ngit clone --recursive https://github.com/facebookresearch/vilbert-multi-task.git\ncd vilbert-multi-task\npip install -r requirements.txt\n```\n\n2. Install pytorch\n```\nconda install pytorch torchvision cudatoolkit=10.0 -c pytorch\n```\n\n3. Install apex, follows https://github.com/NVIDIA/apex\n\n4. Install this codebase as a package in this environment.\n```text\npython setup.py develop\n```\n\n## Data Setup\n\nCheck `README.md` under `data` for more details.  \n\n## Visiolinguistic Pre-training and Multi Task Training\n\n### Pretraining on Conceptual Captions\n\n```\npython train_concap.py --bert_model bert-base-uncased --config_file config/bert_base_6layer_6conect.json --train_batch_size 512 --objective 1 --file_path \u003cpath_to_extracted_cc_features\u003e\n```\n[Download link](https://dl.fbaipublicfiles.com/vilbert-multi-task/pretrained_model.bin)\n\n### Multi-task Training\n\n```\npython train_tasks.py --bert_model bert-base-uncased --from_pretrained \u003cpretrained_model_path\u003e --config_file config/bert_base_6layer_6conect.json --tasks 1-2-4-7-8-9-10-11-12-13-15-17 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name multi_task_model\n```\n\n[Download link](https://dl.fbaipublicfiles.com/vilbert-multi-task/multi_task_model.bin)\n\n\n### Fine-tune from Multi-task trained model\n\n```\npython train_tasks.py --bert_model bert-base-uncased --from_pretrained \u003cmulti_task_model_path\u003e --config_file config/bert_base_6layer_6conect.json --tasks 1 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name finetune_from_multi_task_model\n```\n \n## License\n\nvilbert-multi-task is licensed under MIT license available in [LICENSE](LICENSE) file.\n","funding_links":[],"categories":["其他_机器视觉"],"sub_categories":["网络服务_其他"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fvilbert-multi-task","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffacebookresearch%2Fvilbert-multi-task","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fvilbert-multi-task/lists"}