{"id":19098703,"url":"https://github.com/foolwood/drl","last_synced_at":"2025-08-30T02:03:45.639Z","repository":{"id":51675098,"uuid":"478810749","full_name":"foolwood/DRL","owner":"foolwood","description":"[arXiv22] Disentangled Representation Learning for Text-Video Retrieval","archived":false,"fork":false,"pushed_at":"2022-04-07T05:36:40.000Z","size":6334,"stargazers_count":94,"open_issues_count":0,"forks_count":5,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-30T15:27:06.081Z","etag":null,"topics":["clip","interaction-nets","text-video-search-engine","transformer","video-retrieval"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/foolwood.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-04-07T03:18:54.000Z","updated_at":"2025-04-26T15:03:20.000Z","dependencies_parsed_at":"2022-08-22T04:40:18.503Z","dependency_job_id":null,"html_url":"https://github.com/foolwood/DRL","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/foolwood/DRL","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foolwood%2FDRL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foolwood%2FDRL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foolwood%2FDRL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foolwood%2FDRL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/foolwood","download_url":"https://codeload.github.com/foolwood/DRL/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foolwood%2FDRL/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272793017,"owners_count":24993830,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip","interaction-nets","text-video-search-engine","transformer","video-retrieval"],"created_at":"2024-11-09T03:47:08.150Z","updated_at":"2025-08-30T02:03:44.941Z","avatar_url":"https://github.com/foolwood.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Disentangled Representation Learning for Text-Video Retrieval\n[![MSR-VTT](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/disentangled-representation-learning-for-text/video-retrieval-on-msr-vtt-1ka)](https://paperswithcode.com/sota/video-retrieval-on-msr-vtt-1ka?p=disentangled-representation-learning-for-text)\n[![DiDeMo](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/disentangled-representation-learning-for-text/video-retrieval-on-didemo)](https://paperswithcode.com/sota/video-retrieval-on-didemo?p=disentangled-representation-learning-for-text)\n\nThis is a PyTorch implementation of the paper [Disentangled Representation Learning for Text-Video Retrieval](https://arxiv.org/abs/2203.07111):\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"demo/pipeline.png\" width=\"800\"\u003e\n\u003c/p\u003e\n\n```\n@Article{DRLTVR2022,\n  author  = {Qiang Wang and Yanhao Zhang and Yun Zheng and Pan Pan and Xian-Sheng Hua},\n  journal = {arXiv:2203.07111},\n  title   = {Disentangled Representation Learning for Text-Video Retrieval},\n  year    = {2022},\n}\n```\n\n### Catalog\n\n- [x] Setup\n- [x] Fine-tuning code\n- [x] Visualization demo\n\n### Setup\n\n#### Setup code environment\n```shell\ngit clone https://github.com/foolwood/DRL.git\ncd DRL\nconda create -n drl python=3.9\nconda activate drl\npip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple/\npip install torch==1.8.1+cu102 torchvision==0.9.1+cu102 -f https://download.pytorch.org/whl/torch_stable.html\n```\n\n#### Download CLIP Model (as pretraining)\n\n```shell\ncd tvr/models\nwget https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt\n# wget https://openaipublic.azureedge.net/clip/models/5806e77cd80f8b59890b7e101eabd078d9fb84e6937f9e85e4ecb61988df416f/ViT-B-16.pt\n# wget https://openaipublic.azureedge.net/clip/models/b8cca3fd41ae0c99ba7e8951adf17d267cdb84cd88be6f7c2e0eca1737a03836/ViT-L-14.pt\n```\n\n#### Download Datasets\n\n```shell\ncd data/MSR-VTT\nwget https://www.robots.ox.ac.uk/~maxbain/frozen-in-time/data/MSRVTT.zip ; unzip MSRVTT.zip\nmv MSRVTT/videos/all ./videos ; mv MSRVTT/annotation/MSR_VTT.json ./anns/MSRVTT_data.json\n```\n\n### Fine-tuning code\n\n- Train on MSR-VTT 1k.\n\n```shell\nCUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 \\\nmain.py --do_train 1 --workers 8 --n_display 50 \\\n--epochs 5 --lr 1e-4 --coef_lr 1e-3 --batch_size 128 --batch_size_val 128 \\\n--anno_path data/MSR-VTT/anns --video_path data/MSR-VTT/videos --datatype msrvtt \\\n--max_words 32 --max_frames 12 --video_framerate 1 \\\n--base_encoder ViT-B/32 --agg_module seqTransf \\\n--interaction wti --wti_arch 2 --cdcr 3 --cdcr_alpha1 0.11 --cdcr_alpha2 0.0 --cdcr_lambda 0.001 \\\n--output_dir ckpts/ckpt_msrvtt_wti_cdcr\n```\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"demo/interaction.png\" width=\"800\"\u003e\n\u003c/p\u003e\n\nReproduce the ablation experiments [scripts](scripts/msrvtt.sh)\n\n\u003ctable\u003e\u003ctbody\u003e\n\u003cthead\u003e\n  \u003ctr\u003e\n    \u003cth rowspan=\"2\"\u003econfigs\u003cbr\u003e\u003c/th\u003e\n    \u003cth rowspan=\"2\"\u003efeature\u003c/th\u003e\n    \u003cth rowspan=\"2\"\u003egpus\u003c/th\u003e\n    \u003cth colspan=\"5\"\u003eText-Video\u003c/th\u003e\n    \u003cth colspan=\"5\"\u003eVideo-Text\u003c/th\u003e\n    \u003cth rowspan=\"2\"\u003etrain time (h)\u003cbr\u003e\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003cth\u003eR@1\u003c/th\u003e\n    \u003cth\u003eR@5\u003c/th\u003e\n    \u003cth\u003eR@10\u003c/th\u003e\n    \u003cth\u003eMdR\u003c/th\u003e\n    \u003cth\u003eMnR\u003c/th\u003e\n    \u003cth\u003eR@1\u003c/th\u003e\n    \u003cth\u003eR@5\u003c/th\u003e\n    \u003cth\u003eR@10\u003c/th\u003e\n    \u003cth\u003eMdR\u003c/th\u003e\n    \u003cth\u003eMnR\u003c/th\u003e\n  \u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eCLIP4Clip\u003c/td\u003e\n    \u003ctd\u003eViT/B-32\u003c/td\u003e\n    \u003ctd\u003e4\u003c/td\u003e\n    \u003ctd\u003e42.8 \u003c/td\u003e\n    \u003ctd\u003e72.1\u003c/td\u003e\n    \u003ctd\u003e81.4\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e16.3\u003c/td\u003e\n    \u003ctd\u003e44.1\u003c/td\u003e\n    \u003ctd\u003e70.5\u003c/td\u003e\n    \u003ctd\u003e80.5\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e11.8\u003c/td\u003e\n    \u003ctd\u003e10.5\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003ezero-shot\u003c/td\u003e\n    \u003ctd\u003eViT/B-32\u003c/td\u003e\n    \u003ctd\u003e4\u003c/td\u003e\n    \u003ctd\u003e31.1\u003c/td\u003e\n    \u003ctd\u003e53.7\u003c/td\u003e\n    \u003ctd\u003e63.4\u003c/td\u003e\n    \u003ctd\u003e4.0\u003c/td\u003e\n    \u003ctd\u003e41.6\u003c/td\u003e\n    \u003ctd\u003e26.5\u003c/td\u003e\n    \u003ctd\u003e50.1\u003c/td\u003e\n    \u003ctd\u003e61.7\u003c/td\u003e\n    \u003ctd\u003e5.0\u003c/td\u003e\n    \u003ctd\u003e39.9\u003c/td\u003e\n    \u003ctd\u003e-\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd colspan=\"14\"\u003eInteraction\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eDP+None\u003c/td\u003e\n    \u003ctd\u003eViT/B-32\u003c/td\u003e\n    \u003ctd\u003e4\u003c/td\u003e\n    \u003ctd\u003e42.9\u003c/td\u003e\n    \u003ctd\u003e70.6\u003c/td\u003e\n    \u003ctd\u003e81.4\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e15.4\u003c/td\u003e\n    \u003ctd\u003e43.0\u003c/td\u003e\n    \u003ctd\u003e71.1\u003c/td\u003e\n    \u003ctd\u003e81.1\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e11.8\u003c/td\u003e\n    \u003ctd\u003e2.5\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eDP+seqTransf\u003c/td\u003e\n    \u003ctd\u003eViT/B-32\u003c/td\u003e\n    \u003ctd\u003e4\u003c/td\u003e\n    \u003ctd\u003e42.8\u003c/td\u003e\n    \u003ctd\u003e71.1\u003c/td\u003e\n    \u003ctd\u003e81.1\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e15.6\u003c/td\u003e\n    \u003ctd\u003e44.1\u003c/td\u003e\n    \u003ctd\u003e70.9\u003c/td\u003e\n    \u003ctd\u003e80.9\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e11.7\u003c/td\u003e\n    \u003ctd\u003e2.6\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eXTI+None\u003c/td\u003e\n    \u003ctd\u003eViT/B-32\u003c/td\u003e\n    \u003ctd\u003e4\u003c/td\u003e\n    \u003ctd\u003e40.5\u003c/td\u003e\n    \u003ctd\u003e71.1\u003c/td\u003e\n    \u003ctd\u003e82.6\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e13.6\u003c/td\u003e\n    \u003ctd\u003e42.7\u003c/td\u003e\n    \u003ctd\u003e70.8\u003c/td\u003e\n    \u003ctd\u003e80.2\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e12.5\u003c/td\u003e\n    \u003ctd\u003e14.3\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eXTI+seqTransf\u003c/td\u003e\n    \u003ctd\u003eViT/B-32\u003c/td\u003e\n    \u003ctd\u003e4\u003c/td\u003e\n    \u003ctd\u003e42.4\u003c/td\u003e\n    \u003ctd\u003e71.3\u003c/td\u003e\n    \u003ctd\u003e80.9\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e15.2\u003c/td\u003e\n    \u003ctd\u003e40.1\u003c/td\u003e\n    \u003ctd\u003e69.2\u003c/td\u003e\n    \u003ctd\u003e79.6\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e15.8\u003c/td\u003e\n    \u003ctd\u003e16.8\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eTI+seqTransf\u003c/td\u003e\n    \u003ctd\u003eViT/B-32\u003c/td\u003e\n    \u003ctd\u003e4\u003c/td\u003e\n    \u003ctd\u003e44.8\u003c/td\u003e\n    \u003ctd\u003e73.0\u003c/td\u003e\n    \u003ctd\u003e82.2\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e13.4\u003c/td\u003e\n    \u003ctd\u003e42.6\u003c/td\u003e\n    \u003ctd\u003e72.7\u003c/td\u003e\n    \u003ctd\u003e82.8\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e9.1\u003c/td\u003e\n    \u003ctd\u003e2.6\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eWTI+seqTransf\u003c/td\u003e\n    \u003ctd\u003eViT/B-32\u003c/td\u003e\n    \u003ctd\u003e4\u003c/td\u003e\n    \u003ctd\u003e46.6\u003c/td\u003e\n    \u003ctd\u003e73.4\u003c/td\u003e\n    \u003ctd\u003e83.5\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e13.0\u003c/td\u003e\n    \u003ctd\u003e45.4\u003c/td\u003e\n    \u003ctd\u003e73.4\u003c/td\u003e\n    \u003ctd\u003e81.9\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e9.2\u003c/td\u003e\n    \u003ctd\u003e2.6\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd colspan=\"14\"\u003eChannel DeCorrelation Regularization\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eDP+seqTransf+CDCR\u003c/td\u003e\n    \u003ctd\u003eViT/B-32\u003c/td\u003e\n    \u003ctd\u003e4\u003c/td\u003e\n    \u003ctd\u003e43.9\u003c/td\u003e\n    \u003ctd\u003e71.1\u003c/td\u003e\n    \u003ctd\u003e81.2\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e15.3\u003c/td\u003e\n    \u003ctd\u003e42.3\u003c/td\u003e\n    \u003ctd\u003e70.3\u003c/td\u003e\n    \u003ctd\u003e81.1\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e11.4\u003c/td\u003e\n    \u003ctd\u003e2.6\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eTI+seqTransf+CDCR\u003c/td\u003e\n    \u003ctd\u003eViT/B-32\u003c/td\u003e\n    \u003ctd\u003e4\u003c/td\u003e\n    \u003ctd\u003e45.8\u003c/td\u003e\n    \u003ctd\u003e73.0\u003c/td\u003e\n    \u003ctd\u003e81.9\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e12.8\u003c/td\u003e\n    \u003ctd\u003e43.3\u003c/td\u003e\n    \u003ctd\u003e71.8\u003c/td\u003e\n    \u003ctd\u003e82.7\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e8.9\u003c/td\u003e\n    \u003ctd\u003e2.6\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eWTI+seqTransf+CDCR\u003c/td\u003e\n    \u003ctd\u003eViT/B-32\u003c/td\u003e\n    \u003ctd\u003e4\u003c/td\u003e\n    \u003ctd\u003e47.6\u003c/td\u003e\n    \u003ctd\u003e73.4\u003c/td\u003e\n    \u003ctd\u003e83.3\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e12.8\u003c/td\u003e\n    \u003ctd\u003e45.1\u003c/td\u003e\n    \u003ctd\u003e72.9\u003c/td\u003e\n    \u003ctd\u003e83.5\u003c/td\u003e\n    \u003ctd\u003e2.0\u003c/td\u003e\n    \u003ctd\u003e9.2\u003c/td\u003e\n    \u003ctd\u003e2.6\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\nNote: the performances are slight boosts due to new hyperparameters.\n\n\n### Visualization demo\n\nRun our visualization demo using [matplotlib](demo/show_wti.py) (no GPU needed):\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"demo/wti.png\" width=\"400\"\u003e\n\u003c/p\u003e\n\n### License\n\nSee [LICENSE](LICENSE) for details.\n\n\n### Acknowledgments\nOur code is partly based on [CLIP4Clip](https://github.com/ArrowLuo/CLIP4Clip).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffoolwood%2Fdrl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffoolwood%2Fdrl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffoolwood%2Fdrl/lists"}