{"id":14286839,"url":"https://github.com/CVIR/CoMix","last_synced_at":"2025-08-15T07:31:46.647Z","repository":{"id":67247232,"uuid":"441225126","full_name":"CVIR/CoMix","owner":"CVIR","description":"This repository contains the official implementation of CoMix (NeurIPS 2021) https://arxiv.org/pdf/2110.15128.pdf.","archived":false,"fork":false,"pushed_at":"2022-01-12T13:11:55.000Z","size":46871,"stargazers_count":19,"open_issues_count":1,"forks_count":7,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-08-24T17:26:21.766Z","etag":null,"topics":["action-recognition","computer-vision","contrastive-learning","domain-adaptation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CVIR.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-12-23T15:41:57.000Z","updated_at":"2023-09-05T03:04:24.000Z","dependencies_parsed_at":null,"dependency_job_id":"df439ffb-f867-4cb2-bc53-a94dc184b218","html_url":"https://github.com/CVIR/CoMix","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CVIR%2FCoMix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CVIR%2FCoMix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CVIR%2FCoMix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CVIR%2FCoMix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CVIR","download_url":"https://codeload.github.com/CVIR/CoMix/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229899538,"owners_count":18141525,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["action-recognition","computer-vision","contrastive-learning","domain-adaptation"],"created_at":"2024-08-23T17:01:03.637Z","updated_at":"2024-12-16T02:30:26.589Z","avatar_url":"https://github.com/CVIR.png","language":"Python","funding_links":[],"categories":["Training Paradigms"],"sub_categories":["**Domain Adaption**"],"readme":"# Contrast and Mix (CoMix)\r\nThe repository contains the codes for the paper **Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing** part of Advances in Neural Information Processing Systems (NeurIPS) 2021. \r\n\r\n[Aadarsh Sahoo\u003csup\u003e1\u003c/sup\u003e](https://aadsah.github.io/), [Rutav Shah\u003csup\u003e1\u003c/sup\u003e](https://www.linkedin.com/in/rutav-shah-01a2941a7/?originalSubdomain=in), [Rameswar Panda\u003csup\u003e2\u003c/sup\u003e](https://rpand002.github.io/), [Kate Saenko\u003csup\u003e2,3\u003c/sup\u003e](http://ai.bu.edu/ksaenko.html), [Abir Das\u003csup\u003e1\u003c/sup\u003e](https://cse.iitkgp.ac.in/~adas/)\r\n\r\n\u003csup\u003e1\u003c/sup\u003e IIT Kharagpur, \u003csup\u003e2\u003c/sup\u003e MIT-IBM Watson AI Lab, \u003csup\u003e3\u003c/sup\u003e Boston University\r\n\r\n[[Paper]](https://openreview.net/pdf?id=a1wQOh27zcy) [[Project Page]](https://cvir.github.io/projects/comix)\r\n\r\n\u0026nbsp;\r\n\u003cp align=\"center\"\u003e\r\n\u003cimg src='./Images/Framework.png' width=800\u003e\r\n\u003c/p\u003e\r\n\u003cp align=\"center\"\u003e\r\n\u003cb\u003eFig. Temporal Contrastive Learning with Background Mixing and Target Pseudo-labels.\u003c/b\u003e Temporal contrastive loss (left) contrasts a single temporally augmented positive (same video, different speed) per anchor against rest of the videos in a mini-batch as negatives. Incorporating background mixing (middle) provides additional positives per anchor possessing same action semantics with a different background alleviating background shift across domains. Incorporating target pseudo-labels (right) additionally enhances the discriminabilty by contrasting the target videos with the same pseudo-label as positives against rest of the videos as negatives. \u003c/p\u003e\r\n\u0026nbsp;\r\n\r\n\r\n### Preparing the Environment\r\n\r\n#### Conda \r\nPlease use the `comix_environment.yml` file to create the conda environment `comix` as:\r\n\r\n```\r\nconda env create -f comix_environment.yml\r\n```\r\n\r\n#### Pip\r\nPlease use the `requirements.txt` file to install all the required dependencies as:\r\n\r\n```\r\npip install -r requirements.txt\r\n```\r\n\r\n\r\n### Data Directory Structure\r\nAll the datasets should be stored in the folder `./data` following the convention `./data/\u003cdataset_name\u003e` and it must be passed as an argument to `base_dir=./data/\u003cdataset_name\u003e`. \r\n##### UCF - HMDB\r\nFor `ucf_hmdb` dataset with `base_dir=./data/ucf_hmdb` the structure would be as follows:\r\n\r\n    .\r\n    ├── ...\r\n    ├── data\r\n    │   ├── ucf_hmdb\r\n    │   │   ├── ucf_videos\r\n    |   |   |   ├── \u003cvideo-1\u003e\r\n    |   |   |   |   ├── \u003cframe-1\u003e\r\n    |   |   |   |   ├── \u003cframe-2\u003e\r\n    |   |   |   |   ├── ...\r\n    |   |   |   ├── \u003cvideo-2\u003e\r\n    |   |   |   ├── ...\r\n    │   │   ├── hmdb_videos\r\n    |   |   ├── ucf_BG\r\n    |   |   └── hmdb_BG\r\n    │   └──\r\n    └──\r\n    \r\n##### Jester\r\nFor `Jester` dataset with `base_dir=./data/jester` the structure would be as follows \r\n\r\n    .\r\n    ├── ...\r\n    ├── data\r\n    │   ├── jester\r\n    |   |   ├── jester_videos\r\n    |   |   |   ├── \u003cvideo-1\u003e\r\n    |   |   |   |   ├── \u003cframe-1\u003e\r\n    |   |   |   |   ├── \u003cframe-2\u003e\r\n    |   |   |   |   ├── ...\r\n    |   |   |   ├── \u003cvideo-2\u003e\r\n    |   |   |   ├── ...\r\n    |   |   ├── jester_BG\r\n    |   |   |   ├── \u003cvideo-1\u003e\r\n    |   |   |   |   ├── \u003cBackgorund frame\u003e\r\n    |   |   |   ├── ...\r\n    └── └── └──\r\n\r\n##### Epic-Kitchens\r\nFor `Epic Kitchens` dataset with `base_dir=./data/epic_kitchens` the structure would be as follows (we follow the same structure as in the original dataset) : \r\n\r\n    .\r\n    ├── ...\r\n    ├── data\r\n    │   ├── epic_kitchens\r\n    |   |   ├── epic_kitchens_videos\r\n    |   |   |   ├── train\r\n    |   |   |   |   ├── D1\r\n    |   |   |   |   |   ├── \u003cvideo-1\u003e\r\n    |   |   |   |   |   |   ├── \u003cframe-1\u003e\r\n    |   |   |   |   |   |   ├── \u003cframe-2\u003e\r\n    |   |   |   |   |   |   ├── ...\r\n    |   |   |   |   |   ├── \u003cvideo-2\u003e\r\n    |   |   |   |   |   ├── ...\r\n    |   |   |   |   ├── D2\r\n    |   |   |   |   └── D3\r\n    |   |   |   └── test\r\n    └── └── └── epic_kitchens_BG\r\n\r\n\r\nFor using datasets stored in some other directories, please pass the parameter `base_dir` accordingly. \r\n\r\n### Background Extraction using Temporal Median Filtering\r\n\r\nPlease refer to the folder `./background_extraction` for the codes to extract backgrounds using temporal median filtering.\r\n\r\n\r\n### Data\r\n\r\nAll the required split files are provided inside the directory `./video_splits`.\r\n\r\nThe official download links for the datasets used for this paper are: [[UCF-101]](https://www.crcv.ucf.edu/data/UCF101.php) [[HMDB-51]](https://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/) [[Jester]](https://20bn.com/datasets/jester) [[Epic Kitchens]](https://epic-kitchens.github.io/2021)\r\n\r\n\r\n### Training CoMix\r\n\r\nHere are some of the sample and recomended commands to train CoMix for the transfer task of:\r\n\r\n\r\n `UCF -\u003e HMDB` from `UCF-HMDB` dataset:\r\n\r\n```\r\nCUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --manual_seed 1 --dataset_name UCF-HMDB --src_dataset UCF --tgt_dataset HMDB --batch_size 8 --model_root ./checkpoints_ucf_hmdb --save_in_steps 500 --log_in_steps 50 --eval_in_steps 50 --pseudo_threshold 0.7 --warmstart_models True --num_iter_warmstart 4000 --num_iter_adapt 10000 --learning_rate 0.01 --learning_rate_ws 0.01 --lambda_bgm 0.1 --lambda_tpl 0.01 --base_dir ./data/ucf_hmdb\r\n```\r\n\r\n\r\n `S -\u003e T` from `Jester` dataset:\r\n\r\n```\r\nCUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --manual_seed 1 --dataset_name Jester --src_dataset S --tgt_dataset T --batch_size 8 --model_root ./checkpoints_jester --save_in_steps 500 --log_in_steps 50 --eval_in_steps 50 --pseudo_threshold 0.7 --warmstart_models True --num_iter_warmstart 4000 --num_iter_adapt 10000 --learning_rate 0.01 --learning_rate_ws 0.01 --lambda_bgm 0.1 --lambda_tpl 0.1 --base_dir ./data/jester\r\n```\r\n\r\n\r\n `D1 -\u003e D2` from `Epic-Kitchens` dataset:\r\n\r\n```\r\nCUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --manual_seed 1 --dataset_name Epic-Kitchens --src_dataset D1 --tgt_dataset D2 --batch_size 8 --model_root ./checkpoints_epic_d1_d2 --save_in_steps 500 --log_in_steps 50 --eval_in_steps 50 --pseudo_threshold 0.7 --warmstart_models True --num_iter_warmstart 4000 --num_iter_adapt 10000 --learning_rate 0.01 --learning_rate_ws 0.01 --lambda_bgm 0.01 --lambda_tpl 0.01 --base_dir ./data/epic_kitchens\r\n```\r\n\r\n\r\nFor detailed description regarding the arguments, use:\r\n\r\n```\r\npython main.py --help\r\n```\r\n\r\n### Citing CoMix\r\n\r\nIf you use codes in this repository, consider citing CoMix. Thanks!\r\n\r\n```\r\n@article{sahoo2021contrast,\r\n  title={Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing},\r\n  author={Sahoo, Aadarsh and Shah, Rutav and Panda, Rameswar and Saenko, Kate and Das, Abir},\r\n  journal={Advances in Neural Information Processing Systems},\r\n  volume={34},\r\n  year={2021}\r\n}\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCVIR%2FCoMix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FCVIR%2FCoMix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCVIR%2FCoMix/lists"}