Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/CVIR/CoMix
This repository contains the official implementation of CoMix (NeurIPS 2021) https://arxiv.org/pdf/2110.15128.pdf.
https://github.com/CVIR/CoMix
action-recognition computer-vision contrastive-learning domain-adaptation
Last synced: 29 days ago
JSON representation
This repository contains the official implementation of CoMix (NeurIPS 2021) https://arxiv.org/pdf/2110.15128.pdf.
- Host: GitHub
- URL: https://github.com/CVIR/CoMix
- Owner: CVIR
- License: apache-2.0
- Created: 2021-12-23T15:41:57.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2022-01-12T13:11:55.000Z (over 2 years ago)
- Last Synced: 2023-10-20T19:08:59.071Z (11 months ago)
- Topics: action-recognition, computer-vision, contrastive-learning, domain-adaptation
- Language: Python
- Homepage:
- Size: 44.7 MB
- Stars: 19
- Watchers: 2
- Forks: 6
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- Awesome-Mixup - [Code
README
# Contrast and Mix (CoMix)
The repository contains the codes for the paper **Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing** part of Advances in Neural Information Processing Systems (NeurIPS) 2021.
[Aadarsh Sahoo1](https://aadsah.github.io/), [Rutav Shah1](https://www.linkedin.com/in/rutav-shah-01a2941a7/?originalSubdomain=in), [Rameswar Panda2](https://rpand002.github.io/), [Kate Saenko2,3](http://ai.bu.edu/ksaenko.html), [Abir Das1](https://cse.iitkgp.ac.in/~adas/)
1 IIT Kharagpur, 2 MIT-IBM Watson AI Lab, 3 Boston University
[[Paper]](https://openreview.net/pdf?id=a1wQOh27zcy) [[Project Page]](https://cvir.github.io/projects/comix)
Fig. Temporal Contrastive Learning with Background Mixing and Target Pseudo-labels. Temporal contrastive loss (left) contrasts a single temporally augmented positive (same video, different speed) per anchor against rest of the videos in a mini-batch as negatives. Incorporating background mixing (middle) provides additional positives per anchor possessing same action semantics with a different background alleviating background shift across domains. Incorporating target pseudo-labels (right) additionally enhances the discriminabilty by contrasting the target videos with the same pseudo-label as positives against rest of the videos as negatives.
### Preparing the Environment
#### Conda
Please use the `comix_environment.yml` file to create the conda environment `comix` as:
```
conda env create -f comix_environment.yml
```
#### Pip
Please use the `requirements.txt` file to install all the required dependencies as:
```
pip install -r requirements.txt
```
### Data Directory Structure
All the datasets should be stored in the folder `./data` following the convention `./data/` and it must be passed as an argument to `base_dir=./data/`.
##### UCF - HMDB
For `ucf_hmdb` dataset with `base_dir=./data/ucf_hmdb` the structure would be as follows:
.
├── ...
├── data
│ ├── ucf_hmdb
│ │ ├── ucf_videos
| | | ├──
| | | | ├──
| | | | ├──
| | | | ├── ...
| | | ├──
| | | ├── ...
│ │ ├── hmdb_videos
| | ├── ucf_BG
| | └── hmdb_BG
│ └──
└──
##### Jester
For `Jester` dataset with `base_dir=./data/jester` the structure would be as follows
.
├── ...
├── data
│ ├── jester
| | ├── jester_videos
| | | ├──
| | | | ├──
| | | | ├──
| | | | ├── ...
| | | ├──
| | | ├── ...
| | ├── jester_BG
| | | ├──
| | | | ├──
| | | ├── ...
└── └── └──
##### Epic-Kitchens
For `Epic Kitchens` dataset with `base_dir=./data/epic_kitchens` the structure would be as follows (we follow the same structure as in the original dataset) :
.
├── ...
├── data
│ ├── epic_kitchens
| | ├── epic_kitchens_videos
| | | ├── train
| | | | ├── D1
| | | | | ├──
| | | | | | ├──
| | | | | | ├──
| | | | | | ├── ...
| | | | | ├──
| | | | | ├── ...
| | | | ├── D2
| | | | └── D3
| | | └── test
└── └── └── epic_kitchens_BG
For using datasets stored in some other directories, please pass the parameter `base_dir` accordingly.
### Background Extraction using Temporal Median Filtering
Please refer to the folder `./background_extraction` for the codes to extract backgrounds using temporal median filtering.
### Data
All the required split files are provided inside the directory `./video_splits`.
The official download links for the datasets used for this paper are: [[UCF-101]](https://www.crcv.ucf.edu/data/UCF101.php) [[HMDB-51]](https://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/) [[Jester]](https://20bn.com/datasets/jester) [[Epic Kitchens]](https://epic-kitchens.github.io/2021)
### Training CoMix
Here are some of the sample and recomended commands to train CoMix for the transfer task of:
`UCF -> HMDB` from `UCF-HMDB` dataset:
```
CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --manual_seed 1 --dataset_name UCF-HMDB --src_dataset UCF --tgt_dataset HMDB --batch_size 8 --model_root ./checkpoints_ucf_hmdb --save_in_steps 500 --log_in_steps 50 --eval_in_steps 50 --pseudo_threshold 0.7 --warmstart_models True --num_iter_warmstart 4000 --num_iter_adapt 10000 --learning_rate 0.01 --learning_rate_ws 0.01 --lambda_bgm 0.1 --lambda_tpl 0.01 --base_dir ./data/ucf_hmdb
```
`S -> T` from `Jester` dataset:
```
CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --manual_seed 1 --dataset_name Jester --src_dataset S --tgt_dataset T --batch_size 8 --model_root ./checkpoints_jester --save_in_steps 500 --log_in_steps 50 --eval_in_steps 50 --pseudo_threshold 0.7 --warmstart_models True --num_iter_warmstart 4000 --num_iter_adapt 10000 --learning_rate 0.01 --learning_rate_ws 0.01 --lambda_bgm 0.1 --lambda_tpl 0.1 --base_dir ./data/jester
```
`D1 -> D2` from `Epic-Kitchens` dataset:
```
CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --manual_seed 1 --dataset_name Epic-Kitchens --src_dataset D1 --tgt_dataset D2 --batch_size 8 --model_root ./checkpoints_epic_d1_d2 --save_in_steps 500 --log_in_steps 50 --eval_in_steps 50 --pseudo_threshold 0.7 --warmstart_models True --num_iter_warmstart 4000 --num_iter_adapt 10000 --learning_rate 0.01 --learning_rate_ws 0.01 --lambda_bgm 0.01 --lambda_tpl 0.01 --base_dir ./data/epic_kitchens
```
For detailed description regarding the arguments, use:
```
python main.py --help
```
### Citing CoMix
If you use codes in this repository, consider citing CoMix. Thanks!
```
@article{sahoo2021contrast,
title={Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing},
author={Sahoo, Aadarsh and Shah, Rutav and Panda, Rameswar and Saenko, Kate and Das, Abir},
journal={Advances in Neural Information Processing Systems},
volume={34},
year={2021}
}
```