{"id":13545224,"url":"https://github.com/cmhungsteve/TA3N","last_synced_at":"2025-04-02T15:31:01.288Z","repository":{"id":39717736,"uuid":"188935240","full_name":"cmhungsteve/TA3N","owner":"cmhungsteve","description":"[ICCV 2019 (Oral)] Temporal Attentive Alignment for Large-Scale Video Domain Adaptation (PyTorch)","archived":false,"fork":false,"pushed_at":"2024-11-22T22:18:57.000Z","size":1757,"stargazers_count":262,"open_issues_count":3,"forks_count":40,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-27T21:11:08.069Z","etag":null,"topics":["action-recognition","cvpr2019","domain-adaptation","domain-discrepancy","iccv2019","pytorch","temporal-dynamics","video","video-classification","video-da-datasets"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1907.12743","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cmhungsteve.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-28T01:56:42.000Z","updated_at":"2025-02-01T23:57:55.000Z","dependencies_parsed_at":"2024-01-30T02:29:37.987Z","dependency_job_id":"ef2c386f-25a2-468f-908c-a5460e1e00f8","html_url":"https://github.com/cmhungsteve/TA3N","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmhungsteve%2FTA3N","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmhungsteve%2FTA3N/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmhungsteve%2FTA3N/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmhungsteve%2FTA3N/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cmhungsteve","download_url":"https://codeload.github.com/cmhungsteve/TA3N/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246841616,"owners_count":20842621,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["action-recognition","cvpr2019","domain-adaptation","domain-discrepancy","iccv2019","pytorch","temporal-dynamics","video","video-classification","video-da-datasets"],"created_at":"2024-08-01T11:00:59.343Z","updated_at":"2025-04-02T15:30:59.999Z","avatar_url":"https://github.com/cmhungsteve.png","language":"Python","funding_links":[],"categories":["Python","Other Topics in Video Transfer Learning"],"sub_categories":[],"readme":"# Temporal Attentive Alignment for Video Domain Adaptation\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/temporal-attentive-alignment-for-large-scale/domain-adaptation-on-hmdb-ucf-full)](https://paperswithcode.com/sota/domain-adaptation-on-hmdb-ucf-full?p=temporal-attentive-alignment-for-large-scale)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/temporal-attentive-alignment-for-large-scale/domain-adaptation-on-ucf-hmdb-full)](https://paperswithcode.com/sota/domain-adaptation-on-ucf-hmdb-full?p=temporal-attentive-alignment-for-large-scale)\n\n---\n[\u003cimg align=\"right\" src=\"webpage/OLIVES_new.png\" width=\"15%\"\u003e](https://ghassanalregib.info/)\nThis work was mainly done in the [**Omni Lab for Intelligent Visual Engineering and Science (OLIVES)**](https://ghassanalregib.info/) @ Georgia Tech. \u003cbr\u003e\nFeel free to check our lab's [**Website**](https://ghassanalregib.info/) and [**GitHub**](https://github.com/olivesgatech) for other interesting work!!!\n\n---\nThis is the official PyTorch implementation of our papers:\n\n**Temporal Attentive Alignment for Large-Scale Video Domain Adaptation**  \n[__***Min-Hung Chen***__](https://www.linkedin.com/in/chensteven), [Zsolt Kira](https://www.cc.gatech.edu/~zk15/), [Ghassan AlRegib (Advisor)](https://ghassanalregib.info/), [Jaekwon Yoo](https://www.linkedin.com/in/jaekwon-yoo-8685862b/), [Ruxin Chen](https://www.linkedin.com/in/ruxin-chen-991477119/), [Jian Zheng](https://www.linkedin.com/in/jian-zheng/)  \n[*International Conference on Computer Vision (ICCV), 2019*](http://iccv2019.thecvf.com/) **[Oral (acceptance rate: 4.6%)]**  \n[[arXiv](https://arxiv.org/abs/1907.12743)][[Project](https://minhungchen.netlify.app/project/cdar/)][[Blog](https://mlatgt.blog/2019/09/10/overcoming-large-scale-annotation-requirements-for-understanding-videos-in-the-wild/)][[Presentation (officially recorded)](https://conftube.com/video/8oUPyhwzIDo?tocitem=146)][[Oral](https://youtu.be/j9cDuzmpYP8)][[Poster](webpage/ICCV2019_Steve_TA3N_poster_v1_2.pdf)][[Slides](https://www.dropbox.com/s/s9ud77a1zt0vqbn/Oral_TA3N_ICCV_2019_mute.pdf?dl=0)][[Open Access](http://openaccess.thecvf.com/content_ICCV_2019/html/Chen_Temporal_Attentive_Alignment_for_Large-Scale_Video_Domain_Adaptation_ICCV_2019_paper.html)][[IEEE Xplore](https://ieeexplore.ieee.org/document/9008391)]\n\n**Temporal Attentive Alignment for Video Domain Adaptation**  \n[__***Min-Hung Chen***__](https://www.linkedin.com/in/chensteven), [Zsolt Kira](https://www.cc.gatech.edu/~zk15/), [Ghassan AlRegib (Advisor)](https://ghassanalregib.info/)  \n[*CVPR Workshop (Learning from Unlabeled Videos), 2019*](https://sites.google.com/view/luv2019)  \n[[arXiv](https://arxiv.org/abs/1905.10861)]\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"webpage/Overview.png?raw=true\" width=\"60%\"\u003e\n\u003c/p\u003e\n\nAlthough various image-based domain adaptation (DA) techniques have been proposed in recent years, domain shift in videos is still not well-explored. Most previous\nworks only evaluate performance on small-scale datasets which are saturated. Therefore, we first propose two largescale video DA datasets with much larger domain discrepancy: **UCF-HMDB\u003csub\u003efull\u003c/sub\u003e** and **Kinetics-Gameplay**. Second, we investigate different DA integration methods for videos, and show that simultaneously aligning and learning temporal dynamics achieves effective alignment even without sophisticated DA methods. Finally, we propose **Temporal Attentive Adversarial Adaptation Network (TA\u003csup\u003e3\u003c/sup\u003eN)**, which explicitly attends to the temporal dynamics using domain\ndiscrepancy for more effective domain alignment, achieving state-of-the-art performance on four video DA datasets.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"webpage/SOTA_small.png?raw=true\" width=\"49%\"\u003e\n\u003cimg src=\"webpage/SOTA_large.png?raw=true\" width=\"50%\"\u003e\n\u003c/p\u003e\n\n---\n## Contents\n* [Requirements](#requirements)\n* [Dataset Preparation](#dataset-preparation)\n  * [Data structure](#data-structure)\n  * [File lists for training/validation](#file-lists-for-trainingvalidation)\n  * [Input data](#input-data)\n* [Usage](#usage)\n  * [Training](#training)\n  * [Testing](#testing)\n\u003c!--   * [Video Demo](#video-demo) --\u003e\n* [Options](#options)\n  * [Domain Adaptation](#domain-adaptation)\n  * [More options](#more-options)\n* [Citation](#citation)\n* [Contact](#contact)\n\n---\n## Requirements\n* support Python 3.6, PyTorch 0.4, CUDA 9.0, CUDNN 7.1.4\n* install all the library with: `pip install -r requirements.txt`\n\n---\n## Dataset Preparation\n### Data structure\nYou need to extract frame-level features for each video to run the codes. To extract features, please check [`dataset_preparation/`](dataset_preparation/).\n\nFolder Structure:\n```\nDATA_PATH/\n  DATASET/\n    list_DATASET_SUFFIX.txt\n    RGB/\n      CLASS_01/\n        VIDEO_0001.mp4\n        VIDEO_0002.mp4\n        ...\n      CLASS_02/\n      ...\n\n    RGB-Feature/\n      VIDEO_0001/\n        img_00001.t7\n        img_00002.t7\n        ...\n      VIDEO_0002/\n      ...\n```\n`RGB-Feature/` contains all the feature vectors for training/testing. `RGB/` contains all the raw videos.\n\nThere should be at least two `DATASET` folders: source training set  and validation set. If you want to do domain adaption, you need to have another `DATASET`: target training set.\n\n### File lists for training/validation\nThe file list `list_DATASET_SUFFIX.txt` is required for data feeding. Each line in the list contains the full path of the video folder, video frame number, and video class index. It looks like:\n```\nDATA_PATH/DATASET/RGB-Feature/VIDEO_0001/ 100 0\nDATA_PATH/DATASET/RGB-Feature/VIDEO_0002/ 150 1\n......\n```\nTo generate the file list, please check [`dataset_preparation/`](dataset_preparation/).\n\n### Input data\nHere we provide pre-extracted features and data list files, so you can skip the above two steps and directly try our training/testing codes. You may need to manually edit the path in the data list files.\n* Features\n  * UCF: [download](https://www.dropbox.com/s/ebtc1hz1paz9bmz/ucf101-feat.zip?dl=0)\n  * HMDB: [download](https://www.dropbox.com/s/aiac0ytb9jt83a2/hmdb51-feat.zip?dl=0)\n  * Olympic: [training](https://www.dropbox.com/s/0ljfsp52hydyqht/olympic_train-feat.zip?dl=0) | [validation](https://www.dropbox.com/s/yh09a2th4hf8hqp/olympic_val-feat.zip?dl=0)\n* Data lists\n  * UCF-Olympic\n    * UCF: [training list](https://www.dropbox.com/s/du8d3qrzs9h8phn/list_ucf101_train_ucf_olympic-feature.txt?dl=0) | [validation list](https://www.dropbox.com/s/0qrhuen3o27g9k5/list_ucf101_val_ucf_olympic-feature.txt?dl=0)\n    * Olympic: [training list](https://www.dropbox.com/s/0eafz1kjk71i0i9/list_olympic_train_ucf_olympic-feature.txt?dl=0) | [validation list](https://www.dropbox.com/s/ku27uniw4xm7wpm/list_olympic_val_ucf_olympic-feature.txt?dl=0)\n  * UCF-HMDB\u003csub\u003esmall\u003c/sub\u003e\n    * UCF: [training list](https://www.dropbox.com/s/2g04infpxwysjfb/list_ucf101_train_hmdb_ucf_small-feature.txt?dl=0) | [validation list](https://www.dropbox.com/s/6fjour5n1dcabfy/list_ucf101_val_hmdb_ucf_small-feature.txt?dl=0)\n    * HMDB: [training list](https://www.dropbox.com/s/q6e7jwhr1ktmrrt/list_hmdb51_train_hmdb_ucf_small-feature.txt?dl=0) | [validation list](https://www.dropbox.com/s/qh3h619bdo2q3h1/list_hmdb51_val_hmdb_ucf_small-feature.txt?dl=0)\n  * UCF-HMDB\u003csub\u003efull\u003c/sub\u003e\n    * UCF: [training list](https://www.dropbox.com/s/jrahoh6u8k90iec/list_ucf101_train_hmdb_ucf-feature.txt?dl=0) | [validation list](https://www.dropbox.com/s/7359sfsflfkf60c/list_ucf101_val_hmdb_ucf-feature.txt?dl=0)\n    * HMDB: [training list](https://www.dropbox.com/s/thj7mkzof6pgfmj/list_hmdb51_train_hmdb_ucf-feature.txt?dl=0) | [validation list](https://www.dropbox.com/s/s9yc43u87kjcdhx/list_hmdb51_val_hmdb_ucf-feature.txt?dl=0)\n\n* Kinetics-Gameplay: please fill [**this form**](https://forms.gle/bziHhvQJGmi7hwF26) to request the features and data lists. \u003cbr\u003e\n\u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e\u003cimg alt=\"Creative Commons License\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png\" /\u003e\u003c/a\u003e\nThe Kinetics-Gameplay dataset is licensed under \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003eCC BY-NC-SA 4.0\u003c/a\u003e for non-commercial purposes only.\n[![License: CC BY-NC-SA 4.0](https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc-sa/4.0/)\n\n---\n## Usage\n* training/validation: Run `./script_train_val.sh`\n\u003c!-- * demo video: Run `./script_demo_video.sh` --\u003e\n\nAll the commonly used variables/parameters have comments in the end of the line. Please check [Options](#options).\n\n#### Training\nAll the outputs will be under the directory `exp_path`.\n* Outputs:\n  * model weights: `checkpoint.pth.tar`, `model_best.pth.tar`\n  * log files: `train.log`, `train_short.log`, `val.log`, `val_short.log`\n\n#### Testing\nYou can choose one of model_weights for testing. All the outputs will be under the directory `exp_path`.\n\n* Outputs:\n  * score_data: used to check the model output (`scores_XXX.npz`)\n  * confusion matrix: `confusion_matrix_XXX.png` and `confusion_matrix_XXX-topK.txt`\n\n\u003c!-- #### Video Demo\n`demo_video.py` overlays the predicted categories and confidence values on one video. Please see \"Results\". --\u003e\n\n---\n## Options\n#### Domain Adaptation\n\u003c!-- In both `./script_train_val.sh` and `./script_demo_video.sh`, there are several options related to our Domain Adaptation approaches. --\u003e\nIn `./script_train_val.sh`, there are several options related to our DA approaches.\n* `use_target`: switch on/off the DA mode\n  * `none`: not use target data (no DA)\n  * `uSv`/`Sv`: use target data in a unsupervised/supervised way\n\n\u003c!-- * options for the DA approaches:\n  * discrepancy-based: DAN, JAN\n  * adversarial-based: RevGrad\n  * Normalization-based: AdaBN\n  * Ensemble-based: MCD --\u003e\n\n#### More options\nFor more details of all the arguments, please check [opts.py](opts.py).\n\n#### Notes\nThe options in the scripts have comments with the following types:\n* no comment: user can still change it, but NOT recommend (may need to change the code or have different experimental results)\n* comments with choices (e.g. `true | false`): can only choose from choices\n* comments as `depend on users`: totally depend on users (mostly related to data path)\n\n---\n## Citation\nIf you find this repository useful, please cite our papers:\n```\n@inproceedings{chen2019temporal,\n  title={Temporal attentive alignment for large-scale video domain adaptation},\n  author={Chen, Min-Hung and Kira, Zsolt and AlRegib, Ghassan and Woo, Jaekwon and Chen, Ruxin and Zheng, Jian},\n  booktitle={IEEE International Conference on Computer Vision (ICCV)},\n  year={2019},\n  url={https://arxiv.org/abs/1907.12743}\n}\n\n@article{chen2019taaan,\n  title={Temporal Attentive Alignment for Video Domain Adaptation},\n  author={Chen, Min-Hung and Kira, Zsolt and AlRegib, Ghassan},\n  journal={CVPR Workshop on Learning from Unlabeled Videos},\n  year={2019},\n  url={https://arxiv.org/abs/1905.10861}\n}\n```\n\n---\n### Acknowledgments\nThis work was mainly done in [OLIVES](https://ghassanalregib.info/)@GT with the guidance of Prof. [Ghassan AlRegib](https://ghassanalregib.info/), and the collaboration with Prof. [Zsolt Kira](https://www.cc.gatech.edu/~zk15/) at Georgia Tech. Part of this work was done with the collaboration with [Jaekwon Yoo](https://www.linkedin.com/in/jaekwon-yoo-8685862b/), [Ruxin Chen](https://www.linkedin.com/in/ruxin-chen-991477119/) and [Jian Zheng](https://www.linkedin.com/in/jian-zheng/).\n\u003c!-- in [Sony Interactive Entertainment (SIE)](https://www.sie.com/en/index.html)  --\u003e\n\nSome codes are borrowed from [TSN](https://github.com/yjxiong/temporal-segment-networks), [pytorch-tsn](https://github.com/yjxiong/tsn-pytorch), [TRN-pytorch](https://github.com/metalbubble/TRN-pytorch), and [Xlearn](https://github.com/thuml/Xlearn/tree/master/pytorch).\n\nSpecial thanks to the development team for the product used in the Kinetics-Gameplay dataset: \u003cbr\u003e\n**Detroit: Become Human™ ©Sony Interactive Entertainment Europe, developed by Quantic Dream**\n\n---\n### Contact\n[Min-Hung Chen](https://www.linkedin.com/in/chensteven) \u003cbr\u003e\ncmhungsteve AT gatech DOT edu \u003cbr\u003e\n[\u003cimg align=\"left\" src=\"webpage/OLIVES_new.png\" width=\"15%\"\u003e](https://ghassanalregib.info/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmhungsteve%2FTA3N","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcmhungsteve%2FTA3N","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmhungsteve%2FTA3N/lists"}