{"id":19932248,"url":"https://github.com/amazon-science/tubelet-transformer","last_synced_at":"2025-05-03T11:31:51.437Z","repository":{"id":58509208,"uuid":"499308664","full_name":"amazon-science/tubelet-transformer","owner":"amazon-science","description":"This is an official implementation of TubeR: Tubelet Transformer for Video Action Detection","archived":false,"fork":false,"pushed_at":"2022-09-27T13:59:12.000Z","size":9874,"stargazers_count":34,"open_issues_count":10,"forks_count":11,"subscribers_count":1,"default_branch":"main","last_synced_at":"2023-03-11T11:52:35.817Z","etag":null,"topics":["action-detection","ava","jhmdb","transformer","tubelet-transformer","tuber","ucf"],"latest_commit_sha":null,"homepage":"https://openaccess.thecvf.com/content/CVPR2022/supplemental/Zhao_TubeR_Tubelet_Transformer_CVPR_2022_supplemental.pdf","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amazon-science.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-02T22:28:21.000Z","updated_at":"2023-03-01T02:09:06.000Z","dependencies_parsed_at":"2023-01-18T23:31:05.907Z","dependency_job_id":null,"html_url":"https://github.com/amazon-science/tubelet-transformer","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Ftubelet-transformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Ftubelet-transformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Ftubelet-transformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Ftubelet-transformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amazon-science","download_url":"https://codeload.github.com/amazon-science/tubelet-transformer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224360230,"owners_count":17298319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["action-detection","ava","jhmdb","transformer","tubelet-transformer","tuber","ucf"],"created_at":"2024-11-12T23:09:29.704Z","updated_at":"2024-11-12T23:09:30.351Z","avatar_url":"https://github.com/amazon-science.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TubeR: Tubelet Transformer for Video Action Detection\n\nThis repo contains the supported code to reproduce spatio-temporal action detection results of [TubeR: Tubelet Transformer for Video Action Detection](https://openaccess.thecvf.com/content/CVPR2022/papers/Zhao_TubeR_Tubelet_Transformer_for_Video_Action_Detection_CVPR_2022_paper.pdf). \n\n## Updates\n\n***08/08/2022*** Initial commits\n\n## Results and Models\n\n### AVA 2.1 Dataset\n\n| Backbone | Pretrain |  #view | mAP  |  FLOPs | config |  model |\n| :---: | :---: |  :---: |:----:| :---: | :---: | :---: |\n| CSN-50 | Kinetics-400 | 1 view | 27.2 |  78G | [config](configuration/TubeR_CSN50_AVA21.yaml) |  [S3](https://yzaws-data-log.s3.amazonaws.com/shared/TubeR_cvpr22/TubeR_CSN50_AVA21.pth) |\n| CSN-50 (with long-term context) | Kinetics-400 | 1 view | 28.8 |  78G | [config](TBD) |  Comming soon |\n| CSN-152 | Kinetics-400+IG65M | 1 view | 29.7 |  120G | [config](configuration/TubeR_CSN152_AVA21.yaml) |  [S3](https://yzaws-data-log.s3.amazonaws.com/shared/TubeR_cvpr22/TubeR_CSN152_AVA21.pth) |\n| CSN-152 (with long-term context) | Kinetics-400+IG65M | 1 view | 31.7 |  120G | [config](TBD) |  Comming soon |\n\n\n### AVA 2.2 Dataset\n\n| Backbone | Pretrain |  #view | mAP  |  FLOPs | config |  model |\n| :---: | :---: |  :---: |:----:| :---: | :---: | :---: |\n| CSN-152 | Kinetics-400+IG65M | 1 view | 31.1 |  120G | [config](configuration/TubeR_CSN152_AVA22.yaml) |  [S3](https://yzaws-data-log.s3.amazonaws.com/shared/TubeR_cvpr22/TubeR_CSN152_AVA22.pth) |\n| CSN-152 (with long-term context) | Kinetics-400+IG65M | 1 view | 33.4 |  120G | [config](TBD) |  Comming soon |\n\n### JHMDB Dataset\n| Backbone |  #view | mAP@0.2 |  mAP@0.5 | config |  model |\n| :---: |  :---: | :---: | :---: | :---: | :---: |\n| CSN-152  | 1 view | 87.4 |  82.3 | [config](configuration/Tuber_CSN152_JHMDB.yaml) |  [S3](https://yzaws-data-log.s3.amazonaws.com/shared/TubeR_cvpr22/TubeR_CSN152_JHMDB.pth) |\n\n\n\n## Usage\nThe project is developed based on [GluonCV-torch](https://cv.gluon.ai/).\nPlease refer to [tutorial](https://cv.gluon.ai/build/examples_torch_action_recognition/ddp_pytorch.html) for details.\n\n### Dependency\nThe project is tested working on:\n- Torch 1.12 + CUDA 11.3\n- timm==0.4.5 \n- tensorboardX\n\n### Dataset\nPlease download the [asset.zip](https://yzaws-data-log.s3.amazonaws.com/shared/TubeR_cvpr22/assets.zip) and unzip them at ./datasets.\n\n[AVA]\nPlease refer to [DATASET.md](https://github.com/facebookresearch/SlowFast/blob/main/slowfast/datasets/DATASET.md) for AVA dataset downloading and pre-processing.\n[JHMDB]\nPlease refer to [JHMDB](http://jhmdb.is.tue.mpg.de/) for JHMDB dataset and [Dataset Section](https://github.com/gurkirt/realtime-action-detection#datasets) for UCF dataset. You also can refer to [ACT-Detector](https://github.com/vkalogeiton/caffe/tree/act-detector) to prepare the two datasets.\n\n### Inference\nTo run inference, first modify the config file:\n- set the correct `WORLD_SIZE`, `GPU_WORLD_SIZE`, `DIST_URL`, `WOLRD_URLS` based on experiment setup.\n- set the `LABEL_PATH`, `ANNO_PATH`, `DATA_PATH` to your local directory accordingly.\n- Download the pre-trained model and set `PRETRAINED_PATH` to model path.\n- make sure `LOAD` and `LOAD_FC` are set to True\n\nThen run:\n```\n# run testing\npython3  eval_tuber_ava.py \u003cCONFIG_FILE\u003e \n\n# for example, to evaluate ava from scratch, run:\npython3 eval_tuber_ava.py configuration/TubeR_CSN152_AVA21.yaml\n```\n\n### Training\nTo train TubeR from scratch, first modify the configfile:\n- set the correct `WORLD_SIZE`, `GPU_WORLD_SIZE`, `DIST_URL`, `WOLRD_URLS` based on experiment setup.\n- set the `LABEL_PATH`, `ANNO_PATH`, `DATA_PATH` to your local directory accordingly.\n- Download the pre-trained feature backbone and transformer weights and set `PRETRAIN_BACKBONE_DIR` ([CSN50](https://yzaws-data-log.s3.amazonaws.com/shared/TubeR_cvpr22/irCSN_50_ft_kinetics_from_ig65m_f233743920.mat), [CSN152](https://yzaws-data-log.s3.amazonaws.com/shared/TubeR_cvpr22/irCSN_152_ft_kinetics_from_ig65m_f126851907.mat)), `PRETRAIN_TRANSFORMER_DIR` ([DETR](https://yzaws-data-log.s3.amazonaws.com/shared/TubeR_cvpr22/detr.pth)) accordingly. \n- make sure `LOAD` and `LOAD_FC` are set to False\n  \nThen run:\n```\n# run training from scratch\npython3  train_tuber.py \u003cCONFIG_FILE\u003e\n\n# for example, to train ava from scratch, run:\npython3 train_tuber_ava.py configuration/TubeR_CSN152_AVA21.yaml\n```\n\n## TODO\n[ ]Add tutorial and pre-trained weights for TubeR with long-term memory\n\n[ ]Add weights for UCF24\n\n\n## Citing TubeR\n```\n@inproceedings{zhao2022tuber,\n  title={TubeR: Tubelet transformer for video action detection},\n  author={Zhao, Jiaojiao and Zhang, Yanyi and Li, Xinyu and Chen, Hao and Shuai, Bing and Xu, Mingze and Liu, Chunhui and Kundu, Kaustav and Xiong, Yuanjun and Modolo, Davide and others},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={13598--13607},\n  year={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Ftubelet-transformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famazon-science%2Ftubelet-transformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Ftubelet-transformer/lists"}