{"id":26478663,"url":"https://github.com/kenshohara/3d-resnets-pytorch","last_synced_at":"2025-05-15T04:02:45.710Z","repository":{"id":37617145,"uuid":"103528129","full_name":"kenshohara/3D-ResNets-PyTorch","owner":"kenshohara","description":"3D ResNets for Action Recognition (CVPR 2018)","archived":false,"fork":false,"pushed_at":"2021-01-20T20:31:36.000Z","size":336,"stargazers_count":3968,"open_issues_count":154,"forks_count":930,"subscribers_count":57,"default_branch":"master","last_synced_at":"2025-04-10T05:03:01.804Z","etag":null,"topics":["action-recognition","computer-vision","deep-learning","python","pytorch","video-recognition"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kenshohara.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-14T12:14:44.000Z","updated_at":"2025-04-09T11:43:22.000Z","dependencies_parsed_at":"2022-08-08T21:01:13.401Z","dependency_job_id":null,"html_url":"https://github.com/kenshohara/3D-ResNets-PyTorch","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kenshohara%2F3D-ResNets-PyTorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kenshohara%2F3D-ResNets-PyTorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kenshohara%2F3D-ResNets-PyTorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kenshohara%2F3D-ResNets-PyTorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kenshohara","download_url":"https://codeload.github.com/kenshohara/3D-ResNets-PyTorch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254270640,"owners_count":22042858,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["action-recognition","computer-vision","deep-learning","python","pytorch","video-recognition"],"created_at":"2025-03-20T01:16:49.733Z","updated_at":"2025-05-15T04:02:45.658Z","avatar_url":"https://github.com/kenshohara.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 3D ResNets for Action Recognition\n\n## Update (2020/4/13)\n\nWe published a paper on arXiv.\n\n[\nHirokatsu Kataoka, Tenga Wakamiya, Kensho Hara, and Yutaka Satoh,  \n\"Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs\",  \narXiv preprint, arXiv:2004.04968, 2020.\n](https://arxiv.org/abs/2004.04968)\n\nWe uploaded the pretrained models described in this paper including ResNet-50 pretrained on the combined dataset with Kinetics-700 and Moments in Time.\n\n## Update (2020/4/10)\n\nWe significantly updated our scripts. If you want to use older versions to reproduce our CVPR2018 paper, you should use the scripts in the CVPR2018 branch.\n\nThis update includes as follows:\n* Refactoring whole project\n* Supporting the newer PyTorch versions\n* Supporting distributed training\n* Supporting training and testing on the Moments in Time dataset.\n* Adding R(2+1)D models\n* Uploading 3D ResNet models trained on the Kinetics-700, Moments in Time, and STAIR-Actions datasets\n\n## Summary\n\nThis is the PyTorch code for the following papers:\n\n[\nHirokatsu Kataoka, Tenga Wakamiya, Kensho Hara, and Yutaka Satoh,  \n\"Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs\",  \narXiv preprint, arXiv:2004.04968, 2020.\n](https://arxiv.org/abs/2004.04968)\n\n[\nKensho Hara, Hirokatsu Kataoka, and Yutaka Satoh,  \n\"Towards Good Practice for Action Recognition with Spatiotemporal 3D Convolutions\",  \nProceedings of the International Conference on Pattern Recognition, pp. 2516-2521, 2018.\n](https://ieeexplore.ieee.org/document/8546325)\n\n[\nKensho Hara, Hirokatsu Kataoka, and Yutaka Satoh,  \n\"Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?\",  \nProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6546-6555, 2018.\n](http://openaccess.thecvf.com/content_cvpr_2018/html/Hara_Can_Spatiotemporal_3D_CVPR_2018_paper.html)\n\n[\nKensho Hara, Hirokatsu Kataoka, and Yutaka Satoh,  \n\"Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition\",  \nProceedings of the ICCV Workshop on Action, Gesture, and Emotion Recognition, 2017.\n](http://openaccess.thecvf.com/content_ICCV_2017_workshops/papers/w44/Hara_Learning_Spatio-Temporal_Features_ICCV_2017_paper.pdf)\n\nThis code includes training, fine-tuning and testing on Kinetics, Moments in Time, ActivityNet, UCF-101, and HMDB-51.\n\n## Citation\n\nIf you use this code or pre-trained models, please cite the following:\n\n```bibtex\n@inproceedings{hara3dcnns,\n  author={Kensho Hara and Hirokatsu Kataoka and Yutaka Satoh},\n  title={Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?},\n  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},\n  pages={6546--6555},\n  year={2018},\n}\n```\n\n## Pre-trained models\n\nPre-trained models are available [here](https://drive.google.com/open?id=1xbYbZ7rpyjftI_KCk6YuL-XrfQDz7Yd4).  \nAll models are trained on Kinetics-700 (_K_), Moments in Time (_M_), STAIR-Actions (_S_), or merged datasets of them (_KM_, _KS_, _MS_, _KMS_).  \nIf you want to finetune the models on your dataset, you should specify the following options.\n\n```misc\nr3d18_K_200ep.pth: --model resnet --model_depth 18 --n_pretrain_classes 700\nr3d18_KM_200ep.pth: --model resnet --model_depth 18 --n_pretrain_classes 1039\nr3d34_K_200ep.pth: --model resnet --model_depth 34 --n_pretrain_classes 700\nr3d34_KM_200ep.pth: --model resnet --model_depth 34 --n_pretrain_classes 1039\nr3d50_K_200ep.pth: --model resnet --model_depth 50 --n_pretrain_classes 700\nr3d50_KM_200ep.pth: --model resnet --model_depth 50 --n_pretrain_classes 1039\nr3d50_KMS_200ep.pth: --model resnet --model_depth 50 --n_pretrain_classes 1139\nr3d50_KS_200ep.pth: --model resnet --model_depth 50 --n_pretrain_classes 800\nr3d50_M_200ep.pth: --model resnet --model_depth 50 --n_pretrain_classes 339\nr3d50_MS_200ep.pth: --model resnet --model_depth 50 --n_pretrain_classes 439\nr3d50_S_200ep.pth: --model resnet --model_depth 50 --n_pretrain_classes 100\nr3d101_K_200ep.pth: --model resnet --model_depth 101 --n_pretrain_classes 700\nr3d101_KM_200ep.pth: --model resnet --model_depth 101 --n_pretrain_classes 1039\nr3d152_K_200ep.pth: --model resnet --model_depth 152 --n_pretrain_classes 700\nr3d152_KM_200ep.pth: --model resnet --model_depth 152 --n_pretrain_classes 1039\nr3d200_K_200ep.pth: --model resnet --model_depth 200 --n_pretrain_classes 700\nr3d200_KM_200ep.pth: --model resnet --model_depth 200 --n_pretrain_classes 1039\n```\n\nOld pretrained models are still available [here](https://drive.google.com/drive/folders/1zvl89AgFAApbH0At-gMuZSeQB_LpNP-M?usp=sharing).  \nHowever, some modifications are required to use the old pretrained models in the current scripts.\n\n## Requirements\n\n* [PyTorch](http://pytorch.org/) (ver. 0.4+ required)\n\n```bash\nconda install pytorch torchvision cudatoolkit=10.1 -c soumith\n```\n\n* FFmpeg, FFprobe\n\n* Python 3\n\n## Preparation\n\n### ActivityNet\n\n* Download videos using [the official crawler](https://github.com/activitynet/ActivityNet/tree/master/Crawler).\n* Convert from avi to jpg files using ```util_scripts/generate_video_jpgs.py```\n\n```bash\npython -m util_scripts.generate_video_jpgs mp4_video_dir_path jpg_video_dir_path activitynet\n```\n\n* Add fps infomartion into the json file ```util_scripts/add_fps_into_activitynet_json.py```\n\n```bash\npython -m util_scripts.add_fps_into_activitynet_json mp4_video_dir_path json_file_path\n```\n\n### Kinetics\n\n* Download videos using [the official crawler](https://github.com/activitynet/ActivityNet/tree/master/Crawler/Kinetics).\n  * Locate test set in ```video_directory/test```.\n* Convert from avi to jpg files using ```util_scripts/generate_video_jpgs.py```\n\n```bash\npython -m util_scripts.generate_video_jpgs mp4_video_dir_path jpg_video_dir_path kinetics\n```\n\n* Generate annotation file in json format similar to ActivityNet using ```util_scripts/kinetics_json.py```\n  * The CSV files (kinetics_{train, val, test}.csv) are included in the crawler.\n\n```bash\npython -m util_scripts.kinetics_json csv_dir_path 700 jpg_video_dir_path jpg dst_json_path\n```\n\n### UCF-101\n\n* Download videos and train/test splits [here](http://crcv.ucf.edu/data/UCF101.php).\n* Convert from avi to jpg files using ```util_scripts/generate_video_jpgs.py```\n\n```bash\npython -m util_scripts.generate_video_jpgs avi_video_dir_path jpg_video_dir_path ucf101\n```\n\n* Generate annotation file in json format similar to ActivityNet using ```util_scripts/ucf101_json.py```\n  * ```annotation_dir_path``` includes classInd.txt, trainlist0{1, 2, 3}.txt, testlist0{1, 2, 3}.txt\n\n```bash\npython -m util_scripts.ucf101_json annotation_dir_path jpg_video_dir_path dst_json_path\n```\n\n### HMDB-51\n\n* Download videos and train/test splits [here](http://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/).\n* Convert from avi to jpg files using ```util_scripts/generate_video_jpgs.py```\n\n```bash\npython -m util_scripts.generate_video_jpgs avi_video_dir_path jpg_video_dir_path hmdb51\n```\n\n* Generate annotation file in json format similar to ActivityNet using ```util_scripts/hmdb51_json.py```\n  * ```annotation_dir_path``` includes brush_hair_test_split1.txt, ...\n\n```bash\npython -m util_scripts.hmdb51_json annotation_dir_path jpg_video_dir_path dst_json_path\n```\n\n## Running the code\n\nAssume the structure of data directories is the following:\n\n```misc\n~/\n  data/\n    kinetics_videos/\n      jpg/\n        .../ (directories of class names)\n          .../ (directories of video names)\n            ... (jpg files)\n    results/\n      save_100.pth\n    kinetics.json\n```\n\nConfirm all options.\n\n```bash\npython main.py -h\n```\n\nTrain ResNets-50 on the Kinetics-700 dataset (700 classes) with 4 CPU threads (for data loading).  \nBatch size is 128.  \nSave models at every 5 epochs.\nAll GPUs is used for the training.\nIf you want a part of GPUs, use ```CUDA_VISIBLE_DEVICES=...```.\n\n```bash\npython main.py --root_path ~/data --video_path kinetics_videos/jpg --annotation_path kinetics.json \\\n--result_path results --dataset kinetics --model resnet \\\n--model_depth 50 --n_classes 700 --batch_size 128 --n_threads 4 --checkpoint 5\n```\n\nContinue Training from epoch 101. (~/data/results/save_100.pth is loaded.)\n\n```bash\npython main.py --root_path ~/data --video_path kinetics_videos/jpg --annotation_path kinetics.json \\\n--result_path results --dataset kinetics --resume_path results/save_100.pth \\\n--model_depth 50 --n_classes 700 --batch_size 128 --n_threads 4 --checkpoint 5\n```\n\nCalculate top-5 class probabilities of each video using a trained model (~/data/results/save_200.pth.)  \nNote that ```inference_batch_size``` should be small because actual batch size is calculated by ```inference_batch_size * (n_video_frames / inference_stride)```.\n\n```bash\npython main.py --root_path ~/data --video_path kinetics_videos/jpg --annotation_path kinetics.json \\\n--result_path results --dataset kinetics --resume_path results/save_200.pth \\\n--model_depth 50 --n_classes 700 --n_threads 4 --no_train --no_val --inference --output_topk 5 --inference_batch_size 1\n```\n\nEvaluate top-1 video accuracy of a recognition result (~/data/results/val.json).\n\n```bash\npython -m util_scripts.eval_accuracy ~/data/kinetics.json ~/data/results/val.json --subset val -k 1 --ignore\n```\n\nFine-tune fc layers of a pretrained model (~/data/models/resnet-50-kinetics.pth) on UCF-101.\n\n```bash\npython main.py --root_path ~/data --video_path ucf101_videos/jpg --annotation_path ucf101_01.json \\\n--result_path results --dataset ucf101 --n_classes 101 --n_pretrain_classes 700 \\\n--pretrain_path models/resnet-50-kinetics.pth --ft_begin_module fc \\\n--model resnet --model_depth 50 --batch_size 128 --n_threads 4 --checkpoint 5\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkenshohara%2F3d-resnets-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkenshohara%2F3d-resnets-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkenshohara%2F3d-resnets-pytorch/lists"}