{"id":16510215,"url":"https://github.com/kiyoon/pyvideoai","last_synced_at":"2025-09-11T13:05:50.150Z","repository":{"id":101585425,"uuid":"370617777","full_name":"kiyoon/PyVideoAI","owner":"kiyoon","description":"PyTorch-based Ultimate Deep Learning Research Tool focusing on Video Action Recognition","archived":false,"fork":false,"pushed_at":"2022-10-12T16:28:25.000Z","size":1944,"stargazers_count":24,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-08T15:13:38.910Z","etag":null,"topics":["deep-learning","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kiyoon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-25T08:20:43.000Z","updated_at":"2024-10-30T12:23:00.000Z","dependencies_parsed_at":"2023-06-06T04:00:34.942Z","dependency_job_id":null,"html_url":"https://github.com/kiyoon/PyVideoAI","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/kiyoon/PyVideoAI","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiyoon%2FPyVideoAI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiyoon%2FPyVideoAI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiyoon%2FPyVideoAI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiyoon%2FPyVideoAI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kiyoon","download_url":"https://codeload.github.com/kiyoon/PyVideoAI/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiyoon%2FPyVideoAI/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274640911,"owners_count":25322843,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-11T02:00:13.660Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","pytorch"],"created_at":"2024-10-11T15:54:27.235Z","updated_at":"2025-09-11T13:05:50.142Z","avatar_url":"https://github.com/kiyoon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## This reposity contains official implementation of:\n- [Capturing Temporal Information in a Single Frame: Channel Sampling Strategies for Action Recognition](http://arxiv.org/abs/2201.10394) (Kim et al., BMVC 2022) [`Instruction`](docs/projects/channel_sampling)\n\u003cimg src=\"https://user-images.githubusercontent.com/12980409/151038213-12bdad91-7895-40e7-9304-126079fed637.png\" alt=\"8-frame TC Reordering\" width=\"400\"\u003e\n\n- [An Action Is Worth Multiple Words: Handling Ambiguity in Action Recognition](https://arxiv.org/abs/2210.04933) (Kim et al., BMVC 2022) [`Instruction`](docs/projects/verb_ambiguity)\n\u003cimg src=\"https://user-images.githubusercontent.com/12980409/193856345-e0287624-4c84-46af-b245-c07ff263c424.png\" alt=\"Verb Ambiguity\" width=\"400\"\u003e\n\n# PyVideoAI: Action Recognition Framework\n\nThe only framework that completes your computer vision, action recognition research environment.  \n\n** Key features **  \n- Supports multi-gpu, multi-node training.  \n- STOA models such as I3D, Non-local, TSN, TRN, TSM, MVFNet, ..., and even ImageNet training!\n- Many datasets such as Kinetics-400, EPIC-Kitchens-55, Something-Something-V1/V2, HMDB-51, UCF-101, Diving48, CATER, ...\n- Supports both video decoding (straight from .avi/mp4) and frame extracted (.jpg/png) dataloaders, sparse-sample and dense-sample.\n- Any popular LR scheduling like Cosine Annealing with Warm Restart, Step LR, and Reduce LR on Plateau.\n- Early stopping when training doesn't improve (customise your condition)\n- **Easily add custom model, optimiser, scheduler, loss and dataloader!**\n- Telegram bot reporting experiment status.  \n- TensorBoard reporting stats.  \n- Colour logging  \n- All of the above come with no extra setup. Trust me and try some [examples](https://github.com/kiyoon/PyVideoAI-examples.git).\n\n** Papers implemented **  \n- [*ProSelfLC* (CVPR 2021)](https://arxiv.org/abs/2005.03788).  \n\n\nThis package is motivated by PySlowFast from Facebook AI. The PySlowFast is a cool framework, but it depends too much on their config system and it was difficult to add new models (other codes) or reuse part of the modules from the framework.  \nThis framework by Kiyoon, is designed to replace all the configuration systems to Python files, which enables **easy-addition of custom models/LR scheduling/dataloader** etc.  \nJust modify the function bodies in the config files!\n\nDifference between the two config systems can be found in [CONFIG_SYSTEM.md](docs/CONFIG_SYSTEM.md).\n\n# Getting Started\nJupyter Notebook examples to run:  \n- HMDB-51 data preparation\n- Inference on pre-trained model from the model zoo, and visualise model/dataloader/per-class performance.\n- Training I3D using Kinetics pretrained model\n- Using image model and ImageNet dataset  \n\nis provided in the [examples](https://github.com/kiyoon/PyVideoAI-examples)!\n\n\n# Structure\n\nAll of the executable files are in `tools/`.  \n`dataset_configs/` directory configures datasets. For example, where is the dataset stored, number of classes, single-label or multi-label training, dataset-specific visualisation settings (confusion matrix has different output sizes)  \n`model_configs/` directory configures model architectures. For example, model definition, input preprocessing mean/std.  \n`exp_configs/` directory configures other training settings like optimiser, scheduling, dataloader, number of frames as input. The config file path has to be in `exp_configs/[dataset_name]/[model_name]_[experiment_name].py` format.\n\n# Usage\n\n## Preparing datasets\n\nThis package supports many action recognition datasets such as HMDB-51, EPIC-Kitchens-55, Something-Something-V1, CATER, etc.  \nRefer to [DATASET.md](docs/DATASET.md).\n\n## Training command\n```bash\n# Single GPU\nCUDA_VISIBLE_DEVICES=0 python tools/run_singlenode.sh train 1 -D {dataset_config_name} -M {model_config_name} -E {exp_config_name}\n# Multi GPUs, single node\nCUDA_VISIBLE_DEVICES=0,1,2,3 python tools/run_singlenode.sh train {num_gpus} -D {dataset_config_name} -M {model_config_name} -E {exp_config_name}\n# Multi GPU, multi node (run on every node)\nCUDA_VISIBLE_DEVICES=0,1,2,3 python tools/run_multinode.sh train {num_gpus_per_node} {num_nodes} {node_rank} {master_address} {master_port} -D {dataset_config_name} -M {model_config_name} -E {exp_config_name}\n```\n\n## Telegram Bot\nYou can preview experiment results using Telegram bots!  \n\u003cimg src=\"https://user-images.githubusercontent.com/12980409/122335586-7cb10a80-cf76-11eb-950f-af08c20055d4.png\" alt=\"Telegram bot stat report example\" width=\"400\"\u003e\n\nIf your code raises an exception, it will report you too.  \n\u003cimg src=\"https://user-images.githubusercontent.com/12980409/122337458-5476db00-cf79-11eb-8d71-3e8ecc9faa9a.png\" alt=\"Telegram error report example\" width=\"400\"\u003e\n\nYou can quickly take a look at example video inputs (as GIF or JPEGs) from the dataloader.  \nUse [tools/visualisations/model_and_dataloader_visualiser.py](tools/visualisations/model_and_dataloader_visualiser.py)  \n\u003cimg src=\"https://user-images.githubusercontent.com/12980409/122337617-8a1bc400-cf79-11eb-8c48-b0d52a2c49c5.png\" alt=\"Telegram video input report example\" width=\"200\"\u003e\n\n\n\n- Talk to BotFather and make a bot.  \n- Go to your bot and type anything (/start)  \n- Find chat_id at https://api.telegram.org/bot{token}/getUpdates (replace {token} with your token, excluding braces.)  \n- Add your token and chat_id to `tools/key.ini`.  \n\n```INI\n[Telegram0]\ntoken=\nchat_id=\n```\n\n\n# Model Zoo and Baselines\nRefer to [MODEL_ZOO.md](docs/MODEL_ZOO.md)\n\n# Installation\nRefer to [INSTALL.md](docs/INSTALL.md).\n\nTL;DR,\n\n```bash\nconda create -n videoai python=3.9\nconda activate videoai\nconda install pytorch==1.12.1 torchvision cudatoolkit=10.2 -c pytorch\n### For RTX 30xx GPUs,\n#conda install pytorch==1.12.1 torchvision cudatoolkit=11.3 -c pytorch\n \n\ngit clone --recurse-submodules https://github.com/kiyoon/PyVideoAI.git\ncd PyVideoAI\ngit checkout v0.4\ngit submodule update --recursive\ncd submodules/video_datasets_api\npip install -e .\ncd ../experiment_utils\npip install -e .\ncd ../..\npip install -e .\n```\n\nOptional: Pillow-SIMD and libjepg-turbo to improve dataloading performance.  \nRun this at the end of the installation:  \n\n```bash\nconda uninstall -y --force pillow pil jpeg libtiff libjpeg-turbo\npip   uninstall -y         pillow pil jpeg libtiff libjpeg-turbo\nconda install -yc conda-forge libjpeg-turbo\nCFLAGS=\"${CFLAGS} -mavx2\" pip install --upgrade --no-cache-dir --force-reinstall --no-binary :all: --compile pillow-simd\nconda install -y jpeg libtiff\n```\n\n# Experiment outputs\n\nThe experiment results (log, training stats, weights, tensorboard, plots, etc.) are saved to `data/experiments` by default. This can be huge, so make sure you **make a softlink of a directory you really want to use. (recommended)**  \n\nOtherwise, you can change `pyvideoai/config.py`'s `DEFAULT_EXPERIMENT_ROOT` value. Or, you can also set `--experiment_root`/`-R` argument manually when executing.  \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkiyoon%2Fpyvideoai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkiyoon%2Fpyvideoai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkiyoon%2Fpyvideoai/lists"}