{"id":13583878,"url":"https://github.com/HHTseng/video-classification","last_synced_at":"2025-04-06T21:33:24.507Z","repository":{"id":45784612,"uuid":"157634549","full_name":"HHTseng/video-classification","owner":"HHTseng","description":"Tutorial for video classification/ action recognition using 3D CNN/ CNN+RNN on UCF101","archived":false,"fork":false,"pushed_at":"2020-12-07T07:54:24.000Z","size":10321,"stargazers_count":922,"open_issues_count":38,"forks_count":216,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-08-02T15:54:00.077Z","etag":null,"topics":["action-recognition","cnn","lstm","pytorch-tutorial","resnet","rnn","transfer-learning","ucf101","video-classification"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HHTseng.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-15T01:30:50.000Z","updated_at":"2024-08-01T11:33:13.000Z","dependencies_parsed_at":"2022-07-16T15:17:17.644Z","dependency_job_id":null,"html_url":"https://github.com/HHTseng/video-classification","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HHTseng%2Fvideo-classification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HHTseng%2Fvideo-classification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HHTseng%2Fvideo-classification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HHTseng%2Fvideo-classification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HHTseng","download_url":"https://codeload.github.com/HHTseng/video-classification/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223264990,"owners_count":17116263,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["action-recognition","cnn","lstm","pytorch-tutorial","resnet","rnn","transfer-learning","ucf101","video-classification"],"created_at":"2024-08-01T15:03:51.983Z","updated_at":"2024-11-06T00:30:54.828Z","avatar_url":"https://github.com/HHTseng.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook"],"sub_categories":[],"readme":"# Video Classification\n\nThe repository builds a **quick and simple** code for video classification (or action recognition) using [UCF101](http://crcv.ucf.edu/data/UCF101.php) with PyTorch. A video is viewed as a 3D image or several continuous 2D images (Fig.1). Below are two simple neural nets models:\n\n\n## Dataset\n\n![alt text](./fig/kayaking.gif)\n\n\n[UCF101](http://crcv.ucf.edu/data/UCF101.php) has total 13,320 videos from 101 actions. Videos have various time lengths (frames) and different 2d image size; the shortest is 28 frames. \n\nTo avoid painful video preprocessing like frame extraction and conversion such as [OpenCV](https://opencv.org/) or [FFmpeg](https://www.ffmpeg.org/), here I used a preprocessed dataset from [feichtenhofer](https://github.com/feichtenhofer/twostreamfusion) directly. If you want to convert or extract video frames from scratch, here are some nice tutorials: \n  - https://pythonprogramming.net/loading-video-python-opencv-tutorial/\n  - https://www.pyimagesearch.com/2017/02/06/faster-video-file-fps-with-cv2-videocapture-and-opencv/ \n\n## Models \n\n### 1. 3D CNN (train from scratch)\nUse several 3D kernels of size *(a,b,c)* and channels *n*,  *e.g., (a, b, c, n) = (3, 3, 3, 16)* to convolve with video input, where videos are viewed as 3D images. *Batch normalization* and *dropout* are also used.\n\n\n### 2. **CNN + RNN** (CRNN)\n\nThe CRNN model is a pair of CNN encoder and RNN decoder (see figure below):\n\n  - **[encoder]** A [CNN](https://en.wikipedia.org/wiki/Convolutional_neural_network) function encodes (meaning compressing dimension) every 2D image **x(t)** into a 1D vector **z(t)** by \u003cimg src=\"./fig/f_CNN.png\" width=\"140\"\u003e\n\n  - **[decoder]** A [RNN](https://en.wikipedia.org/wiki/Recurrent_neural_network) receives a sequence input vectors **z(t)** from the CNN encoder and outputs another 1D sequence **h(t)**. A final fully-connected neural net is concatenated at the end for categorical predictions. \n  \n  - Here the decoder RNN uses a long short-term memory [(LSTM)](https://en.wikipedia.org/wiki/Long_short-term_memory) network and the CNN encoder can be:\n    1. trained from scratch\n    2. a pretrained model [ResNet-152](https://arxiv.org/abs/1512.03385) using image dataset [ILSVRC-2012-CLS](http://www.image-net.org/challenges/LSVRC/2012/).\n\n\u003cimg src=\"./fig/CRNN.png\" width=\"650\"\u003e\n\n\n\n## Training \u0026 testing\n- For 3D CNN:\n   1. The videos are resized as **(t-dim, channels, x-dim, y-dim) = (28, 3, 256, 342)** since CNN requires a fixed-size input. The minimal frame number 28 is the consensus of all videos in UCF101.\n   2. *Batch normalization*, *dropout* are used.\n   \n- For CRNN, the videos are resized as **(t-dim, channels, x-dim, y-dim) = (28, 3, 224, 224)** since the ResNet-152 only receives RGB inputs of size (224, 224).\n\n- Training videos = **9,990** vs. testing videos = **3,330**\n\n- In the test phase, the models are almost the same as the training phase, except that dropout has to be removed and batchnorm layer uses moving average and variance instead of mini-batch values. These are taken care by using \"**model.eval()**\".\n\n\n## Usage \nFor tutorial purpose, I try to build code as simple as possible. Essentially, **only 3 files are needed to for each model**. *eg.,* for 3D-CNN model\n  - `UCF101_3DCNN.py`: model parameters, training/testing process.\n  - `function.py`: modules of 3DCNN \u0026 CRNN, data loaders, and some useful functions.\n  - `UCF101actions.pkl`: 101 action names (labels), e.g, *'BenchPress', 'SkyDiving' , 'Bowling', etc.*\n\n### 0. Prerequisites\n- [Python 3.6](https://www.python.org/)\n- [PyTorch 1.0.0](https://pytorch.org/)\n- [Numpy 1.15.0](http://www.numpy.org/)\n- [Sklearn 0.19.2](https://scikit-learn.org/stable/)\n- [Matplotlib](https://matplotlib.org/)\n- [Pandas](https://pandas.pydata.org/)\n- [tqdm](https://github.com/tqdm/tqdm)\n\n\n### 1. Download preprocessed UCF101 dataset\nFor convenience, we use preprocessed UCF101 dataset already sliced into RGB images [feichtenhofer/twostreamfusion](https://github.com/feichtenhofer/twostreamfusion):\n\n\n- **UCF101 RGB:** [**part1**](http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_jpegs_256.zip.001),\n[**part2**](http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_jpegs_256.zip.002),\n[**part3**](http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_jpegs_256.zip.003) \n\n\nPut the 3 parts in same folder to unzip. The folder has default name: **jpegs_256**.\n\n\n\n### 2. Set parameters \u0026 path\n\nIn `UCF101_CRNN.py`, for example set \n```\ndata_path = \"./UCF101/jpegs_256/\"         # UCF101 video path\naction_name_path = \"./UCF101actions.pkl\"\nsave_model_path = \"./model_ckpt/\"\n```\n\n### 3. Train \u0026 test model\n\n- For **3D CNN/ CRNN/ ResNetCRNN** model, in each folder run\n```bash\n$ python UCF101_3DCNN/CRNN/ResNetCRNN.py    \n```\n\n\n### 4. Model ouputs\n\nBy default, the model outputs:\n\n- Training \u0026 testing loss/ accuracy: `epoch_train_loss/score.npy`, `epoch_test_loss/score.npy`\n\n- Model parameters \u0026 optimizer: eg. `CRNN_epoch8.pth`, `CRNN_optimizer_epoch8.pth`. They can be used for retraining or pretrained purpose.\n\nTo check model prediction:\n  - Run ``check_model_prediction.py`` to load best training model and generate all 13,320 video prediction list in [Pandas](https://pandas.pydata.org/) dataframe. File output: `UCF101_Conv3D_videos_prediction.pkl`.\n  - Run `check_video_predictions.ipynb` with [Jupyter Notebook](http://jupyter.org/) and you can see where the model gets wrong:\n\n\u003cimg src=\"./fig/wrong_pred.png\" width=\"600\"\u003e\n\n\n## Version Warrning!\n\nAs of today (May 31, 2019), it is found that in Pytorch 1.1.0 **flatten_parameters()** doesn't work under [torch.no_grad and DataParallel](https://github.com/pytorch/pytorch/issues/21108) (for multiple GPUs). Early versions before Pytorch 1.0.1 still run OK. See [Issues](https://github.com/HHTseng/video-classification/issues)\n\nThanks to [raghavgarg97](https://github.com/raghavgarg97)'s report.\n\n\n## Device \u0026 performance \n\n- The models detect and use multiple GPUs by themselves, where we implemented [torch.nn.DataParallel](https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html).\n\n- A field test using 2 GPUs (nVidia TITAN V, 12Gb mem) with my default model parameters and batch size `30~60`.\n\n- Some **pretrained models** can be found [here](https://drive.google.com/open?id=117mRMS2r09fz4ozkdzN4cExO1sFwcMvs), thanks to the suggestion of [MinLiAmoy](https://github.com/MinLiAmoy?tab=repositories).\n\n\n network                | best epoch | testing accuracy |\n------------            |:-----:| :-----:|\n3D CNN                  |   4  |  50.84 % | \n2D CNN + LSTM           |  25  |  54.62 % | \n2D ResNet152-CNN + LSTM |  53  |**85.68 %** |      \n\n\u003cimg src=\"./fig/loss_3DCNN.png\" width=\"650\"\u003e\n\u003cimg src=\"./fig/loss_CRNN.png\" width=\"650\"\u003e\n\u003cimg src=\"./fig/loss_ResNetCRNN.png\" width=\"650\"\u003e\n\n\n\n\u003cbr\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHHTseng%2Fvideo-classification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FHHTseng%2Fvideo-classification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHHTseng%2Fvideo-classification/lists"}