{"id":16837477,"url":"https://github.com/bryanyzhu/hidden-two-stream","last_synced_at":"2025-09-08T00:08:54.145Z","repository":{"id":114929280,"uuid":"95412289","full_name":"bryanyzhu/Hidden-Two-Stream","owner":"bryanyzhu","description":"Caffe implementation for \"Hidden Two-Stream Convolutional Networks for Action Recognition\"","archived":false,"fork":false,"pushed_at":"2017-12-20T19:39:05.000Z","size":10917,"stargazers_count":194,"open_issues_count":3,"forks_count":68,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-04-02T03:34:57.124Z","etag":null,"topics":["action-recognition","caffe","cnn","hmdb51","optical-flow","real-time","ucf101","unsupervised-learning","video"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bryanyzhu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-26T05:42:14.000Z","updated_at":"2025-02-17T04:37:48.000Z","dependencies_parsed_at":null,"dependency_job_id":"af452d9b-bf64-4939-8f35-512cc4f4a8d9","html_url":"https://github.com/bryanyzhu/Hidden-Two-Stream","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bryanyzhu/Hidden-Two-Stream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanyzhu%2FHidden-Two-Stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanyzhu%2FHidden-Two-Stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanyzhu%2FHidden-Two-Stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanyzhu%2FHidden-Two-Stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bryanyzhu","download_url":"https://codeload.github.com/bryanyzhu/Hidden-Two-Stream/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanyzhu%2FHidden-Two-Stream/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274113098,"owners_count":25224336,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-07T02:00:09.463Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["action-recognition","caffe","cnn","hmdb51","optical-flow","real-time","ucf101","unsupervised-learning","video"],"created_at":"2024-10-13T12:17:34.776Z","updated_at":"2025-09-08T00:08:54.121Z","avatar_url":"https://github.com/bryanyzhu.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"Hidden Two-Stream Convolutional Networks for Action Recognition\n============================\n\nThis is the Caffe implementation of the \"Hidden Two-Stream Convolutional Networks for Action Recognition\". You can refer to paper for more details at [Arxiv](https://arxiv.org/abs/1704.00389).\n\n\nDependencies\n=========\n\nOpenCV 3 (Installation can be refered [here](https://github.com/BVLC/caffe/wiki/OpenCV-3.2-Installation-Guide-on-Ubuntu-16.04))\n\nTested on Ubuntu 16.04 with Titan X GPU, CUDNN 5.1\n\nCompiling\n=========\n\nTo get started, first compile caffe, by configuring a\n\n    \"Makefile.config\" \n\nthen make with \n\n    $ make -j 6 all\n\nTraining\n========\n\n(this assumes you compiled the code sucessfully) \n\nHere, we take UCF101 split 1 as an example. \n\nFirst, go to folder, \n\n    cd models/ucf101_split1_unsup_end\n    \nThen change the `FRAME_PATH` in `train_rgb_split1.txt` and `val_rgb_split1.txt` to where you store the extracted video frames,  \n\n    /FRAME_PATH/WallPushups/v_WallPushups_g21_c06 111 98\n\nThis follows the format as in [TSN](https://github.com/yjxiong/temporal-segment-networks). `111` indicates the number of frames of that video clip, and `98` represents the action label. For more details about how to construct file list for training and validation, we refer you to [here](https://github.com/yjxiong/temporal-segment-networks#construct-file-lists-for-training-and-validation).\n\nThen you need to download the initialization models (pre-trained temporal stream CNN stacked upon pre-trained MotionNet), \n\n[UCF101 split1](https://drive.google.com/open?id=0B-bJpXHBmFWDNnZ2TnE3cVZTNVU) \n\nThen tune the parameters in `end_train_val.prototxt` and `end_solver.prototxt` as you need, or leave as it is. \n\nFinally, you can simply run\n\n    ../../build/tools/caffe train -solver=end_solver.prototxt -weights=ucf101_split1_vgg16_init.caffemodel\n\n\nNOTE: It is highly likely that you may get better performance than us if you carefully tune the hyper-params such as loss weights, learning rate etc. \n\nTesting\n========\n\n(this assumes you compiled the code sucessfully) \n\nFirst, download our trained models:\n\n[UCF101 split 1](https://drive.google.com/open?id=0B-bJpXHBmFWDamFiUmp0UGpwY2c) [UCF101 split 2](https://drive.google.com/open?id=0B-bJpXHBmFWDVlpULU5tcmdGaGs) [UCF101 split 3](https://drive.google.com/open?id=0B-bJpXHBmFWDNmozVDlPSTFWdEE) \n\n[HMDB51 split 1](https://drive.google.com/open?id=0B-bJpXHBmFWDUER6OUdyVmNyenM) [HMDB51 split 2](https://drive.google.com/open?id=0B-bJpXHBmFWDcmxVZmxyUWVJbzQ) [HMDB51 split 3](https://drive.google.com/open?id=0B-bJpXHBmFWDenZpWlFqNm0yMnM) \n\nThen go to this folder\n\n    cd models/ucf101_split1_unsup_end/eval_ucf101\n\nThen run\n\n    python demo_hidden.py\n\nBut maybe you need to set paths correctly in `demo_hidden.py` before you run it, like `model_def_file` and `model_file`. And also change the `FRAME_PATH` in `testlist01_with_labels.txt`. \n\nAfter you get both spatial and hidden predictions, the late fusion code is in folder `./test`, run `late_fusion.m` to get the final two stream predictions.\n\n\nMotionNet\n=========\n\nThe training and testing code of MotionNet is in folder\n\n\tcd models/multiframe_MotionNet\n\nThe pretraied model can be downloaded at [MotionNet](https://drive.google.com/open?id=0B-bJpXHBmFWDVU5DRTY4Ym02TFE).\n\n\nMisc\n====================\n\n1. There is a chance that you may get a little bit higher or lower accuracy on UCF101 and HMDB51 than the numbers reported in our paper, even using our provided trained models. This is normal because your extracted video frames may not be the same as ours, and the quality of image has an impact on the final performance. Thus, no need to raise an issue unless the performance gap is large, e.g. larger than 1%. \n\n2. Since there are so many losses to compute, you may encounter model divergence in the very beginning of the training. You can simply reduce learning rate first to get a good initialization, and then back on track. Or you just rerun training several times. \n\n\nTODO\n====================\n\n- [ ] Experiment on large-scale action datasets, like Sports-1M and Kinetics \n\n\nLicense and Citation\n====================\n\nPlease cite this paper in your publications if you use this code or precomputed results for your research:\n\n    @article{hidden_ar_zhu_2017,\n      title={{Hidden Two-Stream Convolutional Networks for Action Recognition}},\n      author={Yi Zhu and Zhenzhong Lan and Shawn Newsam and Alexander G. Hauptmann},\n      journal={arXiv preprint arXiv:1704.00389},\n      year={2017}\n    }\n\nRelated Projects\n====================\n\n[GuidedNet](https://github.com/bryanyzhu/GuidedNet): Guided Optical Flow Learning\n\n[Two_Stream Pytorch](https://github.com/bryanyzhu/two-stream-pytorch): PyTorch implementation of two-stream networks for video action recognition\n\n\nAcknowledgement\n====================\n\nThe code base is borrowed from [TSN](https://github.com/yjxiong/temporal-segment-networks), [DispNet](https://lmb.informatik.uni-freiburg.de/resources/software.php) and [UnsupFlownet](http://scs.ryerson.ca/~jjyu/). Thanks for open sourcing the code.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbryanyzhu%2Fhidden-two-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbryanyzhu%2Fhidden-two-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbryanyzhu%2Fhidden-two-stream/lists"}