{"id":17983148,"url":"https://github.com/kaiyangzhou/pytorch-vsumm-reinforce","last_synced_at":"2025-04-05T02:11:31.443Z","repository":{"id":51731257,"uuid":"130244363","full_name":"KaiyangZhou/pytorch-vsumm-reinforce","owner":"KaiyangZhou","description":"Unsupervised video summarization with deep reinforcement learning (AAAI'18)","archived":false,"fork":false,"pushed_at":"2023-12-11T16:34:57.000Z","size":394,"stargazers_count":486,"open_issues_count":60,"forks_count":151,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-29T01:13:16.473Z","etag":null,"topics":["computer-vision","deep-learning","machine-learning","policy-network","reinforcement-learning","unsupervised-learning","video-summarization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KaiyangZhou.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-19T16:36:11.000Z","updated_at":"2025-03-28T09:38:14.000Z","dependencies_parsed_at":"2024-10-29T18:38:41.457Z","dependency_job_id":null,"html_url":"https://github.com/KaiyangZhou/pytorch-vsumm-reinforce","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KaiyangZhou%2Fpytorch-vsumm-reinforce","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KaiyangZhou%2Fpytorch-vsumm-reinforce/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KaiyangZhou%2Fpytorch-vsumm-reinforce/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KaiyangZhou%2Fpytorch-vsumm-reinforce/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KaiyangZhou","download_url":"https://codeload.github.com/KaiyangZhou/pytorch-vsumm-reinforce/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247276189,"owners_count":20912288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","machine-learning","policy-network","reinforcement-learning","unsupervised-learning","video-summarization"],"created_at":"2024-10-29T18:16:16.358Z","updated_at":"2025-04-05T02:11:31.424Z","avatar_url":"https://github.com/KaiyangZhou.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pytorch-vsumm-reinforce\nThis repo contains the Pytorch implementation of the AAAI'18 paper - [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://arxiv.org/abs/1801.00054). The original Theano implementation can be found [here](https://github.com/KaiyangZhou/vsumm-reinforce).\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"imgs/pipeline.jpg\" alt=\"train\" width=\"80%\"\u003e\n\u003c/div\u003e\n\nThe main requirements are [pytorch](http://pytorch.org/) (`v0.4.0`) and python `2.7`. Some dependencies that may not be installed in your machine are [tabulate](https://pypi.org/project/tabulate/) and [h5py](https://github.com/h5py/h5py). Please install other missing dependencies.\n\n## Get started\n1. Download preprocessed datasets\n```bash\ngit clone https://github.com/KaiyangZhou/pytorch-vsumm-reinforce\ncd pytorch-vsumm-reinforce\n# download datasets.tar.gz (173.5MB)\nwget http://www.eecs.qmul.ac.uk/~kz303/vsumm-reinforce/datasets.tar.gz\ntar -xvzf datasets.tar.gz\n```\n\n**Updates**: The QMUL server is inaccessible. Download the datasets from this [google drive link](https://drive.google.com/open?id=1Bf0beMN_ieiM3JpprghaoOwQe9QJIyAN).\n\n2. Make splits\n```bash\npython create_split.py -d datasets/eccv16_dataset_summe_google_pool5.h5 --save-dir datasets --save-name summe_splits  --num-splits 5\n```\nAs a result, the dataset is randomly split for 5 times, which are saved as json file.\n\nTrain and test codes are written in `main.py`. To see the detailed arguments, please do `python main.py -h`.\n\n## How to train\n```bash\npython main.py -d datasets/eccv16_dataset_summe_google_pool5.h5 -s datasets/summe_splits.json -m summe --gpu 0 --save-dir log/summe-split0 --split-id 0 --verbose\n```\n\n## How to test\n```bash\npython main.py -d datasets/eccv16_dataset_summe_google_pool5.h5 -s datasets/summe_splits.json -m summe --gpu 0 --save-dir log/summe-split0 --split-id 0 --evaluate --resume path_to_your_model.pth.tar --verbose --save-results\n```\n\nIf argument `--save-results` is enabled, output results will be saved to `results.h5` under the same folder specified by `--save-dir`. To visualize the score-vs-gtscore, simple do\n```bash\npython visualize_results.py -p path_to/result.h5\n```\n\n## Plot\nWe provide codes to plot the rewards obtained at each epoch. Use `parse_log.py` to plot the average rewards\n```bash\npython parse_log.py -p path_to/log_train.txt\n```\nThe plotted image would look like\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"imgs/overall_reward.png\" alt=\"overall_reward\" width=\"50%\"\u003e\n\u003c/div\u003e\n\nIf you wanna plot the epoch-reward curve for some specific videos, do\n```bash\npython parse_json.py -p path_to/rewards.json -i 0\n```\n\nYou will obtain images like\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"imgs/epoch_reward_0.png\" alt=\"epoch_reward\" width=\"30%\"\u003e\n  \u003cimg src=\"imgs/epoch_reward_13.png\" alt=\"epoch_reward\" width=\"30%\"\u003e\n  \u003cimg src=\"imgs/epoch_reward_15.png\" alt=\"epoch_reward\" width=\"30%\"\u003e\n\u003c/div\u003e\n\nIf you prefer to visualize the epoch-reward curve for all training videos, try `parse_json.sh`. Modify the code according to your purpose.\n\n## Visualize summary\nYou can use `summary2video.py` to transform the binary `machine_summary` to real summary video. You need to have a directory containing video frames. The code will automatically write summary frames to a video where the frame rate can be controlled. Use the following command to generate a `.mp4` video\n```bash\npython summary2video.py -p path_to/result.h5 -d path_to/video_frames -i 0 --fps 30 --save-dir log --save-name summary.mp4\n```\nPlease remember to specify the naming format of your video frames on this [line](https://github.com/KaiyangZhou/pytorch-vsumm-reinforce/blob/master/summary2video.py#L22).\n\n## How to use your own data\nWe preprocess data by extracting image features for videos and save them to `h5` file. The file format looks like [this](https://github.com/KaiyangZhou/vsumm-reinforce/issues/1#issuecomment-363492711). After that, you can make split via `create_split.py`. If you wanna train policy network using the entire dataset, just do `train_keys = dataset.keys()`. [Here](https://github.com/KaiyangZhou/pytorch-vsumm-reinforce/blob/master/main.py#L75) is the code where we initialize dataset. If you have any problems, feel free to contact me by email or raise an `issue`.\n\n## Citation\n```\n@article{zhou2017reinforcevsumm, \n   title={Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward},\n   author={Zhou, Kaiyang and Qiao, Yu and Xiang, Tao}, \n   journal={arXiv:1801.00054}, \n   year={2017} \n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaiyangzhou%2Fpytorch-vsumm-reinforce","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkaiyangzhou%2Fpytorch-vsumm-reinforce","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaiyangzhou%2Fpytorch-vsumm-reinforce/lists"}