https://github.com/kaiyangzhou/vsumm-reinforce
AAAI 2018 - Unsupervised video summarization with deep reinforcement learning (Theano)
https://github.com/kaiyangzhou/vsumm-reinforce
reinforcement-learning unsupervised-learning-algorithms video-summarization
Last synced: 5 months ago
JSON representation
AAAI 2018 - Unsupervised video summarization with deep reinforcement learning (Theano)
- Host: GitHub
- URL: https://github.com/kaiyangzhou/vsumm-reinforce
- Owner: KaiyangZhou
- License: mit
- Created: 2017-09-30T17:24:41.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2021-11-30T01:29:57.000Z (almost 4 years ago)
- Last Synced: 2025-04-06T09:51:42.132Z (7 months ago)
- Topics: reinforcement-learning, unsupervised-learning-algorithms, video-summarization
- Language: Python
- Homepage:
- Size: 630 KB
- Stars: 137
- Watchers: 9
- Forks: 35
- Open Issues: 20
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# vsumm-reinforce
This is the official implementation of the AAAI'18 paper [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://arxiv.org/abs/1801.00054). The code is based on Theano (version `0.9.0`).
Pytorch implementation can be found [here](https://github.com/KaiyangZhou/pytorch-vsumm-reinforce).
## Preparation
To get the datasets and models, you will need `wget`.
Run the following commands in order
```bash
git clone https://github.com/KaiyangZhou/vsumm-reinforce
cd vsumm-reinforce
# download datasets.tar.gz
wget http://www.eecs.qmul.ac.uk/~kz303/vsumm-reinforce/datasets.tar.gz
tar -xvzf datasets.tar.gz
# download models.tar.gz
wget http://www.eecs.qmul.ac.uk/~kz303/vsumm-reinforce/models.tar.gz
tar -xvzf models.tar.gz
```
**Updates**: The QMUL server is inaccessible. Download the datasets from this [google drive link](https://drive.google.com/open?id=1Bf0beMN_ieiM3JpprghaoOwQe9QJIyAN).
## How to train
Training code is implemented in `vsum_train.py`. To train a RNN, run
```bash
python vsum_train.py --dataset datasets/eccv16_dataset_tvsum_google_pool5.h5 --max-epochs 60 --hidden-dim 256
```
## How to test
Test code is implemented in `vsum_test.py`. For example, to test with our models, simply run
```bash
python vsum_test.py -model models/model_tvsum_reinforceRNN.h5 -d tvsum
python vsum_test.py -model models/model_tvsum_reinforceRNN_sup.h5 -d tvsum
python vsum_test.py -model models/model_summe_reinforceRNN.h5 -d summe
python vsum_test.py -model models/model_summe_reinforceRNN_sup.h5 -d summe
```
Output results are saved to `log-test/results.h5`. To visualize score-vs-gtscore, you can use `visualize_results.py` by
```bash
python visualize_results.py -p log-test/result.h5
```
## Visualize summary
You can use `summary2video.py` to transform the binary `machine_summary` to real summary video. You need to have a directory containing video frames. The code will automatically write summary frames to a video where the frame rate can be controlled. Use the following command to generate a `.mp4` video
```bash
python summary2video.py -p path_to/result.h5 -d path_to/video_frames -i 0 --fps 30 --save-dir log --save-name summary.mp4
```
Please remember to specify the naming format of your video frames on this [line](https://github.com/KaiyangZhou/vsumm-reinforce/blob/master/summary2video.py#L22).
## Citation
```
@article{zhou2017reinforcevsumm,
title={Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward},
author={Zhou, Kaiyang and Qiao, Yu and Xiang, Tao},
journal={arXiv:1801.00054},
year={2017}
}
```