https://github.com/m-bain/condensedmovies
Story-Based Retrieval with Contextual Embeddings. Largest freely available movie video dataset. [ACCV'20]
https://github.com/m-bain/condensedmovies
dataset precomputed-features retrieval source-videos video-text-retrieval
Last synced: 11 months ago
JSON representation
Story-Based Retrieval with Contextual Embeddings. Largest freely available movie video dataset. [ACCV'20]
- Host: GitHub
- URL: https://github.com/m-bain/condensedmovies
- Owner: m-bain
- Created: 2020-04-21T08:55:01.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2022-09-21T18:39:40.000Z (almost 4 years ago)
- Last Synced: 2024-11-26T01:34:02.009Z (over 1 year ago)
- Topics: dataset, precomputed-features, retrieval, source-videos, video-text-retrieval
- Language: Python
- Homepage:
- Size: 22 MB
- Stars: 163
- Watchers: 10
- Forks: 28
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## CondensedMovies
**** ___N.B: Please use the condensed movies challenge https://github.com/m-bain/CondensedMovies-chall with updated splits since some videos in the original paper are unavailable with missing features ****____
___Contact me directly for the additional dataset queries, details in the challenge repo for feature download.___
###############################################
This repository contains the video dataset, implementation and baselines from Condensed Movies: Story Based Retrieval with Contextual Embeddings.
[Project page](https://www.robots.ox.ac.uk/~vgg/research/condensed-movies) |
[arXiv preprint](https://arxiv.org/abs/2005.04208) |
[Read the paper](https://arxiv.org/pdf/2005.04208.pdf) |
[Preview the data](https://www.robots.ox.ac.uk/~vgg/research/condensed-movies/#preview)
----
### CondensedMovies Dataset
The dataset consists of 3K+ movies, 30K+ professionally captioned clips, 1K+ video hours, 400K+ facetracks & precomputed features from 6 different modalities.
#### Installation
Requirements:
- Storage
- 20GB for features (required for baseline experiments)
- 10GB for facetracks
- 250GB for videos
- Libraries
- ffmpeg (for video download)
- youtube-dl (for video download)
- pandas, numpy
- python 3.6+
#### Prepare Data
1. Navigate to directory `cd CondensedMovies/data_prep/`
2. Edit configuration file `config.json` to download desired subsets of the dataset and their destination.
3. If downloading the source videos (`src: true`), you can edit `youtube-dl.conf` for desired resolution, subtitles etc.
Please see [youtube-dl](https://github.com/ytdl-org/youtube-dl) for more info
4. Run `python download.py`
If you have trouble downloading the source videos or features (due to geographical restrictions or otherwise), please contact me.
### Video-Text Retrieval
#### Baseline (Mixture of Expert Embeddings)
Edit `data_dir` and `save_dir` in `configs/moe.json` for the experiments.
1. `python train.py configs/moe.json`
2. `python test.py --resume $SAVED_EXP_DIR/model_best.pth`
### Visualisation
Run `python visualise_face_tracks.py` with the appropriate arguments to visualise face tracks for a given videoID (requires facetracks and source videos downloaded).
#### TODO:
- [x] youtube download script
- [x] missing videos check
- [x] precomputed features download script
- [x] facetrack visualisation
- [x] dataloader
- [x] video-text retrieval baselines
- [ ] intra-movie baselines + char module
- [ ] release fixed_seg features
#### FAQ
Why did some of the source videos fail to download?
>This is most likely due to geographical restrictions on the videos, email me at maxbain@robots.ox.ac.uk and I can help.
The precomputed features are averaged over the temporal dimension, will you release the original features?
>This is to save space, original features in total are ~1TB, contact me to arrange download of this.
I think clip X is incorrectly identified as being from movie Y, what do do?
>Please let me know any movie identification mistakes and I'll correct it ASAP.
#### Acknowledgements
We would like to thank Samuel Albanie for his help with feature extraction.