Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/hangzhaomit/sound-of-pixels

Codebase for ECCV18 "The Sound of Pixels"
https://github.com/hangzhaomit/sound-of-pixels

computer-vision cross-modality self-supervised-learning sound-separation

Last synced: 4 days ago
JSON representation

Codebase for ECCV18 "The Sound of Pixels"

Host: GitHub
URL: https://github.com/hangzhaomit/sound-of-pixels
Owner: hangzhaomit
License: mit
Created: 2018-11-18T23:49:14.000Z (about 6 years ago)
Default Branch: master
Last Pushed: 2022-04-25T06:26:41.000Z (over 2 years ago)
Last Synced: 2024-12-30T23:17:41.910Z (11 days ago)
Topics: computer-vision, cross-modality, self-supervised-learning, sound-separation
Language: Python
Homepage: http://sound-of-pixels.csail.mit.edu
Size: 1.24 MB
Stars: 373
Watchers: 15
Forks: 74
Open Issues: 11
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# Sound-of-Pixels
Codebase for ECCV18 "The Sound of Pixels".

*This repository is under construction, but the core parts are already there.

## Environment
The code is developed under the following configurations.
- Hardware: 1-4 GPUs (change ```[--num_gpus NUM_GPUS]``` accordingly)
- Software: Ubuntu 16.04.3 LTS, ***CUDA>=8.0, Python>=3.5, PyTorch>=0.4.0***

## Training
1. Prepare video dataset.

a. Download MUSIC dataset from: https://github.com/roudimit/MUSIC_dataset

b. Download videos.

2. Preprocess videos. You can do it in your own way as long as the index files are similar.

b. Make training/validation index files by running:
```
python scripts/create_index_files.py
```
It will create index files ```train.csv```/```val.csv``` with the following format:
```
./data/audio/acoustic_guitar/M3dekVSwNjY.mp3,./data/frames/acoustic_guitar/M3dekVSwNjY.mp4,1580
./data/audio/trumpet/STKXyBGSGyE.mp3,./data/frames/trumpet/STKXyBGSGyE.mp4,493
```
For each row, it stores the information: ```AUDIO_PATH,FRAMES_PATH,NUMBER_FRAMES```

3. Train the default model.
```bash
./scripts/train_MUSIC.sh
```

5. During training, visualizations are saved in HTML format under ```ckpt/MODEL_ID/visualization/```.

## Evaluation
0. (Optional) Download our trained model weights for evaluation.
```bash
./scripts/download_trained_model.sh
```

1. Evaluate the trained model performance.
```bash
./scripts/eval_MUSIC.sh
```

## Reference
If you use the code or dataset from the project, please cite:
```bibtex
@InProceedings{Zhao_2018_ECCV,
author = {Zhao, Hang and Gan, Chuang and Rouditchenko, Andrew and Vondrick, Carl and McDermott, Josh and Torralba, Antonio},
title = {The Sound of Pixels},
booktitle = {The European Conference on Computer Vision (ECCV)},
month = {September},
year = {2018}
}
```