Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/cyrusvahidi/jtfs-gpu

code for the paper "Differentiable Time-Frequency Scattering on GPU" 🌊
https://github.com/cyrusvahidi/jtfs-gpu

Last synced: 6 days ago
JSON representation

code for the paper "Differentiable Time-Frequency Scattering on GPU" 🌊

Host: GitHub
URL: https://github.com/cyrusvahidi/jtfs-gpu
Owner: cyrusvahidi
License: mit
Created: 2022-02-09T09:44:00.000Z (over 2 years ago)
Default Branch: main
Last Pushed: 2023-11-16T16:47:04.000Z (12 months ago)
Last Synced: 2024-10-10T12:50:52.224Z (29 days ago)
Language: Jupyter Notebook
Homepage: https://cyrusvahidi.github.io/jtfs-gpu/
Size: 1.15 GB
Stars: 53
Watchers: 5
Forks: 4
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE.md

Awesome Lists containing this project

README

# Differentiable Time-Frequency Scattering on GPU 🌊

### source code by
[Cyrus Vahidi]()²

[Changhong Wang]()¹, [Han Han]()¹
[Vincent Lostanlen]()¹

[John Muradeli]()

LS2N/CNRS Centrale Nantes¹ Queen Mary University of London²

[![Paper](http://img.shields.io/badge/paper-arxiv.2204.08269-B31B1B.svg)](https://arxiv.org/abs/2204.08269)

![Accompanying webpage 🌐](https://cyrusvahidi.github.io/jtfs-gpu/)

![Kymatio: open-source wavelet scattering in Python ‍💻](https://github.com/kymatio/kymatio/)

Many thanks to all open-source contributors to ![Kymatio](https://github.com/kymatio/kymatio) and its dependencies.

This repository contains code to replicate the paper "Differentiable Time-Frequency Scattering on GPU" (published at DAFx 2022, best paper award).

Time-frequency scattering is available in ![Kymatio](https://github.com/kymatio/kymatio/) in beta and will be released in v0.4. To use this implementation, install Kymatio from source. To replicate the results in this paper, follow the installation instructions below.

We assess Time-Frequency Scattering in Kymatio for 3 machine listening research applications:

* unsupervised manifold learning of spectrotemporal modulations
* hybrid jtfs + convnet supervised musical instrument classification
* texture resynthesis

## How to run

First, install dependencies

### Installation

```bash
# clone project
git clone https://github.com/cyrusvahidi/jtfs-gpu

# install project
cd jtfs-gpu
pip install -e .
pip install -r requirements.txt

# install kymatio from source
cd kymatio
python setup.py develop
```

* The JTFS algorithm source code to replicate the paper can be found ![here](https://github.com/overLordGoldDragon/wavespin/tree/dafx2022-jtfs)
* The latest version of JTFS can be installed directly from the ![Kymatio](https://github.com/kymatio/kymatio/) source

### ConvNet Classifier

[Download Medley-solos-DB](https://zenodo.org/record/3464194)

#### Extract Scattering Features
``` bash
python scripts/process_msdb_features.py --data_dir --feature
```

#### Configure gin
In `/scripts/gin/config.gin` set `MSDB_DATA_DIR` and `MSDB_CSV` according to the absolute path of your MSDB download.

#### Run training

``` bash
python scripts/train_cnn.py
```

### Isomap Visualizations & K-NN Regression
``` bash
python scripts/isomap.py
```
see the output in `/img`

### Scale-Rate Visualizations and Resynthesis
```
cd notebooks
jupyter notebook
```
See `Scale-Rate Visualization.ipynb` and `Resynthesis results.ipynb`

### Notebook Guide

#### Scale-Rate Visualisations

H E L L O

#### Synthetic amplitude-modulated chirp dataset
* Factors of variation:
* $f_c$ carrier frequency
* $f_m$ amplitude modulation frequency
* $\gamma$ chirp rate

#### Manifold Embedding of the Nearest Neighbour Graph
* MFCCs
* Time Scattering
* Time-Frequency Scattering
* Open-L3
* Spectrotemporal Receptive Field (STRF)

#### K-NN regression of synthesizer parameters

#### 2-D CNN classifier

#### Differentiable Resynthesis

### Citation
```
@article{muradeli2022differentiable,
title={Differentiable Time-Frequency Scattering in Kymatio},
author={John Muradeli, Cyrus Vahidi, Changhong Wang, Han Han, Vincent Lostanlen, Mathieu Lagrange, George Fazekas},
journal={arXiv preprint arXiv:2204.08269},
year={2022}
}
```