{"id":23488914,"url":"https://github.com/sai-soum/diff-mst","last_synced_at":"2025-07-05T03:04:05.454Z","repository":{"id":248018811,"uuid":"606014124","full_name":"sai-soum/Diff-MST","owner":"sai-soum","description":"Multitrack music mixing style transfer given a reference song using differentiable mixing console.  ","archived":false,"fork":false,"pushed_at":"2024-11-09T19:06:11.000Z","size":119063,"stargazers_count":49,"open_issues_count":3,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-15T02:54:09.395Z","etag":null,"topics":["automatic-mixing","ddsp","mixing-audio","style-transfer"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sai-soum.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-24T12:02:53.000Z","updated_at":"2025-03-29T05:03:12.000Z","dependencies_parsed_at":"2024-07-11T23:24:30.493Z","dependency_job_id":"078904c8-eb2d-4c13-99af-d2bed73f6391","html_url":"https://github.com/sai-soum/Diff-MST","commit_stats":null,"previous_names":["sai-soum/diff-mst"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sai-soum%2FDiff-MST","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sai-soum%2FDiff-MST/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sai-soum%2FDiff-MST/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sai-soum%2FDiff-MST/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sai-soum","download_url":"https://codeload.github.com/sai-soum/Diff-MST/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248997090,"owners_count":21195797,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automatic-mixing","ddsp","mixing-audio","style-transfer"],"created_at":"2024-12-24T23:15:38.910Z","updated_at":"2025-04-15T02:54:15.665Z","avatar_url":"https://github.com/sai-soum.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003cdiv align=\"center\"\u003e\n\n# Diff-MST: Differentiable Mixing Style Transfer\n[Paper](https://sai-soum.github.io/assets/pdf/Differentiable_Mixing_Style_Transfer.pdf) | [Website](https://sai-soum.github.io/projects/diffmst/) | [Video](https://youtu.be/w90RGZ3IqQw)\n\n\n\u003cimg src=\"./Assets/diffmst-main_modified.jpg\"\u003e\n\n\u003c/div\u003e\n\n\u003c!-- Mixing style transfer using reference mix. \nThere are two mixing console configurations (in `modules.py`)\n1. `BasicMixConsole`: Gain + Pan\n2. `AdvancedMixConsole`: Gain + Pan + Diff EQ + Diff Compressor\n\nMixes for training can be created using either `naive_random_mix` (assigns random parameter values for mixing console to create a mix) or `knowledge_engineering_mix` (uses knowledge engineering to assign parameter values for mixing console to create a mix). Both of these modules can be found in `mixing.py`\n\n --\u003e\n# Repository Structure\n1. `configs` - Contains configuration files for training and inference.\n2. `mst` - Contains the main codebase for the project.\n    - `dataloaders` - Contains dataloaders for the project.\n    - `modules` - Contains the modules for different components of the system.\n    - `mixing` - Contains the mixing modules for creating mixes.\n    - `loss` - Contains the loss functions for the project.\n    - `panns` - contains the most basic components like cnn14, resnet, etc.\n    - `utils` - Contains utility functions for the project.\n3. `scripts` - Contains scripts for running inference.  \n\n# Setup\n- Clone the repository\n```\ngit clone https://github.com/sai-soum/Diff-MST.git\ncd Diff-MST\n```\n\n- Create new Python environment\n```\n# for Linux/macOS\npython3 -m venv env\nsource env/bin/activate\n```\n\n- Install the `mst` package from source\n```\n# Install as editable (for development)\npip install -e .\n\n# Alternatively, do a regular install (read-only)\npip install .\n```\n\n# Usage\n## Train\nWe use [LightningCLI](https://lightning.ai/docs/pytorch/stable/) for training and [Wandb](https://wandb.ai/site) for logging.\n\n### Setup\nIn the `configs` directory, you will find the configuration files for the project.\n- `config.yaml` - Contains the general configuration for the project.\n- `optimizer.yaml` - Contains the optimizer configuration for the project.\n- `data/` - Contains the data configuration for the project.\n- `models/` - Contains the model configuration for the project.\nWe have provided instructions within the configuration files for setting up the project.\n\nFew important configuration parameters:\n- In `configs/data/` change the following\n    - `track_root_dirs` - The root directory for the dataset needs to be setup. You can pass multiple dataset directories as a list. However, you will also need to provide corresponding metadata YAML files containing train, test, and val split. Check `data/` directory for examples.\n    - For method 1: set `generate_mix` to `True` in the model configuration file. Use `medley+cambridge-8.yaml` for training with random mixes of the same song as reference.\n    - For method 2: set `generate_mix` to `False` in the model configuration file. Use `medley+cambridge+jamendo-8.yaml` for training with real unpaired songs as reference.\n    - update `mix_root_dirs` - The root directory for the mix dataset. This is used for training with real unpaired songs as reference. \n- You may benefit from setting a smaller value for `train_buffer_size_gb` and `val_buffer_size_gb` in the data configuration file for initial testing of the code.\n- In `configs/models/`\n    - you can change the audio effects you want to disable by setting a very large value for the corresponding parameter. For example, to disable the compressor, set `active_compressor_epoch` to `1000`.\n    - You can change the loss function used for training by setting the `loss` parameter.\n- In `optimizer.yaml` you can change the learning rate parameters.\n- In `config.yaml` \n    - Update the directory for logging using `save_dir` under `trainer`.\n    - You can use `ckpt_path` to load a pre-trained model for fine-tuning, resuming training, or testing.\n\n\n### Method 1: Training with random mixes of the same song as reference using MRSTFT loss.\n```\nCUDA_VISIBLE_DEVICES=0 python main.py fit \\\n-c configs/config.yaml \\\n-c configs/optimizer.yaml \\\n-c configs/data/medley+cambridge-8.yaml \\\n-c configs/models/naive.yaml\n```\n\nTo run the fine-tuning using AFLoss\n```\nCUDA_VISIBLE_DEVICES=0 python main.py fit \\\n-c configs/config.yaml \\\n-c configs/optimizer.yaml \\\n-c configs/data/medley+cambridge-8.yaml \\\n-c configs/models/naive+feat.yaml\n```\n\nYou can change the number of tracks, the size of training data for an epoch, and the batch size in the data configuration file located at `configs/data/`\n\n### Method 2: Training with real unpaired songs as reference using AFloss.\n\n```\nCUDA_VISIBLE_DEVICES=0 python main.py fit \\\n-c configs/config.yaml \\\n-c configs/optimizer.yaml \\\n-c configs/data/medley+cambridge+jamendo-8.yaml \\\n-c configs/models/unpaired+feat.yaml\n```\n\n## Inference\nTo evaluate the model on real world data, run the ` scripts/eval_all_combo.py` script. \n\nUpdate the model checkpoints and the inference examples directory in the script. \n\n`Python 3.10` was used for training. \n\n\n## Acknowledgements\nThis work is funded and supported by UK Research and Innovation [grant number EP/S022694/1] and Steinberg Media Technologies GmbH under the AI and Music Centre for Doctoral Training (AIM-CDT) at the Centre for Digital Music, Queen Mary University of London, London, UK. \n\n## Citation\nIf you find this work useful, please consider citing our paper:\n```\n@inproceedings{vanka2024diffmst,\n  title={Diff-MST: Differentiable Mixing Style Transfer},\n  author={Vanka, Soumya and Steinmetz, Christian and Rolland, Jean-Baptiste and Reiss, Joshua and Fazekas, Gy{\\\"o}rgy},\n  booktitle={Proc. of the 25th Int. Society for Music Information Retrieval Conf. (ISMIR)},\n  year={2024},\n  organization={Int. Society for Music Information Retrieval (ISMIR)},\n  abbr = {ISMIR},\n  address = {San Francisco, USA},\n}\n```\n\n## License\nThe code is licensed under the terms of the CC-BY-NC-SA 4.0 license. For a human-readable summary of the license, see https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en .\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsai-soum%2Fdiff-mst","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsai-soum%2Fdiff-mst","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsai-soum%2Fdiff-mst/lists"}