{"id":20213635,"url":"https://github.com/adobe-research/deepafx-st","last_synced_at":"2025-04-06T05:17:27.942Z","repository":{"id":46535951,"uuid":"508095500","full_name":"adobe-research/DeepAFx-ST","owner":"adobe-research","description":"DeepAFx-ST - Style transfer of audio effects with differentiable signal processing. Please see https://csteinmetz1.github.io/DeepAFx-ST/","archived":false,"fork":false,"pushed_at":"2023-05-30T17:32:10.000Z","size":191,"stargazers_count":383,"open_issues_count":9,"forks_count":50,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-03-30T03:11:04.451Z","etag":null,"topics":["adaptive-presets","afx","ai","audio","audio-processing","audio-production","compressor","deeplearning","drc","effects","eq","music-production","styletransfer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adobe-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-27T23:52:28.000Z","updated_at":"2025-03-28T09:59:54.000Z","dependencies_parsed_at":"2024-12-17T11:00:49.997Z","dependency_job_id":null,"html_url":"https://github.com/adobe-research/DeepAFx-ST","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adobe-research%2FDeepAFx-ST","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adobe-research%2FDeepAFx-ST/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adobe-research%2FDeepAFx-ST/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adobe-research%2FDeepAFx-ST/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adobe-research","download_url":"https://codeload.github.com/adobe-research/DeepAFx-ST/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247436290,"owners_count":20938533,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adaptive-presets","afx","ai","audio","audio-processing","audio-production","compressor","deeplearning","drc","effects","eq","music-production","styletransfer"],"created_at":"2024-11-14T06:10:34.115Z","updated_at":"2025-04-06T05:17:27.853Z","avatar_url":"https://github.com/adobe-research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# DeepAFx-ST\n    \nStyle transfer of audio effects with differentiable signal processing\n\n\n[![Demo](https://img.shields.io/badge/Web-Demo-blue)](https://csteinmetz1.github.io/DeepAFx-ST)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](TBD)\n[![arXiv](https://img.shields.io/badge/arXiv-2207.08759-b31b1b.svg)](https://arxiv.org/abs/2207.08759)\n\n\u003c!-- \u003cimg width=\"700px\" src=\"docs/new-generic-style-transfer-headline.svg\"\u003e --\u003e\n \n[Christian J. Steinmetz](http://Christiansteinmetz.com)\u003csup\u003e1*\u003c/sup\u003e, [Nicholas J. Bryan](https://ccrma.stanford.edu/~njb/)\u003csup\u003e2\u003c/sup\u003e, and [Joshua D. Reiss](http://www.eecs.qmul.ac.uk/~josh/)\u003csup\u003e1\u003c/sup\u003e\n\n\u003csup\u003e1\u003c/sup\u003e Centre for Digital Music, Queen Mary University of London\u003cbr\u003e\n\u003csup\u003e2\u003c/sup\u003e Adobe Research \u003cbr\u003e\n\u003csup\u003e*\u003c/sup\u003eWork performed in-part while an intern at Adobe Research.\n\n\n[![Demo Video](docs/deepafx-st-headline.png)](https://youtu.be/IZp455wiMk4)\n    \n\n\u003c/div\u003e\n\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n\n- [Abstract](#abstract)\n- [Install \u0026 Usage](#install--usage)\n- [Inference](#inference)\n- [Training](#training)\n- [Style evaluation](#style-evaluation)\n- [License](#license)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n    \n## Abstract\nWe present a framework that can impose the audio effects and production style from one recording to another by example with the goal of simplifying the audio production process.  We train a deep neural network to analyze an input recording and a style reference recording, and predict the control parameters of audio effects used to render the output.\nIn contrast to past work, we integrate audio effects as differentiable operators in our framework, perform backpropagation through audio effects, and optimize end-to-end using an audio-domain loss. We use a self-supervised training strategy enabling automatic control of audio effects without the use of any labeled or paired training data. We survey a range of existing and new approaches for differentiable signal processing, showing how each can be integrated into our framework while discussing their trade-offs. We evaluate our approach on both speech and music tasks, demonstrating that our approach generalizes both to unseen recordings and even to sample rates different than those seen during training. Our approach produces convincing production style transfer results with the ability to transform input recordings to produced recordings, yielding audio effect control parameters that enable interpretability and user interaction.\n\nFor more details, please see:\n\"[Style Transfer of Audio Effects with Differentiable Signal Processing](https://arxiv.org/abs/2207.08759)\", [Christian J. Steinmetz](http://Christiansteinmetz.com), [Nicholas J. Bryan](https://ccrma.stanford.edu/~njb/), [Joshua D. Reiss](http://www.eecs.qmul.ac.uk/~josh/). arXiv, 2022. If you use ideas or code from this work, pleace cite our paper:\n\n```BibTex\n@article{steinmetz2022style,\n      title={Style Transfer of Audio Effects with Differentiable Signal Processing}, \n      author={Christian J. Steinmetz and Nicholas J. Bryan and Joshua D. Reiss},\n      year={2022},\n      eprint={2207.08759},\n      archivePrefix={arXiv},\n      primaryClass={cs.SD}\n}\n```\n\n\n\n\n\n\n## Install \u0026 Usage\n\nClone the repo, create a virtual environment, and then install the `deepafx_st` package.\n\n```\ncd \u003cDeepAFx-ST\u003e\n\n# Option 1: Using virtual envs\npython -m venv env/\nsource env/bin/activate\n\n# Option 2: Using conda\nconda create -n deepafx-st python=3.8 -y\nconda activate deepafx-st\n\n\n# Update pip and install\npip install --upgrade pip\npip install --pre -e .\n\n# Optional if using AWS for data\npip install awscli\n\n# Linux install\napt-get install libsndfile1\napt-get install sox\napt-get install ffmpeg\napt-get install wget\n```\n\nDownload pretrained models and example files and untar in one shot\n```Bash\ncd \u003cDeepAFx-ST\u003e\nwget https://github.com/adobe-research/DeepAFx-ST/releases/download/v0.1.0/checkpoints_and_examples.tar.gz -O - | tar -xz\n\n```\nNote, you can also find our pretrained checkpoints via the Github UI stored in a tagged release at [https://github.com/adobe-research/DeepAFx-ST/tags](https://github.com/adobe-research/DeepAFx-ST/tags).\n\nAfter download and untar'ing, your `checkpoint` and `examples` folder structures should be the following:\n```Bash\n\u003cDeepAFx-ST\u003e/checkpoints/README.md\n\u003cDeepAFx-ST\u003e/checkpoints/cdpam/\n\u003cDeepAFx-ST\u003e/checkpoints/probes/\n\u003cDeepAFx-ST\u003e/checkpoints/proxies/\n\u003cDeepAFx-ST\u003e/checkpoints/style/\n\u003cDeepAFx-ST\u003e/examples/voice_raw.wav\n\u003cDeepAFx-ST\u003e/examples/voice_produced.wav\n```\n\n\n## Inference\n\nApply pre-trained models to your own audio examples with the `process.py` script.\nSimply call the `scripts/process.py` passing your input audio `-i` along with your reference `-r` and the path to a pretrained model checkpoint `-c.\n\n```\ncd \u003cDeepAFx-ST\u003e\npython scripts/process.py -i \u003cinput_audio\u003e.wav -r \u003cref_audio\u003e.wav -c \u003ccheckpoint.ckpt\u003e \n\n# Speech models\n\n# Autodiff speech model\npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c ./checkpoints/style/libritts/autodiff/lightning_logs/version_1/checkpoints/epoch=367-step=1226911-val-libritts-autodiff.ckpt\n\n# Proxy0 speech model\npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c ./checkpoints/style/libritts/proxy0/lightning_logs/version_0/checkpoints/epoch\\=327-step\\=1093551-val-libritts-proxy0.ckpt\n\n# Proxy2 speech model\npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c ./checkpoints/style/libritts/proxy2/lightning_logs/version_0/checkpoints/epoch\\=84-step\\=283389-val-libritts-proxy2.ckpt \n\n# SPSA speech model\npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c  checkpoints/style/libritts/spsa/lightning_logs/version_2/checkpoints/epoch\\=367-step\\=1226911-val-libritts-spsa.ckpt \n\n# TCN1 speech model\npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c  checkpoints/style/libritts/tcn1/lightning_logs/version_1/checkpoints/epoch\\=367-step\\=1226911-val-libritts-tcn1.ckpt \n\n# TCN2 speech model\npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c  checkpoints/style/libritts/tcn2/lightning_logs/version_1/checkpoints/epoch\\=396-step\\=1323597-val-libritts-tcn2.ckpt \n\n# Music models\n\n# Autodiff music model\npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c checkpoints/style/jamendo/autodiff/lightning_logs/version_0/checkpoints/epoch\\=362-step\\=1210241-val-jamendo-autodiff.ckpt\n\n# Proxy0 music model\npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c checkpoints/style/jamendo/proxy0/lightning_logs/version_0/checkpoints/epoch\\=362-step\\=1210241-val-jamendo-proxy0.ckpt \n\n# proxy0m music model \npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c checkpoints/style/jamendo/proxy0m/lightning_logs/version_0/checkpoints/epoch\\=331-step\\=276887-val-jamendo-proxy0.ckpt \n\n# Proxy2 music model\npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c checkpoints/style/jamendo/proxy2/lightning_logs/version_0/checkpoints/epoch\\=8-step\\=30005-val-jamendo-proxy2.ckpt \n\n# Proxy2m music model\npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c checkpoints/style/jamendo/proxy2m/lightning_logs/version_0/checkpoints/epoch\\=341-step\\=285227-val-jamendo-proxy2.ckpt\n\n# SPSA music model\npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c checkpoints/style/jamendo/spsa/lightning_logs/version_0/checkpoints/epoch\\=362-step\\=1210241-val-jamendo-spsa.ckpt \n\n# TCN1 music model\npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c checkpoints/style/jamendo/tcn1/lightning_logs/version_0/checkpoints/epoch\\=362-step\\=1210241-val-jamendo-tcn1.ckpt \n\n# TCN2 music model\npython scripts/process.py -i examples/voice_raw.wav -r examples/voice_produced.wav -c checkpoints/style/jamendo/tcn2/lightning_logs/version_0/checkpoints/epoch\\=286-step\\=956857-val-jamendo-tcn2.ckpt \n\n```\n\n## Training\n\n### Datasets\n\nTraining and evaluating the models will require one or more of the datasets. Download all the datasets with the following. \n\n```\npython scripts/download.py --datasets daps vctk jamendo libritts musdb --output /path/to/output --download --process\n```\n\nYou can download individual datasets if desired.\n\n```\npython scripts/download.py --datasets daps --output /path/to/output --download --process\n```\nNote, data download can take several days due to the dataset server speeds. We recommend downloading once and making your own storage setup. You will need approx. 1TB of local storage space to download and pre-process all datasets.\n\nFor the style classifcation task we need to render the synthetic style datasets. This can be done for DAPS and MUSDB18 using the [`scripts/run_generate_styles.sh`](scripts/run_generate_styles.sh) script. \nYou will need to update the paths in this script to reflect your local file system and then call the script. \n\n```\n./script/run_generate_styles.sh`\n```\n\n### Style transfer\n\nA number of predefined training configurations are defined in bash scripts in the `configs/` directory.\nWe perform experiments on speech using [LibriTTS] and on music using the [MTG-Jamendo dataset](). \nBy default, this will train 6 different model configurations using a different method for differentiable .\nThis will place one job on each GPU, and assumes at least 6 GPUs, each with at least 16 GB of VRAM.\nYou can launch training by calling the appropriate \n\n```\n./configs/train_all_libritts_style.sh\n./configs/train_all_jamendo_style.sh\n```\nNote, you will need to modify the data paths in the scripts above to fit your setup.\n\nThere are four main configurations for training the style transfer models. \nThis is specified by the `--processor_model` flag when launching the training script, and\nmust be one of the following:\n\n1. `tcn1` - End-to-end audio processing neural network (1 network) with control parameters.\n2. `tcn2` - End-to-end audio processing neural network (2 networks) with control parameters.\n3. `proxy0` - Neural network proxies with control parameters for audio processing.\n4. `proxy2` - Neural network proxies with control parameters for audio processing.\n5. `spsa` - Gradient estimation with DSP audio effects using SPSA methods.\n6. `autodiff` - DSP audio effects implemnted directly in PyTorch.\n\n\n### Proxy training\n\nIf desired you can also re-train the neural network audio effect proxies. \nTraining the controller with neural proxies requires first pretrianing the proxy networks to emulate a parametric EQ and dynamic range compressor.\nCalling the proxy pre-training scripts in the `configs/` directly will train these models using the training set from VCTK.\n\n```\n./configs/train_all_proxies.sh\n```\n\n### Probe training\n\nLinear probes for the production style classification task can be trained with the following script.\n\n```\n./configs/train_all_probes.sh\n```\nNote, some additional data pre-processing is required and needs updating.\n\n## Style evaluation\n\nEvaluating the pretrained models along with the baselines is carried out with the `eval.py` script.\nA predefined evaluation configuration to reproduce our results can be called as follows. \nBe sure to update the paths at the top of the script to reflect the location of the datasets. \n\n```\n./configs/eval_style.sh\n```\n\nTo evaluate a set of models call the script passing the directory containing the pretrained checkpoints.\nThe following example demonstrates evaluating models trained on daps.\n\n### Probe evaluation\n\nPretrained linear probes can be evaluated separately with their own script.\n\n```\n./configs/eval_probes.sh\n```\n\n### Timing \n\nWe compute timings on both CPU and GPU for the different approaches using `python scripts/timing.py`.\n\n```\nrb_infer : sec/step 0.0186    0.0037 RTF\ndsp_infer : sec/step 0.0172    0.0034 RTF\nautodiff_cpu_infer : sec/step 0.0295    0.0059 RTF\nautodiff_gpu_infer : sec/step 0.0049    0.0010 RTF\ntcn1_cpu_infer : sec/step 0.6580    0.1316 RTF\ntcn2_cpu_infer : sec/step 1.3409    0.2682 RTF\ntcn1_gpu_infer : sec/step 0.0114    0.0023 RTF\ntcn2_gpu_infer : sec/step 0.0223    0.0045 RTF\nautodiff_gpu_grad : sec/step 0.3086    0.0617 RTF\nnp_norm_gpu_grad : sec/step 0.4346    0.0869 RTF\nnp_hh_gpu_grad : sec/step 0.4379    0.0876 RTF\nnp_fh_gpu_grad : sec/step 0.4339    0.0868 RTF\ntcn1_gpu_grad : sec/step 0.4382    0.0876 RTF\ntcn2_gpu_grad : sec/step 0.6424    0.1285 RTF\nspsa_gpu_grad : sec/step 0.4132    0.0826 RTF\n```\n\nThe above results were from a machine with the following configuration. \n\n```\nIntel(R) Xeon(R) CPU E5-2623 v3 @ 3.00GHz (16 core)\nGeForce GTX 1080 Ti\n```\n\n\n## License\nUnless otherwise specified via local comments per file, all code and models are licensed via the [Adobe Research License](LICENSE). Copyright (c) Adobe Systems Incorporated. All rights reserved.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadobe-research%2Fdeepafx-st","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadobe-research%2Fdeepafx-st","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadobe-research%2Fdeepafx-st/lists"}