{"id":15122639,"url":"https://github.com/damrsn/neuralnote","last_synced_at":"2025-10-25T06:02:58.653Z","repository":{"id":147108198,"uuid":"609103288","full_name":"DamRsn/NeuralNote","owner":"DamRsn","description":"Audio Plugin for Audio to MIDI transcription using deep learning.","archived":false,"fork":false,"pushed_at":"2025-01-16T10:00:44.000Z","size":9494,"stargazers_count":1746,"open_issues_count":12,"forks_count":96,"subscribers_count":48,"default_branch":"master","last_synced_at":"2025-04-12T13:57:32.730Z","etag":null,"topics":["audio","audio-plugin","juce-framework","machine-learning","midi","vst"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DamRsn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-03T11:36:02.000Z","updated_at":"2025-04-12T10:40:08.000Z","dependencies_parsed_at":"2024-06-01T19:30:53.589Z","dependency_job_id":"3e18a10e-2c86-4ade-bae9-2c612d398f41","html_url":"https://github.com/DamRsn/NeuralNote","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":"eyalamirmusic/JUCECmakeRepoPrototype","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DamRsn%2FNeuralNote","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DamRsn%2FNeuralNote/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DamRsn%2FNeuralNote/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DamRsn%2FNeuralNote/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DamRsn","download_url":"https://codeload.github.com/DamRsn/NeuralNote/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254149957,"owners_count":22022851,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["audio","audio-plugin","juce-framework","machine-learning","midi","vst"],"created_at":"2024-09-26T02:01:13.001Z","updated_at":"2025-10-25T06:02:53.599Z","avatar_url":"https://github.com/DamRsn.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NeuralNote \u003cimg style=\"float: right;\" src=\"NeuralNote/Assets/logo.png\" width=\"100\" /\u003e\n\nNeuralNote is the audio plugin that brings **state-of-the-art Audio to MIDI conversion** into\nyour favorite Digital Audio Workstation.\n\n- Works with any tonal instrument (voice included)\n- Supports polyphonic transcription\n- Supports pitch bend detection\n- Lightweight and very fast transcription\n- Allows to adjust the parameters while listening to the transcription\n- Allows to scale and time quantize transcribed MIDI directly in the plugin\n\n## Install NeuralNote\n\nDownload the latest release for your platform [here](https://github.com/DamRsn/NeuralNote/releases) (Windows, macOS (\nUniversal) and Linux supported)!\n\nInstallers are available for both Windows and Mac, including Standalone, VST3, and AU (Mac only) versions. The\ninstallers allow users to select which format(s) they want to install. On macOS, the code is signed, while on Windows,\nit is not. This means you may need to take a few additional steps to use NeuralNote on Windows.\n\nFor Linux, raw binaries are provided for VST3 and Standalone. You can install them by copying the files to the\nappropriate locations.\n\n## Usage\n\n![UI](NeuralNote_UI.png)\n\nNeuralNote comes as a simple AudioFX plugin (VST3/AU/Standalone app) to be applied on the track to transcribe.\n\nThe workflow is very simple:\n\n- Gather some audio\n    - Click record. Works when recording for real or when playing the track in a DAW.\n    - Or drop an audio file on the plugin. (.wav, .aiff, .flac, .mp3 and .ogg (vorbis) supported)\n- The MIDI transcription instantly appears in the piano roll section.\n- Listen to the result by clicking the play button.\n    - Play with the different settings to adjust the transcription, even while listening to it\n    - Individually adjust the level of the source audio and of the synthesized transcription\n- Once you're satisfied, export the MIDI transcription with a simple drag and drop from the plugin to a MIDI track.\n\n**Watch our presentation video for the Neural Audio Plugin\ncompetition [here](https://www.youtube.com/watch?v=6_MC0_aG_DQ)**.\n\nNeuralNote uses internally the model from Spotify's [basic-pitch](https://github.com/spotify/basic-pitch). See\ntheir [blogpost](https://engineering.atspotify.com/2022/06/meet-basic-pitch/)\nand [paper](https://arxiv.org/abs/2203.09893) for more information. In NeuralNote, basic-pitch is run\nusing [RTNeural](https://github.com/jatinchowdhury18/RTNeural) for the CNN part\nand [ONNXRuntime](https://github.com/microsoft/onnxruntime) for the feature part (Constant-Q transform calculation +\nHarmonic Stacking).\nAs part of this project, [we contributed to RTNeural](https://github.com/jatinchowdhury18/RTNeural/pull/89) to add 2D\nconvolution support.\n\n## Build from source\n\nRequirements are: `git`, `cmake`, and your OS's preferred compiler suite.\n\nUse this when cloning:\n\n```\ngit clone --recurse-submodules --shallow-submodules https://github.com/DamRsn/NeuralNote\n ```\n\nThe following OS-specific build scripts have to be executed at least once before being able to use the project as a\nnormal CMake project. The script downloads onnxruntime static library (that we created\nwith [ort-builder](https://github.com/olilarkin/ort-builder)) before calling CMake.\n\n#### macOS\n\n```\n$ ./build.sh\n```\n\n#### Windows\n\nDue to [a known issue](https://github.com/DamRsn/NeuralNote/issues/21), if you're not using Visual Studio 2022 (MSVC\nversion: 19.35.x, check `cl` output), then you'll need to manually build onnxruntime.lib like so:\n\n1. Ensure you have Python installed; if not, download at https://www.python.org/downloads/windows/ (this does not\n   currently work with Python 3.11, prefer Python 3.10).\n\n2. Execute each of the following lines in a command prompt:\n\n```\ngit clone --depth 1 --recurse-submodules --shallow-submodules https://github.com/tiborvass/libonnxruntime-neuralnote ThirdParty\\onnxruntime\ncd ThirdParty\\onnxruntime\npython3 -m venv venv\n.\\venv\\Scripts\\activate.bat\npip install -r requirements.txt\n.\\convert-model-to-ort.bat model.onnx\n.\\build-win.bat model.required_operators_and_types.with_runtime_opt.config\ncopy model.with_runtime_opt.ort ..\\..\\Lib\\ModelData\\features_model.ort\ncd ..\\..\n```\n\nNow you can get back to building NeuralNote as follows:\n\n```\n\u003e .\\build.bat\n```\n\n#### IDEs\n\nOnce the build script has been executed at least once, you can load this project in your favorite IDE\n(CLion/Visual Studio/VSCode/etc) and click 'build' for one of the targets.\n\n## Reuse code from NeuralNote’s transcription engine\n\nAll the code to perform the transcription is in `Lib/Model` and all the model weights are in `Lib/ModelData/`. Feel free\nto use only this part of the code in your own project! We'll try to isolate it more from the rest of the repo in the\nfuture and make it a library.\n\nThe code to generate the files in `Lib/ModelData/` is not currently available as it required a lot of manual operations.\nBut here's a description of the process we followed to create those files:\n\n- `features_model.onnx` was generated by converting a keras model containing only the CQT + Harmonic Stacking part of\n  the full basic-pitch graph using `tf2onnx` (with manually added weights for batch normalization).\n- the `.json` files containing the weights of the basic-pitch cnn were generated from the tensorflow-js model available\n  in the [basic-pitch-ts repository](https://github.com/spotify/basic-pitch-ts), then converted to onnx with `tf2onnx`.\n  Finally, the weights were gathered manually to `.npy` thanks to [Netron](https://netron.app/) and finally applied to a\n  split keras model created with [basic-pitch](https://github.com/spotify/basic-pitch) code.\n\nThe original basic-pitch CNN was split in 4 sequential models wired together, so they can be run with RTNeural.\n\n## Bug reports and feature requests\n\nIf you have any request/suggestion concerning the plugin or encounter a bug, please file a GitHub issue.\n\n## Contributing\n\nContributions are most welcome! If you want to add some features to the plugin or simply improve the documentation,\nplease open a PR!\n\n## License\n\nNeuralNote software and code is published under the Apache-2.0 license. See the [license file](LICENSE).\n\n#### Third Party libraries used and license\n\nHere's a list of all the third party libraries used in NeuralNote and the license under which they are used.\n\n- [JUCE](https://juce.com/) (JUCE Starter)\n- [RTNeural](https://github.com/jatinchowdhury18/RTNeural) (BSD-3-Clause license)\n- [ONNXRuntime](https://github.com/microsoft/onnxruntime) (MIT License)\n- [ort-builder](https://github.com/olilarkin/ort-builder) (MIT License)\n- [basic-pitch](https://github.com/spotify/basic-pitch) (Apache-2.0 license)\n- [basic-pitch-ts](https://github.com/spotify/basic-pitch-ts) (Apache-2.0 license)\n- [minimp3](https://github.com/lieff/minimp3) (CC0-1.0 license)\n\n## Could NeuralNote transcribe audio in real-time?\n\nUnfortunately no and this for a few reasons:\n\n- Basic Pitch uses the Constant-Q transform (CQT) as input feature. The CQT requires really long audio chunks (\u003e 1s) to\n  get amplitudes for the lowest frequency bins. This makes the latency too high to have real-time transcription.\n- The basic pitch CNN has an additional latency of approximately 120ms.\n- The note events creation algorithm processes the posteriorgrams backward (from future to past) and is hence\n  non-causal.\n\nBut if you have ideas please share!\n\n## Credits\n\nNeuralNote was developed by [Damien Ronssin](https://github.com/DamRsn) and [Tibor Vass](https://github.com/tiborvass).\nThe plugin user interface was designed by Perrine Morel.\n\n#### Contributors\n\nMany thanks to the contributors!\n\n- [jatinchowdhury18](https://github.com/jatinchowdhury18): File browser.\n- [trirpi](https://github.com/trirpi)\n    - More scale options in `SCALE QUANTIZE`.\n    - Horizontal zoom for the audio waveform and the piano roll.\n- [polygon](https://github.com/polygon) and [SamuMazzi](https://github.com/SamuMazzi): Linux support.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdamrsn%2Fneuralnote","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdamrsn%2Fneuralnote","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdamrsn%2Fneuralnote/lists"}