{"id":17618165,"url":"https://github.com/kinwaicheuk/reconvat","last_synced_at":"2025-04-30T19:47:09.726Z","repository":{"id":49436257,"uuid":"383326680","full_name":"KinWaiCheuk/ReconVAT","owner":"KinWaiCheuk","description":"ReconVAT: a semi-supervised automatic music transcription (AMT) model ","archived":false,"fork":false,"pushed_at":"2024-01-19T07:19:57.000Z","size":7356,"stargazers_count":36,"open_issues_count":3,"forks_count":5,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-30T19:23:03.575Z","etag":null,"topics":["automatic-music-transcription","deep-learning","music","pytorch","semi-supervised-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KinWaiCheuk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-06T03:21:53.000Z","updated_at":"2025-02-26T14:54:58.000Z","dependencies_parsed_at":"2024-01-19T09:17:10.633Z","dependency_job_id":"cc291718-e70f-4554-9dbb-8b1292e870c8","html_url":"https://github.com/KinWaiCheuk/ReconVAT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KinWaiCheuk%2FReconVAT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KinWaiCheuk%2FReconVAT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KinWaiCheuk%2FReconVAT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KinWaiCheuk%2FReconVAT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KinWaiCheuk","download_url":"https://codeload.github.com/KinWaiCheuk/ReconVAT/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251773799,"owners_count":21641698,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automatic-music-transcription","deep-learning","music","pytorch","semi-supervised-learning"],"created_at":"2024-10-22T19:26:46.595Z","updated_at":"2025-04-30T19:47:09.676Z","avatar_url":"https://github.com/KinWaiCheuk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ReconVAT\nThis repository is for the paper **ReconVAT: A Semi-Supervised Automatic Music Transcription Framework Towards Real-World Applications**.\n\nDemo page is available at: https://kinwaicheuk.github.io/ReconVAT/\n\nSupplementary Materials: TODO\n\n\n## Requirement\nYou can install the following libraries at once via `pip install -r requirements.txt`.\n\n\n## Using pretrained models\n\nFor your convenience, we have provided 3 example audio clips for you to try our models out. But to transcribe your own music, you need to first downsample them to 16kHz and save them as Flac format. Then simply put your audio clips in the path `Application/Input`, then run the follow code:\n```python\npython transcribe_files.py with model_type=\u003carg\u003e device=\u003carg\u003e\n```\n\n* `model_type`: Pick the model to transcribe your music. `ReconVAT` or `baseline_Multi_Inst`. Default is `ReconVAT`.\n* `device`: the device to be trained on. Either `cpu` or `cuda:0`. Default is `cuda:0`\n\nYou might also need to install ffmpeg in order to do audio downsampling.\nOn macOS:\n```python\nbrew install ffmpeg\n```\n\nOn Linux:\n```\nApt-get install ffmpeg\n```\n\n## Training from scratch\n### Step1: Downloading Dataset \nMAPS dataset (as labelled dataset in our experiments): [download](https://adasp.telecom-paris.fr/resources/2010-07-08-maps-database/)\n\nMAESTRO (we use v2.0.0 as our unlabelled dataset in our experiments): [download](https://magenta.tensorflow.org/datasets/maestro)\n\nMusicNet dataset (for training strings and woodwinds): [download](https://homes.cs.washington.edu/~thickstn/musicnet.html)\n\nAfter downloading these dataset, unzip them to their respective folders `MAPS`, `MAESTRO`, and `MusicNet`.\n\n### Step2: Preprocessing\nOur model takes 16kHz audio as the input, therefore we need to downsample all the audio clips first. Our model also takes tsv files as the labels, so we also need to convert midi files into tsv files.\n\nThese preprocessing functions can be found in the jupyter notebook named as `Preprocessing.ipynb`.\n\nWhen the dataset is ready, the PyTorch Dataset class should be able toload these datasets without errors.\n\n### Step3: Training the model\nThe python script can be run using using the sacred syntax `with`.\n\nUnet_VAT mode:\n```python\npython train_UNet_VAT.py with train_on=\u003carg\u003e small=\u003carg\u003e VAT=\u003carg\u003e reconstruction=\u003carg\u003e device=\u003carg\u003e\n```\n\nUnet_VAT with the onset module:\n```python\npython train_UNet_Onset_VAT.py with train_on=\u003carg\u003e small=\u003carg\u003e VAT=\u003carg\u003e reconstruction=\u003carg\u003e device=\u003carg\u003e\n```\n\nBaseline model Multi-instrument:\n```python\npython train_baseline_Multi_Inst.py with train_on=\u003carg\u003e small=\u003carg\u003e device=\u003carg\u003e\n```\n\nOnsets and Frames: (VAT can be activated in this baseline model, but according to our experiments, VAT does not work with this baseline model)\n```python\npython train_baseline_onset_frame_VAT.py with train_on=\u003carg\u003e small=\u003carg\u003e device=\u003carg\u003e\n```\n\nThe following two baseline model requires a huge amount of GPU memory\n\nThickstun:\n```python\npython train_baseline_Thickstun.py with train_on=\u003carg\u003e small=\u003carg\u003e device=\u003carg\u003e\n```\n\nPrestack:\n```python\npython train_baseline_Prestack.py with train_on=\u003carg\u003e small=\u003carg\u003e device=\u003carg\u003e\n```\n\n\n* `train_on`: the dataset to be trained on. Either `MAPS` or `String` or `Wind`\n* `small`: Activate the small version of MAPS. `True` or `False`\n* `supersmall`: Activate the oneshot version of MAPS. The `small` argument has to be `True` in order for this argument to be useful\n* `reconstruction`: to include the reconstruction loss or not. Either `True` or `False`\n* `VAT`: VAT module, `True` or `False`\n* `device`: the device to be trained on. Either `cpu` or `cuda:0`\n\n## Evaluating the model and exporting the midi files\n\n```python\npython evaluate.py with weight_file=\u003carg\u003e reconstruction=\u003carg\u003e device=\u003carg\u003e\n```\n\n* `weight_file`: The weight files should be located inside the `trained_weight` folder\n* `dataset`: which dataset to evaluate on, can be either `MAPS` or `MAESTRO` or `MusicNet`.\n* `device`: the device to be trained on. Either `cpu` or `cuda:0`\n\nThe transcripted midi files, accuracy reports are saved inside the `results` folder.\n\n## Other notes\nFor Mac users, you need to add the following code in helper_functions.py\n\n```python\nimport matplotlib\nmatplotlib.use('TkAgg')\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkinwaicheuk%2Freconvat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkinwaicheuk%2Freconvat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkinwaicheuk%2Freconvat/lists"}