{"id":16402553,"url":"https://github.com/spijkervet/clmr","last_synced_at":"2025-04-05T18:10:31.195Z","repository":{"id":40202445,"uuid":"254389163","full_name":"Spijkervet/CLMR","owner":"Spijkervet","description":"Official PyTorch implementation of Contrastive Learning of Musical Representations","archived":false,"fork":false,"pushed_at":"2024-07-25T11:19:27.000Z","size":81161,"stargazers_count":318,"open_issues_count":16,"forks_count":50,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-29T17:09:09.112Z","etag":null,"topics":["contrastive-learning","music-classification","music-information-retrieval","self-supervised-learning"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2103.09410","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Spijkervet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-09T14:10:00.000Z","updated_at":"2025-03-17T03:17:43.000Z","dependencies_parsed_at":"2024-11-16T19:02:06.366Z","dependency_job_id":null,"html_url":"https://github.com/Spijkervet/CLMR","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spijkervet%2FCLMR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spijkervet%2FCLMR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spijkervet%2FCLMR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spijkervet%2FCLMR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Spijkervet","download_url":"https://codeload.github.com/Spijkervet/CLMR/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247378149,"owners_count":20929297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["contrastive-learning","music-classification","music-information-retrieval","self-supervised-learning"],"created_at":"2024-10-11T05:46:33.556Z","updated_at":"2025-04-05T18:10:31.131Z","avatar_url":"https://github.com/Spijkervet.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Contrastive Learning of Musical Representations\n\nPyTorch implementation of [Contrastive Learning of Musical Representations](https://arxiv.org/abs/2103.09410) by Janne Spijkervet and John Ashley Burgoyne.\n\n![CLMR](https://github.com/spijkervet/clmr/actions/workflows/clmr.yml/badge.svg)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1Njz8EoN4br587xjpRKcssMuqQY6Cc5nj#scrollTo=aeKVT59FhWzV)\n\n[![arXiv](https://img.shields.io/badge/arXiv-2103.09410-b31b1b.svg)](https://arxiv.org/abs/2103.09410)\n[![Supplementary Material](https://img.shields.io/badge/Supplementary%20Material-2103.09410-blue.svg)](https://github.com/Spijkervet/CLMR/releases/download/2.1/CLMR.-.Supplementary.Material.pdf)\n\n\u003cdiv class=\"header-image\" style=\"display: flex; flex-direction: column; align-items: center;\"\u003e\n  \u003ca href=\"https://spijkervet.github.io/CLMR/examples/clmr-onnxruntime-web\" target=\"_blank\" style=\"text-decoration: none;\"\u003e\n      \u003cspan style=\"font-size: 1.5rem; font-weight: 100;\"\u003eCLMR x\u003c/span\u003e\n      \u003cimg class=\"onnx-image\" style=\"height: 20px;\" src=\"https://onnxruntime.ai/images/svg/ONNX-Runtime-logo.svg\"/\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\nYou can run a pre-trained CLMR model directly from within your browser using ONNX Runtime: [here](https://spijkervet.github.io/CLMR/examples/clmr-onnxruntime-web).\n\n\nIn this work, we introduce SimCLR to the music domain and contribute a large chain of audio data augmentations, to form a simple framework for self-supervised learning of raw waveforms of music: CLMR. We evaluate the performance of the self-supervised learned representations on the task of music classification.\n\n- We achieve competitive results on the MagnaTagATune and Million Song Datasets relative to fully supervised training, despite only using a linear classifier on self-supervised learned representations, i.e., representations that were learned task-agnostically without any labels.\n- CLMR enables efficient classification: with only 1% of the labeled data, we achieve similar scores compared to using 100% of the labeled data.\n- CLMR is able to generalise to out-of-domain datasets: when training on entirely different music datasets, it is still able to perform competitively compared to fully supervised training on the target dataset.\n\n*This is the CLMR v2 implementation, for the original implementation go to the [`v1`](https://github.com/Spijkervet/CLMR/tree/v1) branch*\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"50%\" alt=\"CLMR model\" src=\"https://github.com/Spijkervet/CLMR/blob/master/media/clmr_model.png?raw=true\"\u003e\n\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n  An illustration of CLMR.\n\u003c/div\u003e\n\n\nThis repository relies on my SimCLR implementation, which can be found [here](https://github.com/spijkervet/simclr) and on my `torchaudio-augmentations` package, found [here](https://github.com/Spijkervet/torchaudio-augmentations).\n\n\n\n## Quickstart\n```\ngit clone https://github.com/spijkervet/clmr.git \u0026\u0026 cd clmr\n\npip3 install -r requirements.txt\n# or\npython3 setup.py install\n```\n\nThe following command downloads MagnaTagATune, preprocesses it and starts self-supervised pre-training on 1 GPU (with 8 simultaneous CPU workers) and linear evaluation:\n```\npython3 preprocess.py --dataset magnatagatune\n\n# add --workers 8 to increase the number of parallel CPU threads to speed up online data augmentations + training.\npython3 main.py --dataset magnatagatune --gpus 1 --workers 8\n\npython3 linear_evaluation.py --gpus 1 --workers 8 --checkpoint_path [path to checkpoint.pt, usually in ./runs]\n```\n\n## Pre-train on your own folder of audio files\nSimply run the following command to pre-train the CLMR model on a folder containing .wav files (or .mp3 files when editing `src_ext_audio=\".mp3\"` in `clmr/datasets/audio.py`). You may need to convert your audio files to the correct sample rate first, before giving it to the encoder (which accepts `22,050Hz` per default).\n\n```\npython preprocess.py --dataset audio --dataset_dir ./directory_containing_audio_files\n\npython main.py --dataset audio --dataset_dir ./directory_containing_audio_files\n```\n\n\n## Results\n\n### MagnaTagATune\n\n| Encoder / Model | Batch-size / epochs | Fine-tune head |  ROC-AUC |  PR-AUC |\n|-------------|-------------|-------------|-------------|-------------|\n| SampleCNN / CLMR | 48 / 10000 | Linear Classifier | 88.7 | **35.6** |\nSampleCNN / CLMR | 48 / 10000 | MLP (1 extra hidden layer) |  **89.3** | **36.0** |\n| [SampleCNN (fully supervised)](https://www.mdpi.com/2076-3417/8/1/150) | 48 / - | - | 88.6 | 34.4 |\n| [Pons et al. (fully supervised)](https://arxiv.org/pdf/1711.02520.pdf) | 48 / - | - | 89.1 | 34.92 |\n\n### Million Song Dataset\n\n| Encoder / Model | Batch-size / epochs | Fine-tune head |  ROC-AUC |  PR-AUC |\n|-------------|-------------|-------------|-------------|-------------|\n| SampleCNN / CLMR | 48 / 1000 | Linear Classifier | 85.7 | 25.0 |\n| [SampleCNN (fully supervised)](https://www.mdpi.com/2076-3417/8/1/150) | 48 / - | - | **88.4** | - |\n| [Pons et al. (fully supervised)](https://arxiv.org/pdf/1711.02520.pdf) | 48 / - | - | 87.4 | **28.5** |\n\n\n## Pre-trained models\n*Links go to download*\n\n| Encoder (batch-size, epochs) | Fine-tune head | Pre-train dataset | ROC-AUC | PR-AUC\n|-------------|-------------|-------------|-------------|-------------|\n[SampleCNN (96, 10000)](https://github.com/Spijkervet/CLMR/releases/download/2.0/clmr_checkpoint_10000.zip) | [Linear Classifier](https://github.com/Spijkervet/CLMR/releases/download/2.0/finetuner_checkpoint_200.zip) | MagnaTagATune |  88.7 (89.3) | 35.6 (36.0)\n[SampleCNN (48, 1550)](https://github.com/Spijkervet/CLMR/releases/download/1.0/clmr_checkpoint_1550.pt) | [Linear Classifier](https://github.com/Spijkervet/CLMR/releases/download/1.0-l/finetuner_checkpoint_20.pt) | MagnaTagATune | 87.71 (88.47) | 34.27 (34.96)\n\n## Training\n### 1. Pre-training\nSimply run the following command to pre-train the CLMR model on the MagnaTagATune dataset.\n```\npython main.py --dataset magnatagatune\n```\n\n### 2. Linear evaluation\nTo test a trained model, make sure to set the `checkpoint_path` variable in the `config/config.yaml`, or specify it as an argument:\n```\npython linear_evaluation.py --checkpoint_path ./clmr_checkpoint_10000.pt\n```\n\n## Configuration\nThe configuration of training can be found in: `config/config.yaml`. I personally prefer to use files instead of long strings of arguments when configuring a run. Every entry in the config file can be overrided with the corresponding flag (e.g. `--max_epochs 500` if you would like to train with 500 epochs).\n\n## Logging and TensorBoard\nTo view results in TensorBoard, run:\n```\ntensorboard --logdir ./runs\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspijkervet%2Fclmr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspijkervet%2Fclmr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspijkervet%2Fclmr/lists"}