{"id":19222931,"url":"https://github.com/kyflores/scorelm","last_synced_at":"2026-06-24T09:31:44.121Z","repository":{"id":197367778,"uuid":"670020102","full_name":"kyflores/ScoreLM","owner":"kyflores","description":"Analyze musical scores with language models!","archived":false,"fork":false,"pushed_at":"2023-11-16T23:35:01.000Z","size":72,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"develop","last_synced_at":"2025-02-23T09:44:18.094Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kyflores.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-24T05:53:40.000Z","updated_at":"2023-09-28T03:31:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"cd31c989-90fa-47ee-b965-b67bc1924c8f","html_url":"https://github.com/kyflores/ScoreLM","commit_stats":null,"previous_names":["kyflores/scorelm"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kyflores/ScoreLM","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyflores%2FScoreLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyflores%2FScoreLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyflores%2FScoreLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyflores%2FScoreLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kyflores","download_url":"https://codeload.github.com/kyflores/ScoreLM/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyflores%2FScoreLM/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34724746,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-24T02:00:07.484Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T15:05:54.992Z","updated_at":"2026-06-24T09:31:44.098Z","avatar_url":"https://github.com/kyflores.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ScoreLM\nAnalyze musical scores with language models!\n\n## Setup\nFirst, create a new virtualenv or conda environment with python \u0026 pip.\nThen:\n```\npip install transformers datasets accelerate music21 deepspeed\n```\n\nFinally, note that the scripts generate some very large files, like datasets or model\nweights. **Please DO NOT commit these to git!** The default names for these files are in the gitignore,\nbut please be careful.\n\n## Data generation\nRun `generate_data.py` to process scores from composers in music21's database into the model's language.\nThis script produces `dataset/\u003ccomposername\u003e.jsonl` for each composer. To create a training dataset, use\n`cat` to merge the composers you want into one file called `data.jsonl`.\n```\ncat bach.jsonl mozart.jsonl \u003e data.jsonl\n```\nFilenames can be repeated to include them multiple times. This can be used to influence the composition\nof the dataset, or balance it. For instance `palestrina.jsonl` is the largest by a wide margin, so other\ncomposers might need to be repeated to get more diverse and interesting generations.\n\n## Training\nUse `train.py` to train the model. This script finetunes Eleuther's Pythia model for the score text\nformat. Right now we use Pythia for a few reasons:\n* Easy to access from Hugging Face, no downloading delta weights and merging like the official llama.\n* Offers many scaling options for different hardware configurations. 70m is very manageable and can\n  be trained on average hardware without `deepspeed`\n* Pythia is trained on [The Pile](https://arxiv.org/pdf/2101.00027.pdf), which contains code text. Pretraining\n  on code (or other formal languages like mathematics) is useful if the score text format resembles code.\n\nCheck out one of the [pythia model cards](https://huggingface.co/EleutherAI/pythia-12b-v0) for more info.\n\nParameters are selected in `train_cfg.json`.\nIn particular, check out these parameters:\n```\nmodel_name:\n    Set by default to EleutherAI/pythia-70m-deduped, the smallest Pythia variant.\n    If you have enough VRAM, you can select 160m, 410m, etc.\nmax_length:\n    The sequence length the model can process, in tokens. Reduce if you're running out of VRAM.\nbatchsize:\n    How many items to process in a batch. Use the largest batchsize your VRAM allows. Reduce if\n    out of VRAM. When reducing batchsize, you may also need to reduce lr.\n```\nTODO:\n* Support checkpointing and checkpoint loading.\n\n### Deepspeed\nDeepspeed implements optimizations for training that can significantly reduce the VRAM\nrequirement, at the cost of additional main RAM use. To use deepspeed,\n```\naccelerate launch --config_file accel_config.yaml train.py\n```\nIf you receive an error about CPU Adam like this: `AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'`\nThen try reinstalling deepspeed with these arguments: `DS_BUILD_CPU_ADAM=1  BUILD_UTILS=1  pip install deepspeed -U `\nI had this problem on ROCm setups but not CUDA.\n\n## Inference\nUse `infer.py` to output text with a trained model. This scripts loads the model\nin a directory named `score-lm`, which is created by the training script when it finishes.\n```\npython infer.py --help\npython infer.py -p \"|\"\n```\nThe `-p` argument is the text to start generating from. Since a `|` is the start of a new measure,\nit is the usual way to start generation of a new score. You could also manually enter your own score,\nand let it continue from that.\n\n`temperature \u0026 topk` are the key parameters to modify during training. Both will result\nin more randomness when set higher. At low temperatures (\u003c0.5) you'll likely get the same\nchord repeated over and over, and at higher temperatures you'll get scores that Nancarrow would love.\nTODO:\n* expose these on the CLI\n\n## Reconstructing the score.\n`infer.py` will take the generated output from the model and try to reconstruct a musical score\nout of it. Naturally, the model output is imperfect so the decoder will ignore and fix certain\nerrors such as:\n* Invalid note names (like H) or octaves (like -6) are ignored.\n  This is done on a note by note basis within a chord, so a single bad note doesn't drop the entire chord.\n* Invalid durations (anything too small for musescore to display) are changed to 0.5.\n* Nothing is done to duplicates, but it's probably worth fixing. Musescore throws a warning,\n  but renders them anyway.\n* We simplify ties to be only `start` or `None`, ignoring the `continue` or `stop` codes in music21.\n* Currently, we do not follow the measure tokens generated by the model, and instead append\n  everything into one long stream and let music21 figure out where the measure boundaries should be.\n  * It's a future goal to benchmark the model's ability to generate notes that properly sum to  a measure.\n* Non-power-of-two duration (like triplets) handling seems bugged and needs more investigation, but\n  there aren't many triplets in the training data at the moment. They render but Musecore produces\n  warnings.\n\n## WSL2\nDeepspeed hits an out of memory issue on WSL2, which is discussed in [this thread](https://github.com/microsoft/DeepSpeed/issues/2977)\nApply the workaround suggest, removing the pin memory call.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyflores%2Fscorelm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyflores%2Fscorelm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyflores%2Fscorelm/lists"}