{"id":27146586,"url":"https://github.com/euclaise/slimtrainer","last_synced_at":"2025-08-22T01:11:29.080Z","repository":{"id":197630755,"uuid":"699025924","full_name":"euclaise/SlimTrainer","owner":"euclaise","description":"Full finetuning of large language models without large memory requirements","archived":false,"fork":false,"pushed_at":"2024-01-01T07:49:06.000Z","size":87,"stargazers_count":93,"open_issues_count":2,"forks_count":3,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-08T09:47:43.958Z","etag":null,"topics":["language-modeling","llama","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"0bsd","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/euclaise.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-01T17:46:25.000Z","updated_at":"2025-02-14T15:53:30.000Z","dependencies_parsed_at":"2023-12-01T20:23:08.913Z","dependency_job_id":"f2d52ded-5746-4a3e-96bd-478cf67538f9","html_url":"https://github.com/euclaise/SlimTrainer","commit_stats":{"total_commits":205,"total_committers":1,"mean_commits":205.0,"dds":0.0,"last_synced_commit":"22f9566e751a2cd00cc88bf9b4233122d27d259a"},"previous_names":["euclaise/slimtrainer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/euclaise%2FSlimTrainer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/euclaise%2FSlimTrainer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/euclaise%2FSlimTrainer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/euclaise%2FSlimTrainer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/euclaise","download_url":"https://codeload.github.com/euclaise/SlimTrainer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247828793,"owners_count":21002975,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["language-modeling","llama","transformers"],"created_at":"2025-04-08T10:48:08.224Z","updated_at":"2025-04-08T10:48:08.781Z","avatar_url":"https://github.com/euclaise.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"This repository is currently in an early, expiremental stage, and should not be considered production-ready.\n\nSlimTrainer and Adalite allow for full parameter 16-bit finetuning of language models up to 7B on a single 24GB GPU.\n\nThe optimizer uses the backpropagation fusing technique from [LOMO](https://github.com/OpenLMLab/LOMO), but uses a custom optimizer instead of using simple SGD.\n\nThe small batch size and extreme memory requirements extensive exploration of potential optimizer variants, resulting in a custom optimizer, Adalite, based on Adafactor and LAMB.\n\nFurther development is being pursued, including quantization of embedding and optimizer states, which should allow for larger batch sizes.\n\nNote that long sequence lengths require more memory. A sequence length of 512 at a batch size of 1, with flash attention and Adalite, for LLaMA 7B, just fits at a batch size of 1 (and a batch size of 2 is so close that I suspect it could be achieved with some efficiency tweaks). StableLM 3B fits at a batch size of 4.\n\nAn implementation of [Sigma reparameterization](https://proceedings.mlr.press/v202/zhai23a/zhai23a.pdf) and a naive version of embedding quantization are also present in this repository. Apple found that sigma parameterization was sufficient to allow them to pretrain a transformer with SGD+momentum. Unfortunately, momentum alone takes too much memory, and I have not seen any improvement for finetuning using sigma reparameterization with non-momentum SGD.\n\nThe techniques used in this repository should also help with training in less constrained settings, but I haven't tested it.  In such contexts, with a sufficiently large batch size, the `OverlapLion` optimizer may be useable.\n\n\n## Example code\n\n```\n# ...\n# Assume the following variables exist:\n# - ds: Dataset\n# - TOTAL_STEPS: step count\n# - model: Some autoregressive huggingface model\n# - dc: Some data collator\n\nfrom SlimTrainer import (\n    SlimTrainer,\n    Adalite,\n    CosineDecayWithWarmup,\n    SlimTrainer\n)\n\noptim = Adalite()\nscheduler = CosineDecayWithWarmup(optim, total_steps=TOTAL_STEPS, warmup_steps=TOTAL_STEPS*0.2, max_lr=2e-5)\n\ntrainer = SlimTrainer(\n    model,\n    optim,\n    scheduler,\n    train_data=ds,\n    epochs=1,\n    batch_size=1,\n    data_collator=dc\n)\n\ntrainer.train()\n\nmodel.save_pretrained(\"result\")\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feuclaise%2Fslimtrainer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feuclaise%2Fslimtrainer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feuclaise%2Fslimtrainer/lists"}