{"id":17594804,"url":"https://github.com/huggingface/finetrainers","last_synced_at":"2025-12-14T08:47:24.473Z","repository":{"id":258259661,"uuid":"862802466","full_name":"huggingface/finetrainers","owner":"huggingface","description":"Scalable and memory-optimized training of diffusion models","archived":false,"fork":false,"pushed_at":"2025-06-04T18:27:56.000Z","size":56851,"stargazers_count":1298,"open_issues_count":69,"forks_count":140,"subscribers_count":27,"default_branch":"main","last_synced_at":"2025-11-13T02:42:02.409Z","etag":null,"topics":["ai","art","artificial-intelligence","diffusers","diffusion","diffusion-models","pytorch","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huggingface.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-25T08:05:23.000Z","updated_at":"2025-11-12T10:02:22.000Z","dependencies_parsed_at":"2024-11-08T03:27:19.557Z","dependency_job_id":"7681d4c7-1e15-47dd-b416-390cc892fc8c","html_url":"https://github.com/huggingface/finetrainers","commit_stats":{"total_commits":36,"total_committers":10,"mean_commits":3.6,"dds":0.5277777777777778,"last_synced_commit":"d63a826f37758eccf226710f94f6c3a4d4ee7a25"},"previous_names":["a-r-r-o-w/cogvideox-factory","a-r-r-o-w/finetrainers","huggingface/finetrainers"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/huggingface/finetrainers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Ffinetrainers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Ffinetrainers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Ffinetrainers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Ffinetrainers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huggingface","download_url":"https://codeload.github.com/huggingface/finetrainers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Ffinetrainers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27723701,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-14T02:00:11.348Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","art","artificial-intelligence","diffusers","diffusion","diffusion-models","pytorch","transformers"],"created_at":"2024-10-22T07:08:42.958Z","updated_at":"2025-12-14T08:47:24.451Z","avatar_url":"https://github.com/huggingface.png","language":"Python","funding_links":[],"categories":["Python","11. Specialized Domains"],"sub_categories":[],"readme":"# finetrainers 🧪\n\nFinetrainers is a work-in-progress library to support (accessible) training of diffusion models and various commonly used training algorithms.\n\n\u003ctable align=\"center\"\u003e\n\u003ctr\u003e\n  \u003ctd align=\"center\"\u003e\u003cvideo src=\"https://github.com/user-attachments/assets/aad07161-87cb-4784-9e6b-16d06581e3e5\"\u003eYour browser does not support the video tag.\u003c/video\u003e\u003c/td\u003e\n  \u003ctd align=\"center\"\u003e\u003cvideo src=\"https://github.com/user-attachments/assets/c23d53e2-b422-4084-9156-3fce9fd01dad\"\u003eYour browser does not support the video tag.\u003c/video\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003cth align=\"center\"\u003eCogVideoX LoRA training as the first iteration of this project\u003c/th\u003e\n  \u003cth align=\"center\"\u003eReplication of PikaEffects\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## Table of Contents\n\n- [Quickstart](#quickstart)\n- [Features](#features)\n- [News](#news)\n- [Support Matrix](#support-matrix)\n- [Featured Projects](#featured-projects-)\n- [Acknowledgements](#acknowledgements)\n\n## Quickstart\n\nClone the repository and make sure the requirements are installed: `pip install -r requirements.txt` and install `diffusers` from source by `pip install git+https://github.com/huggingface/diffusers`. The requirements specify `diffusers\u003e=0.32.1`, but it is always recommended to use the `main` branch of Diffusers for the latest features and bugfixes. Note that the `main` branch for `finetrainers` is also the development branch, and stable support should be expected from the release tags.\n\nCheckout to the latest stable release tag:\n\n```bash\ngit fetch --all --tags\ngit checkout tags/v0.2.0\n```\n\nFollow the instructions mentioned in the [README](https://github.com/a-r-r-o-w/finetrainers/tree/v0.2.0-release) for the latest stable release.\n\n#### Using the main branch\n\nTo get started quickly with example training scripts on the main development branch, refer to the following:\n- [LTX-Video Pika Effects Crush](./examples/training/sft/ltx_video/crush_smol_lora/)\n- [CogVideoX Pika Effects Crush](./examples/training/sft/cogvideox/crush_smol_lora/)\n- [Wan T2V Pika Effects Crush](./examples/training/sft/wan/crush_smol_lora/)\n\nThe following are some simple datasets/HF orgs with good datasets to test training with quickly:\n- [Disney Video Generation Dataset](https://huggingface.co/datasets/Wild-Heart/Disney-VideoGeneration-Dataset)\n- [bigdatapw Video Dataset Collection](https://huggingface.co/bigdata-pw)\n- [Finetrainers HF Dataset Collection](https://huggingface.co/finetrainers)\n\nPlease checkout [`docs/models`](./docs/models/) and [`examples/training`](./examples/training/) to learn more about supported models for training \u0026 example reproducible training launch scripts. For a full list of arguments that can be set for training, refer to [`docs/args`](./docs/args.md).\n\n\u003e [!IMPORTANT] \n\u003e It is recommended to use Pytorch 2.5.1 or above for training. Previous versions can lead to completely black videos, OOM errors, or other issues and are not tested. For fully reproducible training, please use the same environment as mentioned in [environment.md](./docs/environment.md).\n\n## Features\n\n- DDP, FSDP-2 \u0026 HSDP, CP support\n- LoRA and full-rank finetuning; Conditional Control training\n- Memory-efficient single-GPU training\n- Multiple attention backends supported - `flash`, `flex`, `sage`, `xformers` (see [attention](./docs/models/attention.md) docs)\n- Auto-detection of commonly used dataset formats\n- Combined image/video datasets, multiple chainable local/remote datasets, multi-resolution bucketing \u0026 more\n- Memory-efficient precomputation support with/without on-the-fly precomputation for large scale datasets\n- Standardized model specification format for training arbitrary models\n- Fake FP8 training (QAT upcoming!)\n\n## News\n\n- 🔥 **2025-04-25**: Support for different attention providers added!\n- 🔥 **2025-04-21**: Wan I2V supported added!\n- 🔥 **2025-04-12**: Channel-concatenated control conditioning support added for CogView4 and Wan!\n- 🔥 **2025-04-08**: `torch.compile` support added!\n- 🔥 **2025-04-06**: Flux support added!\n- 🔥 **2025-03-07**: CogView4 support added!\n- 🔥 **2025-03-03**: Wan T2V support added!\n- 🔥 **2025-03-03**: We have shipped a complete refactor to support multi-backend distributed training, better precomputation handling for big datasets, model specification format (externally usable for training custom models), FSDP \u0026 more.\n- 🔥 **2025-02-12**: We have shipped a set of tooling to curate small and high-quality video datasets for fine-tuning. See [video-dataset-scripts](https://github.com/huggingface/video-dataset-scripts) documentation page for details!\n- 🔥 **2025-02-12**: Check out [eisneim/ltx_lora_training_i2v_t2v](https://github.com/eisneim/ltx_lora_training_i2v_t2v/)! It builds off of `finetrainers` to support image to video training for LTX-Video and STG guidance for inference.\n- 🔥 **2025-01-15**: Support for naive FP8 weight-casting training added! This allows training HunyuanVideo in under 24 GB upto specific resolutions.\n- 🔥 **2025-01-13**: Support for T2V full-finetuning added! Thanks to [@ArEnSc](https://github.com/ArEnSc) for taking up the initiative!\n- 🔥 **2025-01-03**: Support for T2V LoRA finetuning of [CogVideoX](https://huggingface.co/docs/diffusers/main/api/pipelines/cogvideox) added!\n- 🔥 **2024-12-20**: Support for T2V LoRA finetuning of [Hunyuan Video](https://huggingface.co/docs/diffusers/main/api/pipelines/hunyuan_video) added! We would like to thank @SHYuanBest for his work on a training script [here](https://github.com/huggingface/diffusers/pull/10254).\n- 🔥 **2024-12-18**: Support for T2V LoRA finetuning of [LTX Video](https://huggingface.co/docs/diffusers/main/api/pipelines/ltx_video) added!\n\n## Support Matrix\n\nThe following trainers are currently supported:\n\n- [SFT Trainer](./docs/trainer/sft_trainer.md)\n- [Control Trainer](./docs/trainer/control_trainer.md)\n\n\u003e [!NOTE]\n\u003e The following numbers were obtained from the [release branch](https://github.com/a-r-r-o-w/finetrainers/tree/v0.0.1). The `main` branch is unstable at the moment and may use higher memory.\n\n\u003cdiv align=\"center\"\u003e\n\n| **Model Name**                                 | **Tasks**     | **Min. LoRA VRAM\u003csup\u003e*\u003c/sup\u003e**     | **Min. Full Finetuning VRAM\u003csup\u003e^\u003c/sup\u003e**     |\n|:----------------------------------------------:|:-------------:|:----------------------------------:|:---------------------------------------------:|\n| [LTX-Video](./docs/models/ltx_video.md)        | Text-to-Video | 5 GB                               | 21 GB                                         |\n| [HunyuanVideo](./docs/models/hunyuan_video.md) | Text-to-Video | 32 GB                              | OOM                                           |\n| [CogVideoX-5b](./docs/models/cogvideox.md)     | Text-to-Video | 18 GB                              | 53 GB                                         |\n| [Wan](./docs/models/wan.md)                    | Text-to-Video | TODO                               | TODO                                          |\n| [CogView4](./docs/models/cogview4.md)          | Text-to-Image | TODO                               | TODO                                          |\n| [Flux](./docs/models/flux.md)                  | Text-to-Image | TODO                               | TODO                                          |\n\n\u003c/div\u003e\n\n\u003csub\u003e\u003csup\u003e*\u003c/sup\u003eNoted for training-only, no validation, at resolution `49x512x768`, rank 128, with pre-computation, using **FP8** weights \u0026 gradient checkpointing. Pre-computation of conditions and latents may require higher limits (but typically under 16 GB).\u003c/sub\u003e\u003cbr/\u003e\n\u003csub\u003e\u003csup\u003e^\u003c/sup\u003eNoted for training-only, no validation, at resolution `49x512x768`, with pre-computation, using **BF16** weights \u0026 gradient checkpointing.\u003c/sub\u003e\n\nIf you would like to use a custom dataset, refer to the dataset preparation guide [here](./docs/dataset/README.md).\n\n## Featured Projects 🔥\n\nCheckout some amazing projects citing `finetrainers`:\n- [Diffusion as Shader](https://github.com/IGL-HKUST/DiffusionAsShader)\n- [SkyworkAI's SkyReels-A1](https://github.com/SkyworkAI/SkyReels-A1) \u0026 [SkyReels-A2](https://github.com/SkyworkAI/SkyReels-A2)\n- [Aether](https://github.com/OpenRobotLab/Aether)\n- [MagicMotion](https://github.com/quanhaol/MagicMotion)\n- [eisneim's LTX Image-to-Video](https://github.com/eisneim/ltx_lora_training_i2v_t2v/)\n- [wileewang's TransPixar](https://github.com/wileewang/TransPixar)\n- [Feizc's Video-In-Context](https://github.com/feizc/Video-In-Context)\n\nCheckout the following UIs built for `finetrainers`:\n- [jbilcke's VideoModelStudio](https://github.com/jbilcke-hf/VideoModelStudio)\n- [neph1's finetrainers-ui](https://github.com/neph1/finetrainers-ui)\n\n## Acknowledgements\n\n* `finetrainers` builds on top of \u0026 takes inspiration from great open-source libraries - `transformers`, `accelerate`, `torchtune`, `torchtitan`, `peft`, `diffusers`, `bitsandbytes`, `torchao` and `deepspeed` - to name a few.\n* Some of the design choices of `finetrainers` were inspired by [`SimpleTuner`](https://github.com/bghira/SimpleTuner).\n`","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Ffinetrainers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuggingface%2Ffinetrainers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Ffinetrainers/lists"}