{"id":19805454,"url":"https://github.com/argmaxinc/diffusionkit","last_synced_at":"2025-05-14T18:07:04.625Z","repository":{"id":244091837,"uuid":"810517562","full_name":"argmaxinc/DiffusionKit","owner":"argmaxinc","description":"On-device Image Generation for Apple Silicon","archived":false,"fork":false,"pushed_at":"2025-04-11T00:31:24.000Z","size":1136,"stargazers_count":612,"open_issues_count":11,"forks_count":35,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-04-29T11:41:04.652Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://argmaxinc.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/argmaxinc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-06-04T21:12:40.000Z","updated_at":"2025-04-29T04:11:33.000Z","dependencies_parsed_at":"2024-08-27T08:30:37.253Z","dependency_job_id":"8db528e0-f851-446a-8f1b-5b436234ae92","html_url":"https://github.com/argmaxinc/DiffusionKit","commit_stats":null,"previous_names":["argmaxinc/diffusionkit"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argmaxinc%2FDiffusionKit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argmaxinc%2FDiffusionKit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argmaxinc%2FDiffusionKit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argmaxinc%2FDiffusionKit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/argmaxinc","download_url":"https://codeload.github.com/argmaxinc/DiffusionKit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254198515,"owners_count":22030966,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T09:04:01.675Z","updated_at":"2025-05-14T18:06:59.614Z","avatar_url":"https://github.com/argmaxinc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DiffusionKit\n\n[![Latest Python Version](https://img.shields.io/pypi/v/diffusionkit)](https://pypi.org/project/diffusionkit)\n\nRun Diffusion Models on Apple Silicon with Core ML and MLX\n\nThis repository comprises\n- `diffusionkit`, a Python package for converting PyTorch models to Core ML format and performing image generation with [MLX](https://github.com/ml-explore/mlx) in Python\n- `DiffusionKit`, a Swift package for on-device inference of diffusion models using Core ML and MLX\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"assets/diffusionkit.png\" width=256\u003e\n\u003c/div\u003e\n\n\n## Installation\n\nThe following installation steps are required for:\n- MLX inference\n- PyTorch to Core ML model conversion\n\n### Python Environment Setup\n\n```bash\nconda create -n diffusionkit python=3.11 -y\nconda activate diffusionkit\ncd /path/to/diffusionkit/repo\npip install -e .\n```\n\n### Hugging Face Hub Credentials\n\n\u003cdetails\u003e\n  \u003csummary\u003e Click to expand \u003c/summary\u003e\n\n\n[Stable Diffusion 3](https://huggingface.co/stabilityai/stable-diffusion-3-medium) requires users to accept the terms before downloading the checkpoint.\n\n[FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) also requires users to accept the terms before downloading the checkpoint.\n\nOnce you accept the terms, sign in with your Hugging Face hub READ token as below:\n\u003e [!IMPORTANT]\n\u003e If using a fine-grained token, it is also necessary to [edit permissions](https://huggingface.co/settings/tokens) to allow `Read access to contents of all public gated repos you can access`\n\n```bash\nhuggingface-cli login --token YOUR_HF_HUB_TOKEN\n```\n\n\u003c/details\u003e\n\n\n## \u003ca name=\"converting-models-to-coreml\"\u003e\u003c/a\u003e Converting Models from PyTorch to Core ML\n\n\u003cdetails\u003e\n  \u003csummary\u003e Click to expand \u003c/summary\u003e\n\n**Step 1:** Follow the installation steps from the previous section\n\n**Step 2:** Verify you've accepted the [StabilityAI license terms](https://huggingface.co/stabilityai/stable-diffusion-3-medium) and have allowed gated access on your [HuggingFace token](https://huggingface.co/settings/tokens)\n\n**Step 3:** Prepare the denoise model (MMDiT) Core ML model files (`.mlpackage`)\n\n```shell\npython -m python.src.diffusionkit.tests.torch2coreml.test_mmdit --sd3-ckpt-path stabilityai/stable-diffusion-3-medium --model-version 2b -o \u003coutput-mlpackages-directory\u003e --latent-size {64, 128}\n```\n\n**Step 4:** Prepare the VAE Decoder Core ML model files (`.mlpackage`)\n\n```shell\npython -m python.src.diffusionkit.tests.torch2coreml.test_vae --sd3-ckpt-path stabilityai/stable-diffusion-3-medium -o \u003coutput-mlpackages-directory\u003e --latent-size {64, 128}\n```\n\nNote:\n- `--sd3-ckpt-path` can be a path any HuggingFace repo (e.g. `stabilityai/stable-diffusion-3-medium`) OR a path to a local `sd3_medium.safetensors` file\n\u003c/details\u003e\n\n## \u003ca name=\"image-generation-with-python-mlx\"\u003e\u003c/a\u003e Image Generation with Python MLX\n\n\u003cdetails\u003e\n  \u003csummary\u003e Click to expand \u003c/summary\u003e\n\n### CLI ###\n\nMost simple:\n```shell\ndiffusionkit-cli --prompt \"a photo of a cat\" --output-path \u003c/path/to/output/image.png\u003e\n```\n\nSome notable optional arguments for:\n- Reproduciblity of results, use `--seed`\n- image-to-image, use `--image-path` (path to input image) and `--denoise` (value between 0. and 1.)\n- Enabling T5 encoder in SD3, use `--t5` (FLUX must use T5 regardless of this argument)\n- Different resolutions, use `--height` and `--width`\n- Using a local checkpoint, use `--local-ckpt \u003c/path/to/ckpt.safetensors\u003e` (e.g. `~/models/stable-diffusion-3-medium/sd3_medium.safetensors`).\n\nPlease refer to the help menu for all available arguments: `diffusionkit-cli -h`.\n\nNote: When using `FLUX.1-dev`, verify you've accepted the [FLUX.1-dev licence](https://huggingface.co/black-forest-labs/FLUX.1-dev) and have allowed gated access on your [HuggingFace token](https://huggingface.co/settings/tokens)\n\n### Code ###\n\nFor Stable Diffusion 3:\n```python\nfrom diffusionkit.mlx import DiffusionPipeline\npipeline = DiffusionPipeline(\n  shift=3.0,\n  use_t5=False,\n  model_version=\"argmaxinc/mlx-stable-diffusion-3-medium\",\n  low_memory_mode=True,\n  a16=True,\n  w16=True,\n)\n```\n\nFor FLUX:\n```python\nfrom diffusionkit.mlx import FluxPipeline\npipeline = FluxPipeline(\n  shift=1.0,\n  model_version=\"argmaxinc/mlx-FLUX.1-schnell\", # model_version=\"argmaxinc/mlx-FLUX.1-dev\" for FLUX.1-dev\n  low_memory_mode=True,\n  a16=True,\n  w16=True,\n)\n```\n\nFinally, to generate the image, use the `generate_image()` function:\n```python\nHEIGHT = 512\nWIDTH = 512\nNUM_STEPS = 4  #  4 for FLUX.1-schnell, 50 for SD3 and FLUX.1-dev\nCFG_WEIGHT = 0. # for FLUX.1-schnell, 5. for SD3\n\nimage, _ = pipeline.generate_image(\n  \"a photo of a cat\",\n  cfg_weight=CFG_WEIGHT,\n  num_steps=NUM_STEPS,\n  latent_size=(HEIGHT // 8, WIDTH // 8),\n)\n```\nSome notable optional arguments:\n- For image-to-image, use `image_path` (path to input image) and `denoise` (value between 0. and 1.) input variables.\n- For seed, use `seed` input variable.\n- For negative prompt, use `negative_text` input variable.\n\nThe generated `image` can be saved with:\n```python\nimage.save(\"path/to/save.png\")\n```\n\n\u003c/details\u003e\n\n## Image Generation with Swift\n\n\u003cdetails\u003e\n  \u003csummary\u003e Click to expand \u003c/summary\u003e\n\n### Core ML Swift\n\n[Apple Core ML Stable Diffusion](https://github.com/apple/ml-stable-diffusion) is the initial Core ML backend for DiffusionKit. Stable Diffusion 3 support is upstreamed to that repository while we build the holistic Swift inference package.\n\n### MLX Swift\n🚧\n\n\u003c/details\u003e\n\n## License\n\nDiffusionKit is released under the MIT License. See [LICENSE](LICENSE) for more details.\n\n## Citation\n\nIf you use DiffusionKit for something cool or just find it useful, please drop us a note at [info@takeargmax.com](mailto:info@takeargmax.com)!\n\nIf you use DiffusionKit for academic work, here is the BibTeX:\n\n```bibtex\n@misc{diffusionkit-argmax,\n   title = {DiffusionKit},\n   author = {Argmax, Inc.},\n   year = {2024},\n   URL = {https://github.com/argmaxinc/DiffusionKit}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fargmaxinc%2Fdiffusionkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fargmaxinc%2Fdiffusionkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fargmaxinc%2Fdiffusionkit/lists"}