{"id":14135515,"url":"https://github.com/dvlab-research/ControlNeXt","last_synced_at":"2025-08-04T17:31:47.045Z","repository":{"id":247011466,"uuid":"814423356","full_name":"dvlab-research/ControlNeXt","owner":"dvlab-research","description":"Controllable video and image Generation, SVD, Animate Anyone, ControlNet, ControlNeXt, LoRA","archived":false,"fork":false,"pushed_at":"2024-09-25T07:45:41.000Z","size":36024,"stargazers_count":1418,"open_issues_count":43,"forks_count":70,"subscribers_count":26,"default_branch":"main","last_synced_at":"2024-11-28T20:04:20.754Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dvlab-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-13T01:49:51.000Z","updated_at":"2024-11-28T16:10:58.000Z","dependencies_parsed_at":"2024-07-12T13:47:22.884Z","dependency_job_id":"2fe38244-1a87-405e-8aec-219261fb91aa","html_url":"https://github.com/dvlab-research/ControlNeXt","commit_stats":null,"previous_names":["dvlab-research/controlnext"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FControlNeXt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FControlNeXt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FControlNeXt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FControlNeXt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dvlab-research","download_url":"https://codeload.github.com/dvlab-research/ControlNeXt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228667477,"owners_count":17954184,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-16T08:02:13.955Z","updated_at":"2024-12-07T22:31:08.952Z","avatar_url":"https://github.com/dvlab-research.png","language":"Python","funding_links":[],"categories":["Additional conditions","Python","\u003cspan id=\"head1\"\u003e *Keywords* \u003c/span\u003e"],"sub_categories":["[Addictive PEFT](#content)"],"readme":"\n# 🌀 ControlNeXt\n\n\n\n##   [📝 Project Page](https://pbihao.github.io/projects/controlnext/index.html)  |  [📚 Paper](https://arxiv.org/abs/2408.06070) | [🗂️ Demo (SDXL)](https://huggingface.co/spaces/Eugeoter/ControlNeXt)\n\n\n**ControlNeXt** is our official implementation for controllable generation, supporting both images and videos while incorporating diverse forms of control information. In this project, we propose a new method that reduces trainable parameters by up to 90% compared with ControlNet, achieving faster convergence and outstanding efficiency. This method can be directly combined with other LoRA techniques to alter style and ensure more stable generation. Please refer to the examples for more details.\n\nWe provide an online demo of [ControlNeXt-SDXL](./ControlNeXt-SDXL/). Due to the high resource requirements of SVD, we are unable to offer it online.\n\n\u003e This project is still undergoing iterative development. The code and model may be updated at any time. More information will be provided later.\n\n# Experiences\nWe share more training experiences [there](./experiences.md) and in the [Issue](https://github.com/dvlab-research/ControlNeXt/issues/14#issuecomment-2290450333).\nWe spent a lot of time to find these. Now share with all of you. May these will help you!\n\n# Model Zoo\n\n- **ControlNeXt-SDXL** [ [Link](ControlNeXt-SDXL) ] : Controllable image generation. Our model is built upon [Stable Diffusion XL ](stabilityai/stable-diffusion-xl-base-1.0). Fewer trainable parameters, faster convergence, improved efficiency, and can be integrated with LoRA.\n\n- **ControlNeXt-SDXL-Training** [ [Link](ControlNeXt-SDXL-Training) ] :  The training scripts for our `ControlNeXt-SDXL` [ [Link](ControlNeXt-SDXL) ].\n\n- **ControlNeXt-SVD-v2** [ [Link](ControlNeXt-SVD-v2) ] :  Generate the video controlled by the sequence of human poses. In the v2 version, we implement several improvements: a higher-quality collected training dataset, larger training and inference batch frames, higher generation resolution, enhanced human-related video generation through continual training, and pose alignment for inference to improve overall performance.\n\n- **ControlNeXt-SVD-v2-Training** [ [Link](ControlNeXt-SVD-v2-Training) ] :  The training scripts for our `ControlNeXt-SVD-v2` [ [Link](ControlNeXt-SVD-v2) ].\n\n- **ControlNeXt-SVD** [ [Link](ControlNeXt-SVD) ] :  Generate the video controlled by the sequence of human poses. This can be seen as an attempt to replicate the implementation of [AnimateAnyone](https://github.com/HumanAIGC/AnimateAnyone). However, our model is built upon [Stable Video Diffusion](https://stability.ai/stable-video), employing a more concise architecture.\n\n- **ControlNeXt-SD1.5** [ [Link](ControlNeXt-SD1.5) ] : Controllable image generation. Our model is built upon [Stable Diffusion 1.5](https://huggingface.co/runwayml/stable-diffusion-v1-5). Fewer trainable parameters, faster convergence, improved efficiency, and can be integrated with LoRA.\n\n- **ControlNeXt-SD1.5-Training** [ [Link](ControlNeXt-SD1.5-Training) ] : The training scripts for our `ControlNeXt-SD1.5` [ [Link](ControlNeXt-SD1.5) ].\n\n- **ControlNeXt-SD3** [ [Link](ControlNeXt-SD3) ] :  We are regret to inform that ControlNeXt-SD3 is trained with protected and private data and code, and therefore cannot be released. \n\n\n\n# 🎥 Examples\n### For more examples, please refer to our [Project page](https://pbihao.github.io/projects/controlnext/index.html).\n\n### [ControlNeXt-SDXL](ControlNeXt-SDXL)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"ControlNeXt-SDXL/examples/demo/demo1.jpg\" width=\"80%\" alt=\"demo1\"\u003e\n  \u003cimg src=\"ControlNeXt-SDXL/examples/demo/demo2.jpg\" width=\"80%\" alt=\"demo2\"\u003e\n  \u003cimg src=\"ControlNeXt-SDXL/examples/demo/demo3.jpg\" width=\"80%\" alt=\"demo3\"\u003e\n  \u003cimg src=\"ControlNeXt-SDXL/examples/demo/demo5.jpg\" width=\"80%\" alt=\"demo5\"\u003e\n\u003c/p\u003e\n\n### [ControlNeXt-SVD-v2](ControlNeXt-SVD-v2)\nIf you can't load the videos, you can also directly download them from [here](examples/demos) and [here](examples/video).\nOr you can view them from our [Project Page](https://pbihao.github.io/projects/controlnext/index.html) or [BiliBili](https://www.bilibili.com/video/BV1wJYbebEE7/?buvid=YC4E03C93B119ADD4080B0958DE73F9DDCAC\u0026from_spmid=dt.dt.video.0\u0026is_story_h5=false\u0026mid=y82Gz7uArS6jTQ6zuqJj3w%3D%3D\u0026p=1\u0026plat_id=114\u0026share_from=ugc\u0026share_medium=iphone\u0026share_plat=ios\u0026share_session_id=4E5549FC-0710-4030-BD2C-CDED80B46D08\u0026share_source=WEIXIN\u0026share_source=weixin\u0026share_tag=s_i\u0026timestamp=1723123770\u0026unique_k=XLZLhCq\u0026up_id=176095810\u0026vd_source=3791450598e16da25ecc2477fc7983db).\n\n\u003ctable style=\"margin: 0 auto; border-collapse: collapse;\"\u003e\n    \u003ctr\u003e\n        \u003ctd width=\"40%\" style=\"border: none;\"\u003e\n            \u003cvideo width=\"100%\" height=\"auto\" style=\"display: block; margin: 0px auto;\" controls autoplay loop src=\"https://github.com/user-attachments/assets/9d45a00a-d3cd-48e1-aa78-1d3158bfd4f4\" muted=\"false\"\u003e\u003c/video\u003e\n        \u003c/td\u003e\n        \u003ctd width=\"40%\" style=\"border: none;\"\u003e\n            \u003cvideo width=\"100%\" height=\"auto\" style=\"display: block; margin: 0px auto;\" controls autoplay loop src=\"https://github.com/user-attachments/assets/1004960a-82de-4f0d-a329-ba676b8cbd0d\" muted=\"false\"\u003e\u003c/video\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd width=\"40%\" style=\"border: none;\"\u003e\n            \u003cvideo width=\"100%\" height=\"auto\" style=\"display: block; margin: 0px auto;\" controls autoplay loop src=\"https://github.com/user-attachments/assets/7db1acd1-0c61-4855-91bb-e4e8f8989393\" muted=\"false\"\u003e\u003c/video\u003e\n        \u003c/td\u003e\n        \u003ctd width=\"40%\" style=\"border: none;\"\u003e\n            \u003cvideo width=\"100%\" height=\"auto\" style=\"display: block; margin: 0px auto;\" controls autoplay loop src=\"https://github.com/user-attachments/assets/0f32df53-1827-404d-806a-23e65d357504\" muted=\"false\"\u003e\u003c/video\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n\n\u003c/table\u003e\n\n\u003cvideo width=\"80%\" height=\"auto\" style=\"display: block; margin: 0px auto;\" controls autoplay loop src=\"https://github.com/user-attachments/assets/c69b4f34-0851-4637-a9ef-fb91beed5666\" muted=\"false\"\u003e\u003c/video\u003e\n\n\u003cvideo width=\"80%\" height=\"auto\" style=\"display: block; margin: 0px auto;\" controls autoplay loop src=\"https://github.com/user-attachments/assets/32a4d24b-bc39-4ea9-9fd4-ed78b4eec116\" muted=\"false\"\u003e\u003c/video\u003e\n\n\n### [ControlNeXt-SVD](ControlNeXt-SVD)\nIf you can't load the videos, you can also directly download them from [here](ControlNeXt-SVD/outputs).\n\n\u003cvideo width=\"80%\" height='auto' style=\"display: block; margin: 20px auto;\" controls autoplay loop src=\"https://github.com/dvlab-research/ControlNeXt/assets/22709028/c0ec7591-2100-42d9-99dd-cc55c5fa006f\" muted=\"false\"\u003e\u003c/video\u003e\n\n\u003cvideo width=\"80%\" height='auto' style=\"display: block; margin: 20px auto;\" controls autoplay loop src=\"https://github.com/dvlab-research/ControlNeXt/assets/22709028/5da1ba44-cb92-49c2-95f4-05b8e02ed6eb\" muted=\"false\"\u003e\u003c/video\u003e\n\n\u003c!-- \u003cvideo width=\"40%\" height='auto' style=\"display: block; margin: 20px auto;\" controls autoplay loop src=\"https://github.com/dvlab-research/ControlNeXt/assets/22709028/bc819ca7-81f3-4d63-901a-a1a4b4afc395\" muted=\"false\"\u003e\u003c/video\u003e\n\n\u003cvideo width=\"40%\" height='auto' style=\"display: block; margin: 20px auto;\" controls autoplay loop src=\"https://github.com/dvlab-research/ControlNeXt/assets/22709028/adc01423-fcec-467e-a7ab-87a6e1ef5f62\" muted=\"false\"\u003e\u003c/video\u003e --\u003e\n\n\u003ctable\u003e\n\u003ctr\u003e\n    \u003ctd width=50% style=\"border: none\"\u003e\n        \u003cvideo width=\"80%\" height='auto' style=\"display: block; margin: 0px auto;\" controls autoplay loop src=\"https://github.com/dvlab-research/ControlNeXt/assets/22709028/bc819ca7-81f3-4d63-901a-a1a4b4afc395\" muted=\"false\"\u003e\u003c/video\u003e\n    \u003c/td\u003e\n    \u003ctd width=50% style=\"border: none\"\u003e\n        \u003cvideo width=\"80%\" height='auto' style=\"display: block; margin: 0px auto;\" controls autoplay loop src=\"https://github.com/dvlab-research/ControlNeXt/assets/22709028/adc01423-fcec-467e-a7ab-87a6e1ef5f62\" muted=\"false\"\u003e\u003c/video\u003e\n    \u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n\n\n### [ControlNeXt-SD1.5](ControlNeXt-SD1.5)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"ControlNeXt-SD1.5/examples/deepfashion_multiview/eval_img/DreamShaper.jpg\" width=\"90%\" alt=\"DreamShaper\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"ControlNeXt-SD1.5/examples/deepfashion_multiview/eval_img/Anythingv3_fischl.jpg\" width=\"90%\" alt=\"Anythingv3\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"ControlNeXt-SD1.5/examples/deepfashion_caption/eval_img/chinese_style.jpg\" width=\"90%\" alt=\"Anythingv3\"\u003e\n\u003c/p\u003e\n\n\n### If you find this work useful, please consider citing:\n```\n@article{peng2024controlnext,\n  title={ControlNeXt: Powerful and Efficient Control for Image and Video Generation},\n  author={Peng, Bohao and Wang, Jian and Zhang, Yuechen and Li, Wenbo and Yang, Ming-Chang and Jia, Jiaya},\n  journal={arXiv preprint arXiv:2408.06070},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdvlab-research%2FControlNeXt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdvlab-research%2FControlNeXt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdvlab-research%2FControlNeXt/lists"}