{"id":21990436,"url":"https://github.com/OpenSparseLLMs/Skip-DiT","last_synced_at":"2025-07-23T00:31:42.508Z","repository":{"id":264670953,"uuid":"893080979","full_name":"OpenSparseLLMs/Skip-DiT","owner":"OpenSparseLLMs","description":"✈️ [ICCV 2025] Towards Stabilized and Efficient Diffusion Transformers through Long-Skip-Connections with Spectral Constraints","archived":false,"fork":false,"pushed_at":"2025-07-10T10:34:55.000Z","size":39654,"stargazers_count":71,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-10T18:04:53.035Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2411.17616","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenSparseLLMs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-11-23T13:37:33.000Z","updated_at":"2025-07-10T10:35:00.000Z","dependencies_parsed_at":"2025-03-28T10:22:25.661Z","dependency_job_id":"2edb1379-4a72-484d-965d-e5e737b4d387","html_url":"https://github.com/OpenSparseLLMs/Skip-DiT","commit_stats":null,"previous_names":["opensparsellms/skip-dit"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/OpenSparseLLMs/Skip-DiT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenSparseLLMs%2FSkip-DiT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenSparseLLMs%2FSkip-DiT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenSparseLLMs%2FSkip-DiT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenSparseLLMs%2FSkip-DiT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenSparseLLMs","download_url":"https://codeload.github.com/OpenSparseLLMs/Skip-DiT/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenSparseLLMs%2FSkip-DiT/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266596689,"owners_count":23953891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-29T20:01:09.224Z","updated_at":"2025-07-23T00:31:37.451Z","avatar_url":"https://github.com/OpenSparseLLMs.png","language":"Python","funding_links":[],"categories":["Accelerate"],"sub_categories":[],"readme":"# Accelerating Vision Diffusion Transformers with Skip Branches\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://github.com/OpenSparseLLMs/Skip-DiT\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Skip-DiT-Code\u0026message=Github\u0026color=blue\u0026logo=github-pages\"\u003e\u003c/a\u003e \u0026ensp;\n  \u003ca href=\"https://arxiv.org/abs/2411.17616\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Paper\u0026message=Arxiv:Skip-DiT\u0026color=red\u0026logo=arxiv\"\u003e\u003c/a\u003e \u0026ensp;\n  \u003ca href=\"https://huggingface.co/GuanjieChen/Skip-DiT\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Skip-DiT\u0026message=HuggingFace\u0026color=yellow\"\u003e\u003c/a\u003e \u0026ensp;\n\u003c/div\u003e\n\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"visuals/video-demo.gif\" width=\"90%\" \u003e\u003c/img\u003e\n  \u003cbr\u003e\n  \u003cem\u003e\n      (Results of Latte with skip-branches on text-to-video and class-to-video tasks. Left: text-to-video with 1.7x and 2.0x speedup. Right: class-to-video with 2.2x and 2.5x speedup. Latency is measured on one A100.) \n  \u003c/em\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"visuals/image-demo.jpg\" width=\"100%\" \u003e\u003c/img\u003e\n  \u003cbr\u003e\n  \u003cem\u003e\n      (Results of HunYuan-DiT with skip-branches on text-to-image task. Latency is measured on one A100.) \n  \u003c/em\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\n\u003c!-- \u003e**Accelerating Vision Diffusion Transformers with Skip Branches**\n\u003e\n\u003e [Guanjie Chen](), [Xinyu Zhao](),[Yucheng Zhou](), [Tianlong Chen](), [Yu Cheng]()\n\u003e\n\u003e [Arxiv](), [Huggingface](https://huggingface.co/GuanjieChen/Skip-DiT/tree/main) --\u003e\n\n### About\nThis repository contains the official PyTorch implementation of the paper: **[Accelerating Vision Diffusion Transformers with Skip Branches](https://arxiv.org/abs/2411.17616)**. In this work, we enhance standard DiT models by introducing **Skip-DiT**, which incorporates skip branches to improve feature smoothness. We also propose **Skip-Cache**, a method that leverages skip branches to cache DiT features across timesteps during inference.The effectiveness of our approach is validated on various DiT backbones for both video and image generation, demonstrating how skip branches preserve generation quality while achieving significant speedup. Experimental results show that **Skip-Cache** provides a $1.5\\times$ speedup with minimal computational cost and a $2.2\\times$ speedup with only a slight reduction in quantitative metrics. All the codes and checkpoints are publicly available at [huggingface](https://huggingface.co/GuanjieChen/Skip-DiT/tree/main) and [github](https://github.com/OpenSparseLLMs/Skip-DiT.git). More visualizations can be found [here](#visualization).\n\n### Pipeline\n![pipeline](visuals/pipeline.jpg)\nIllustration of Skip-DiT and Skip-Cache for DiT visual generation caching. (a) The vanilla DiT block for image and video generation. (b) Skip-DiT modifies the vanilla DiT model using skip branches to connect shallow and deep DiT blocks. (c) Given a Skip-DiT with $L$ layers, during inference, at the $t-1$ step, the first layer output  ${x'}^{t-1}\\_{0}$ and cached $L-1$ layer output ${x'}^t_{L-1}$ are forwarded through the skip branches to the final DiT block to generate the denoising output, without executing DiT blocks 2 to $L-1$.\n\n### Feature Smoothness\n![feature](visuals/feature.jpg)\nFeature smoothness analysis of DiT in the class-to-video generation task using DDPM. Normalized disturbances, controlled by strength coefficients $\\alpha$ and $\\beta$, are introduced to the model with and without skip connections. We compare the similarity between the original and perturbed features. The feature difference surface of Latte, with and without skip connections, is visualized at steps 10 and 250 of DDPM. The results show significantly better feature smoothness in Skip-DiT. Furthermore, we identify feature smoothness as a critical factor limiting the effectiveness of cross-timestep feature caching in DiT. This insight provides a deeper understanding of caching efficiency and its impact on performance.\n\n### Pretrained Models\n| Model | Task | Training Data | Backbone | Size(G) | Skip-Cache |\n|:--:|:--:|:--:|:--:|:--:|:--:|\n| [Latte-skip](https://huggingface.co/GuanjieChen/Skip-DiT/blob/main/DiT-XL-2-skip.pt) | text-to-video |Vimeo|Latte|8.76| ✅ |\n| [DiT-XL/2-skip](https://huggingface.co/GuanjieChen/Skip-DiT/blob/main/Latte-skip.pt) | class-to-image |ImageNet|DiT-XL/2|11.40|✅ |\n| [ucf101-skip](https://huggingface.co/GuanjieChen/Skip-DiT/blob/main/ucf101-skip.pt) | class-to-video|UCF101|Latte|2.77|✅ |\n| [taichi-skip](https://huggingface.co/GuanjieChen/Skip-DiT/blob/main/taichi-skip.pt) | class-to-video|Taichi-HD|Latte|2.77|✅ |\n| [skytimelapse-skip](https://huggingface.co/GuanjieChen/Skip-DiT/blob/main/skylapse-skip.pt) | class-to-video|SkyTimelapse|Latte|2.77|✅ |\n| [ffs-skip](https://huggingface.co/GuanjieChen/Skip-DiT/blob/main/ffs-skip.pt) | class-to-video|FaceForensics|Latte|2.77|✅ |\n\nPretrained text-to-image Model of [HunYuan-DiT](https://github.com/Tencent/HunyuanDiT) can be found in [Huggingface](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2/tree/main/t2i/model) and [Tencent-cloud](https://dit.hunyuan.tencent.com/download/HunyuanDiT/model-v1_2.zip).\n### Installation\nTo prepare environments for `class-to-video`, `text-to-video` tasks, please refer to [Latte](https://github.com/Vchitect/Latte) or you can:\n```shell\ncd class-to-video\nconda env create -f environment.yaml\nconda activate latte\n```\n\nTo prepare environments for `class-to-image` task, please refer to [DiT](https://github.com/facebookresearch/DiT) or you can:\n```shell\ncd class-to-image\nconda env create -f environment.yaml\nconda activate DiT\n```\n\nTo prepare environments for text-to-image task, please refer to [Hunyuan-DiT](https://github.com/Tencent/HunyuanDiT):\n```shell\ncd text-to-image\nconda env create -f environment.yaml\nconda activate HunyuanDiT\n```\n\n### Download pretrained models\nTo download models, first install:\n```shell\npython -m pip install \"huggingface_hub[cli]\"\n```\nTo download models needed in text-to-video task:\n```shell\n# 1. Download the vae, encoder, tokenizer and orinal checkpoints of Latte. Models are download default to ./text-to-video/pretrained/\ncd text-to-video\npython pretrained/download_ckpts.py\n\n# 2. Download the checkpoint of Latte-skip\nhuggingface-cli download GuanjieChen/Skip-DiT/Latte-skip.pt -d ./pretrained/\n```\n\nTo download models needed in class-to-video tasks.\n```shell\ncd class-to-video\n# 1. Download vae models and original checkpoints of Latte:\npython pretrained/download_ckpts.py\n\n# 2. Download the checkpoint of Latte-skip\ntask_name=ffs # or UCF101, skytimelapse, taichi\nhuggingface-cli download GuanjieChen/Skip-DiT/${task}-skip.pt -d ./pretrained/\n```\n\nTo download models needed in text-to-image task:\n```shell\ncd text-to-image\nmkdir ckpts\nhuggingface-cli download Tencent-Hunyuan/HunyuanDiT-v1.2 --local-dir ./ckpts\n```\n\nTo download models needed in class-to-image task:\n```shell\ncd class-to-image\nmkdir ckpts\n# 1. download DiT-XL/2\nwget -P ckpts https://dl.fbaipublicfiles.com/DiT/models/DiT-XL-2-256x256.pt\n# 2. download DiT-XL/2-skip\nhuggingface-cli download GuanjieChen/Skip-DiT/DiT-XL-2-skip.pt -d ./ckpts\n```\n\n### Inference \u0026 Acceleration\nTo infer with text-to-video models:\n```shell\ncd text-to-video\n# 1. infer with original latte\n./sample/t2v.sh\n# 2. infer with skip dit\n./sample/t2v_skip.sh\n# 3. accelerate with skip-cache\n./sample/t2v_skip_cache.sh\n```\n\nTo infer with class-to-video models:\n```shell\ntask_name=ffs # or UCF101, skytimelapse, taichi-hd\ncd class-to-video\n# 1. infer with original latte\n./sample/${task_name}.sh\n# 2. infer with skip dit\n./sample/${task_name}_skip.sh\n# 3. accelerate with skip-cache\n./sample/${task_name}_cache.sh\n```\n\nTo infer with text-to-image model, please follow the instructions in the official implementation of [HunYuan-DiT](https://github.com/Tencent/HunyuanDiT.git). To use **Skip-Cache**, add the following command to inference scripts `./infer.sh`:\n```shell\n--deepcache -N {2,3,4...}\n```\n\n\nTo infer with class-to-image models:\n```shell\ncd class-to-image\n# 1. infer with DiT-XL/2\npython sample.py --ckpt path/to/DiT-XL-2.pt --model DiT-XL/2\n# 2. infer with DiT-XL/2-skip\npython sample.py --ckpt path/to/DiT-XL-2-skip.pt --model DiT-skip\n# 3. accelerate with skip-cache\npython sample.py --ckpt path/to/DiT-XL-2-skip.pt --model DiT-cache-2 # or DiT-cache-3, DiT-cache-4...\n```\n\n### Training\nTo train the DiT-XL/2-skip:\n1. Download the [ImageNet](https://www.image-net.org/) dataset.\n2. Implement the TODO in the train.py\n3. run the script `run_train.sh`\n\nTo train the class-to-video models:\n1. Download the datasets offered by [Xin Ma](https://huggingface.co/maxin-cn) in huggingface: [skytimelapse](Skip-DiT-open/maxin-cn/SkyTimelapse), [taichi](Skip-DiT-open/maxin-cn/Taichi-HD), [ffs](Skip-DiT-open/maxin-cn/FaceForensics). And you have to download [ucf101](https://www.crcv.ucf.edu/data/UCF101/UCF101.rar) from the website.\n2. Implement the TODOs in the training configs under `class-to-video/configs`\n3. Run the training scripts under `class-to-image/train_scripts`\n\nTo train the text-to-video model:\n1. Prepare your text-video datasets and implement the `text-to-video/datasets/t2v_joint_dataset.py`\n2. Run the two-stage training strategy:\n   1. Freeze all the parameters except skip-branches. Set `freeze=True` in `text-to-video/configs/train_t2v.yaml`. And then run the training scripts.\n   2. Overall training. Set `freeze=False` in `text-to-video/configs/train_t2v.yaml`. And then run the training scripts.\n\n### Acknowledgement\nSkip-DiT has been greatly inspired by the following amazing works and teams: [DeepCache](https://arxiv.org/abs/2312.00858), [Latte](https://github.com/Vchitect/Latte), [DiT](https://github.com/facebookresearch/DiT), and [HunYuan-DiT](https://github.com/Tencent/HunyuanDiT), we thank all the contributors for open-sourcing.\n\n### License\nThe code and model weights are licensed under [LICENSE](./class-to-image/LICENSE).\n\n\n### Visualization\n#### Text-to-Video\n![text-to-video visualizations](visuals/case_t2v.jpg)\n#### Class-to-Video\n![class-to-video visualizations](visuals/case_c2v.jpg)\n#### Text-to-image\n![text-to-image visualizations](visuals/case_t2i.jpg)\n#### Class-to-image\n![class-to-image visualizations](visuals/case_c2i.jpg)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenSparseLLMs%2FSkip-DiT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FOpenSparseLLMs%2FSkip-DiT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenSparseLLMs%2FSkip-DiT/lists"}