{"id":27752109,"url":"https://github.com/wileewang/TransPixeler","last_synced_at":"2025-04-29T03:02:41.618Z","repository":{"id":271326470,"uuid":"907686106","full_name":"wileewang/TransPixeler","owner":"wileewang","description":"CVPR2025","archived":false,"fork":false,"pushed_at":"2025-04-28T06:27:15.000Z","size":372,"stargazers_count":846,"open_issues_count":20,"forks_count":68,"subscribers_count":22,"default_branch":"main","last_synced_at":"2025-04-28T07:28:01.472Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wileewang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-24T06:37:35.000Z","updated_at":"2025-04-28T06:27:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"3111a1a3-1db9-4a25-aa39-7b788cb7bde6","html_url":"https://github.com/wileewang/TransPixeler","commit_stats":null,"previous_names":["wileewang/transpixar"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wileewang%2FTransPixeler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wileewang%2FTransPixeler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wileewang%2FTransPixeler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wileewang%2FTransPixeler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wileewang","download_url":"https://codeload.github.com/wileewang/TransPixeler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251425692,"owners_count":21587435,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-29T03:02:39.938Z","updated_at":"2025-04-29T03:02:41.610Z","avatar_url":"https://github.com/wileewang.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"## TransPixeler: Advancing Text-to-Video Generation with Transparency (CVPR2025)\n\u003cbr\u003e\n\u003ca href=\"https://arxiv.org/abs/2501.03006\"\u003e\u003cimg src='https://img.shields.io/badge/arXiv-2501.03006-b31b1b.svg'\u003e\u003c/a\u003e\n\u003ca href='https://wileewang.github.io/TransPixeler'\u003e\u003cimg src='https://img.shields.io/badge/Project_Page-TransPixeler-blue'\u003e\u003c/a\u003e\n\u003ca href='https://huggingface.co/spaces/wileewang/TransPixar'\u003e\u003cimg src='https://img.shields.io/badge/HuggingFace-TransPixeler-yellow'\u003e\u003c/a\u003e\n\u003ca href=\"https://discord.gg/7Xds3Qjr\"\u003e\u003cimg src=\"https://img.shields.io/badge/Discord-join-blueviolet?logo=discord\u0026amp\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/wileewang/TransPixar/blob/main/wechat_group.jpg\"\u003e\u003cimg src=\"https://img.shields.io/badge/Wechat-Join-green?logo=wechat\u0026amp\"\u003e\u003c/a\u003e\n\u003cbr\u003e\n\n[Luozhou Wang*](https://wileewang.github.io/), \n[Yijun Li**](https://yijunmaverick.github.io/), \n[Zhifei Chen](), \n[Jui-Hsien Wang](http://juiwang.com/), \n[Zhifei Zhang](https://zzutk.github.io/), \n[He Zhang](https://sites.google.com/site/hezhangsprinter), \n[Zhe Lin](https://sites.google.com/site/zhelin625/home), \n[Ying-Cong Chen†](https://www.yingcong.me)\n\nHKUST(GZ), HKUST, Adobe Research.\n\n\\* Internship Project  \n\\** Project Lead  \n† Corresponding Author\n\nText-to-video generative models have made significant strides, enabling diverse applications in entertainment, advertising, and education. However, generating RGBA video, which includes alpha channels for transparency, remains a challenge due to limited datasets and the difficulty of adapting existing models. Alpha channels are crucial for visual effects (VFX), allowing transparent elements like smoke and reflections to blend seamlessly into scenes.  \nWe introduce TransPixar, a method to extend pretrained video models for RGBA generation while retaining the original RGB capabilities. TransPixar leverages a diffusion transformer (DiT) architecture, incorporating alpha-specific tokens and using LoRA-based fine-tuning to jointly generate RGB and alpha channels with high consistency. By optimizing attention mechanisms, TransPixeler preserves the strengths of the original RGB model and achieves strong alignment between RGB and alpha channels despite limited training data.  \nOur approach effectively generates diverse and consistent RGBA videos, advancing the possibilities for VFX and interactive content creation.\n\n\u003c!-- insert a teaser gif --\u003e\n\u003c!-- \u003cimg src=\"assets/mi.gif\"  width=\"640\" /\u003e --\u003e\n\n\n\n## 📰 News\n\n- **[2025.04.28]** We have introduced a new development branch [`wan`](https://github.com/wileewang/TransPixar/tree/wan) that integrates the [Wan2.1](https://github.com/Wan-Video/Wan2.1) video generation model to support **joint generation** tasks. This branch includes training code tailored for generating both RGB and associated modalities (e.g., segmentation maps, alpha masks) from a shared text prompt.\n\n- **[2025.02.26]** **TransPixeler** is accepted by CVPR 2025! See you in Nashville!\n\n- **[2025.01.19]** We've renamed our project from **TransPixar** to **TransPixeler**!!\n\n- **[2025.01.17]** We’ve created a [Discord group](https://discord.gg/7Xds3Qjr) and a [WeChat group](https://github.com/wileewang/TransPixar/blob/main/wechat_group.jpg)! Everyone is welcome to join for discussions and collaborations.\n\n- **[2025.01.14]** Added new tasks to the repository's roadmap, including support for Hunyuan and LTX video models, and ComfyUI integration.\n\n- **[2025.01.07]** Released project page, arXiv paper, inference code, and Hugging Face demo.\n\n\n\n\n## 🔥 New Branch for Joint Generation with Wan2.1\n\nWe have introduced a new development branch [`wan`](https://github.com/wileewang/TransPixar/tree/wan) that integrates the [Wan2.1](https://github.com/Wan-Video/Wan2.1) video generation model to support **joint generation** tasks.\n\nIn the `wan` branch, we have developed and released training code tailored for joint generation scenarios, enabling the simultaneous generation of RGB videos and associated modalities (e.g., segmentation maps, alpha masks) from a shared text prompt.\n\n**Key features of the `wan` branch:**\n- **Integration of Wan2.1**: Leverages the capabilities of the Wan2.1 video generation model for enhanced performance.\n- **Joint Generation Support**: Facilitates the concurrent generation of RGB and paired modality videos.\n- **Dataset Structure**: Expects each sample to include:\n  - A primary video file (`001.mp4`) representing the RGB content.\n  - A paired secondary video file (`001_seg.mp4`) with a fixed `_seg` suffix, representing the associated modality.\n  - A caption text file (`001.txt`) with the same base name as the primary video.\n- **Periodic Evaluation**: Supports periodic video sampling during training by setting `eval_every_step` or `eval_every_epoch` in the configuration.\n- **Customized Pipelines**: Offers tailored training and inference pipelines designed specifically for joint generation tasks.\n\n👉 To utilize the joint generation features, please checkout the [`wan`](https://github.com/wileewang/TransPixar/tree/wan) branch.\n\n\n\n\n## Contents\n\n* [Installation](#installation)\n* [TransPixar LoRA Weights](#transpixar-lora-hub) \n* [Training](#training)\n* [Inference](#inference)\n* [Acknowledgement](#acknowledgement)\n* [Citation](#citation)\n\n\n\n## Installation\n\n```bash\n# For the main branch\nconda create -n TransPixeler python=3.10\nconda activate TransPixeler\npip install -r requirements.txt\n```\n\n**Note:**  \nIf you want to use the **Wan2.1 model**, please first checkout the `wan` branch:\n\n```bash\ngit checkout wan\n```\n\n## TransPixeler LoRA Weights\n\nOur pipeline is designed to support various video tasks, including Text-to-RGBA Video, Image-to-RGBA Video.\n\nWe provide the following pre-trained LoRA weights:\n\n| Task          | Base Model                                                    | Frames | LoRA weights                                                       | Inference VRAM |\n|---------------|---------------------------------------------------------------|--------|--------------------------------------------------------------------|----------------|\n| T2V + RGBA    | [THUDM/CogVideoX-5B](https://huggingface.co/THUDM/CogVideoX-5b)       | 49     | [link](https://huggingface.co/wileewang/TransPixar/blob/main/cogvideox_rgba_lora.safetensors) | ~24GB          |\n\n\n## Training - RGB + Alpha Joint Generation\nWe have open-sourced the training code for **Mochi** on RGBA joint generation. Please refer to the [Mochi README](Mochi/README.md) for details.\n\n\n## Inference - Gradio Demo\nIn addition to the [Hugging Face online demo](https://huggingface.co/spaces/wileewang/TransPixar), users can also launch a local inference demo based on CogVideoX-5B by running the following command:\n\n```bash\npython app.py\n```\n\n## Inference - Command Line Interface (CLI)\nTo generate RGBA videos, navigate to the corresponding directory for the video model and execute the following command:\n```bash\npython cli.py \\\n    --lora_path /path/to/lora \\\n    --prompt \"...\"\n```\n\n---\n\n## Acknowledgement\n\n* [finetrainers](https://github.com/a-r-r-o-w/finetrainers): We followed their implementation of Mochi training and inference.\n* [CogVideoX](https://github.com/THUDM/CogVideo): We followed their implementation of CogVideoX training and inference.\n\nWe are grateful for their exceptional work and generous contribution to the open-source community.\n\n## Citation\n\n```bibtex\n@misc{wang2025transpixeler,\n      title={TransPixeler: Advancing Text-to-Video Generation with Transparency}, \n      author={Luozhou Wang and Yijun Li and Zhifei Chen and Jui-Hsien Wang and Zhifei Zhang and He Zhang and Zhe Lin and Ying-Cong Chen},\n      year={2025},\n      eprint={2501.03006},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2501.03006}, \n}\n```\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=wileewang/TransPixeler\u0026type=Date)](https://star-history.com/#wileewang/TransPixar\u0026Date)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwileewang%2FTransPixeler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwileewang%2FTransPixeler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwileewang%2FTransPixeler/lists"}