{"id":37115052,"url":"https://github.com/hohonu-vicml/trailblazer","last_synced_at":"2026-01-14T13:30:54.964Z","repository":{"id":214862490,"uuid":"736629470","full_name":"hohonu-vicml/TrailBlazer","owner":"hohonu-vicml","description":"[SIGGRAPH Asia 2024] TrailBlazer: Trajectory Control for Diffusion-Based Video Generation","archived":false,"fork":false,"pushed_at":"2024-05-31T22:11:21.000Z","size":103680,"stargazers_count":100,"open_issues_count":4,"forks_count":12,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-06-09T13:40:35.691Z","etag":null,"topics":["boundingbox","controlability","deeplearning","diffusion","keyframe-animation","videogeneration","zeroshot"],"latest_commit_sha":null,"homepage":"https://hohonu-vicml.github.io/Trailblazer.Page/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hohonu-vicml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-28T12:29:16.000Z","updated_at":"2025-06-03T07:54:04.000Z","dependencies_parsed_at":"2024-04-10T15:30:54.569Z","dependency_job_id":"968e88f4-c8d9-494b-b783-49e043c44582","html_url":"https://github.com/hohonu-vicml/TrailBlazer","commit_stats":null,"previous_names":["hohonu-vicml/trailblazer"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/hohonu-vicml/TrailBlazer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hohonu-vicml%2FTrailBlazer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hohonu-vicml%2FTrailBlazer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hohonu-vicml%2FTrailBlazer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hohonu-vicml%2FTrailBlazer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hohonu-vicml","download_url":"https://codeload.github.com/hohonu-vicml/TrailBlazer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hohonu-vicml%2FTrailBlazer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28421217,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T10:47:48.104Z","status":"ssl_error","status_checked_at":"2026-01-14T10:46:19.031Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["boundingbox","controlability","deeplearning","diffusion","keyframe-animation","videogeneration","zeroshot"],"created_at":"2026-01-14T13:30:54.127Z","updated_at":"2026-01-14T13:30:54.937Z","avatar_url":"https://github.com/hohonu-vicml.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# ___***TrailBlazer***___\n\n[![Paper](https://img.shields.io/badge/cs.CV-Paper-b31b1b?logo=arxiv\u0026logoColor=red)](https://arxiv.org/abs/2401.00896)\n[![Project Page](https://img.shields.io/badge/TrailBlazer-Website-green?logo=googlechrome\u0026logoColor=green)](https://hohonu-vicml.github.io/Trailblazer.Page/)\n[![HuggingFace](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Demo-blue)](https://huggingface.co/spaces/hohonu-vicml/Trailblazer)\n[![Video](https://img.shields.io/badge/YouTube-Project-c4302b?logo=youtube\u0026logoColor=red)](https://www.youtube.com/watch?v=kEN-32wN-xQ)\n[![Video](https://img.shields.io/badge/YouTube-Result-c4302b?logo=youtube\u0026logoColor=red)](https://www.youtube.com/watch?v=P-PSkS7sNco)\n[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2Fhohonu-vicml%2FTrailblazer\u0026count_bg=%238B00FB\u0026title_bg=%23555555\u0026icon=\u0026icon_color=%23E7E7E7\u0026title=hits\u0026edge_flat=false)](https://hits.seeyoufarm.com)\n\n\nThis repository contains the implementation of the following paper: \u003e\n**TrailBlazer: Trajectory Control for Diffusion-Based Video Generation**\u003cbr\u003e \u003e\n[Wan-Duo Kurt Ma](https://www.linkedin.com/in/kurt-ma/)\u003csup\u003e1\u003c/sup\u003e, [J.P.\nLewis](http://www.scribblethink.org/)\u003csup\u003e2\u003c/sup\u003e, [ W. Bastiaan\nKleijn](https://people.wgtn.ac.nz/bastiaan.kleijn)\u003csup\u003e1\u003c/sup\u003e,\u003cbr\u003e Victoria\nUniversity of Wellington\u003csup\u003e1\u003c/sup\u003e, NVIDIA Research\u003csup\u003e2\u003c/sup\u003e\n\n## :fire: Overview\n![teaser](./assets/teaser.gif)\n\nLarge text-to-video (T2V) models such as Sora have the potential to revolutionize visual effects and the creation of some types of movies. Current T2V models require tedious trial-and-error experimentation to achieve desired results, however. This motivates the search for methods to directly control desired attributes. In this work, we take a step toward this goal, introducing a method for high-level, temporally-coherent control over the basic trajectories and appearance of objects. Our algorithm, **TrailBlazer**, allows the general positions and (optionally) appearance of objects to controlled simply by keyframing approximate bounding boxes and (optionally) their corresponding prompts.\n\nImportantly, our method does not require a pre-existing control video signal that already contains an accurate outline of the desired motion, yet the synthesized motion is surprisingly natural with emergent effects including perspective and movement toward the virtual camera as the box size increases. The method is efficient, making use of a pre-trained T2V model and requiring no training or fine-tuning, with negligible additional computation. Specifically, the bounding box controls are used as soft masks to guide manipulation of the self-attention and cross-attention modules in the video model. While our visual results are limited by those of the underlying model, the algorithm may generalize to future models that use standard self- and cross-attention components.\n\n## :fire: Requirements\n\nThe codebase is tested under **NVIDIA GeForce RTX 3090** with the python library\n**pytorch-2.1.2+cu121** and **diffusers-0.21.4**. We strongly recommend using a\nspecific version of Diffusers as it is continuously evolving. For PyTorch, you\ncould probably use other version under 2.x.x. With RTX 3090, I follow the\n[post](https://discuss.pytorch.org/t/geforce-rtx-3090-with-cuda-capability-sm-86-is-not-compatible-with-the-current-pytorch-installation/123499)\nto avoid the compatibility of sm_86 issue.\n\n## :fire: Timeline\n\n-   [2024/04/08]: Our new v2 preprint is now appeared on ArXiv (See [link](https://arxiv.org/abs/2401.00896))\n\n-   [2024/03/23]: A new ArXiv update will be made.\n\n-   [2024/03/22]: We release the multiple object synthesis (See\n    [link](doc/Command.md#multiple-objects)), and the\n    [Peekaboo](https://github.com/microsoft/Peekaboo) integration (See\n    [link](doc/Peekaboo.md))\n\n-   [2024/02/07]: The Gradio app is updated with better keyframe interface (See\n    ([link](assets/gradio/gradio.jpg)))\n\n-   [2024/02/06]: We now have Gradio web app at Huggingface Space!\n\n-   [2024/02/01]: The official codebase released\n\n-   [2024/01/03]: Paper released\n\n-   [2023/12/31]: Paper submitted on ArXiv\n\n## :fire: Usage\n\n#### [Prepare]\n\nFirst of all, download the pre-trained zeroscope model\n([link](https://huggingface.co/cerspense/zeroscope_v2_576w)). You need to\nregister huggingface and make access token ([link](https://huggingface.co/))\n\n```bash\ngit clone https://huggingface.co/cerspense/zeroscope_v2_576w ${MODEL_ROOT}/cerspense/zeroscope_v2_576w\n```\n\nwhere MODEL_ROOT is your preference that stores the model. Then, clone this Repo and cd into it:\n```bash\ngit clone https://github.com/hohonu-vicml/Trailblazer \u0026\u0026 cd Trailbalzer\n```\n\n#### [Run it]\n\nOur executable script is located in the \"bin\" folder, and the core module is\nimplemented in the \"TrailBlazer\" folder under the project root. Therefore, no\nadditional dependencies need to be added to PYTHONPATH; you can simply run the\ncommand below :smirk: :\n\n```bash\npython bin/CmdTrailBlazer.py -mr ${MODEL_ROOT} --config config/XXXX.yaml  ## single experiment\npython bin/CmdTrailBlazer.py -mr ${MODEL_ROOT} --config config/  ## run all yamls in a folder\n```\n\n:cupid:**UPDATE**:cupid:: TrailBlazer has just released Gradio app for the\nalternative interface. Please checkout our documentation\n([Gradio.md](doc/Gradio.md)) for more information. To run the app, simply run:\n\n```bash\npython bin/CmdGradio.py ${MODEL_ROOT} # no -mr here\n```\n\nWhen the shell environment variable ZEROSCOPE_MODEL_ROOT is specified, then you\ncan ignore the -mr (--model-root) argument above.\n\n```bash\nexport ZEROSCOPE_MODEL_ROOT=/path/to/your/diffusion/root\n# then you can ignore -mr term to simplify the command\npython bin/CmdTrailBlazer.py --config config/XXXX.yaml\n```\n\nPlease see [here](doc/Command.md) for more information about the command set\nused in TrailBlazer.\n\n#### [Config]\n\nA list of config example files is stored in the `config` folder. Feel free to\nrun each of them and the result will be written in the `/tmp` folder. For more\ninformation how to design the config file, and the visual result of each config.\nPlease visit [here](doc/Config.md) and [there](config/README.md) for more\ndetails about config structure and the visual result, respectively.\n\n## :fire: Contribution\n\nThis project is still working in progress, and there are numerous directions in\nwhich it can be improved. Please don't hesitate to contact us if you are\ninterested, or feel free to make a pull request to strengthen the ideas.\n\n## :fire: TODO\n\nWe regret to inform you that this repository is currently not fully accessible\nto the public. Nevertheless, the majority of the core modules have been made\navailable (e.g., Single, Multiple objects synthesis, and Peekaboo comparison).\nOur next release will include useful tools for measuring metrics.\n\n## :fire: Fun\n\n\n\u003cimg src=\"./assets/v1-TrailBlazer/SpeedKeys-Cat.0000.gif\"\u003e\n\nPoor cat: Someone, Stop me!\n\n\u003cimg src=\"./assets/v1-TrailBlazer/Cat2Dog.0000.gif\"\u003e\n\nAm I a cat, or a dog...\n\nPlease inform us if you have generated any interesting videos!\n\n## :fire: Citation\n\n**TrailBlazer** is built on top of its mother project [**DirectedDiffusion**](https://hohonu-vicml.github.io/DirectedDiffusion.Page/), which recently published at AAAI2024. If you find our work useful for your research, please consider citing our paper.\n\n   ```bibtex\n   @article{ma2023trailblazer,\n       title={TrailBlazer: Trajectory Control for Diffusion-Based Video Generation},\n       author={Wan-Duo Kurt Ma and J. P. Lewis and W. Bastiaan Kleijn},\n       year={2023},\n       eprint={2401.00896},\n       archivePrefix={arXiv},\n       primaryClass={cs.CV}\n   }\n\n   @article{ma2023directed,\n       title={Directed Diffusion: Direct Control of Object Placement through Attention Guidance},\n       author={Wan-Duo Kurt Ma and J. P. Lewis and Avisek Lahiri and Thomas Leung and W. Bastiaan Kleijn},\n       year={2023},\n       eprint={2302.13153},\n       archivePrefix={arXiv},\n       primaryClass={cs.CV}\n   }\n   ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhohonu-vicml%2Ftrailblazer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhohonu-vicml%2Ftrailblazer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhohonu-vicml%2Ftrailblazer/lists"}