{"id":21135233,"url":"https://github.com/cloneofsimo/repa-rf","last_synced_at":"2025-07-09T03:32:47.524Z","repository":{"id":260749981,"uuid":"882241555","full_name":"cloneofsimo/repa-rf","owner":"cloneofsimo","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-02T09:37:02.000Z","size":0,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-02T10:23:20.452Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloneofsimo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-02T09:25:07.000Z","updated_at":"2024-11-02T10:12:05.000Z","dependencies_parsed_at":"2024-11-02T10:34:11.647Z","dependency_job_id":null,"html_url":"https://github.com/cloneofsimo/repa-rf","commit_stats":null,"previous_names":["cloneofsimo/repa-rf"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloneofsimo%2Frepa-rf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloneofsimo%2Frepa-rf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloneofsimo%2Frepa-rf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloneofsimo%2Frepa-rf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloneofsimo","download_url":"https://codeload.github.com/cloneofsimo/repa-rf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225482022,"owners_count":17481179,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T06:46:46.601Z","updated_at":"2024-11-20T06:46:47.197Z","avatar_url":"https://github.com/cloneofsimo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# REPA - Representation Alignment for Diffusion Transformers \n\nUnofficial PyTorch implementation of [\"Representation Alignment for Generation: Training Diffusion Transformers is Easier Than You Think\"](https://arxiv.org/abs/2410.06940), with DDP and recitified flow objective.\n\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"contents/image.png\" alt=\"REPA overview diagram\"/\u003e\n\u003c/p\u003e\n\n## Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/cloneofsimo/repa-rf\ncd repa\n\n# Create conda environment\nconda create -n repa python=3.10\nconda activate repa\n\n# Install PyTorch 2.0+ and other dependencies\npip install torch torchvision click transformers diffusers\n```\n\n## Getting Started\n\n### Preparing Datasets\n\nWe have dataset preparation code for CC12M in preprocessing_script. It will download [cc12m-wds](https://huggingface.co/datasets/pixparse/cc12m-wds) from HuggingFace.\n\n### Training\n\nBasic training command:\n\n```bash\ntorchrun --nproc_per_node=8 trainer.py \\\n    --run_name test_run \\\n    --model_width 512 \\\n    --learning_rate 1e-4 \\\n    --dataset_url \"path/to/dataset/{0000..2151}.tar\" \\\n    --test_dataset_url \"path/to/test/dataset/{2152..2168}.tar\" \\\n    --num_epochs 2 \\\n    --batch_size 64 \\\n    --max_steps 5000 \\\n    --evaluate_every 1000 \\\n    --alignment_layer 8 \\\n    --repa_lambda 0.5 \\\n    --model_depth 9 \\\n    --model_head_dim 32 \\\n    --compile_models True\n```\n\nKey parameters:\n- `model_width`: Width of the transformer model (128, 512, 2048)\n- `repa_lambda`: Weight for the REPA loss (0.25-1.0)\n- `alignment_layer`: Which layer to apply REPA (typically 8)\n\n\n## Model Architecture and Components\n\nThe codebase includes:\n\n- `model.py`: Core DiT architecture with REPA modifications\n- `trainer.py`: Distributed training loop implementation\n- `utils.py`: Helper functions and utilities\n- `dinov2.py`: DINOv2 model implementation for representation alignment\n\nKey components:\n1. DiT/SiT transformer backbone \n2. REPA loss\n3. Distributed training support\n\n## Results (from Paper)\n\n| Model | Training Steps | FID↓ | IS↑ |\n|-------|---------------|------|------|\n| SiT-XL/2 | 7M | 8.3 | 131.7 |\n| +REPA | 400K | 7.9 | 122.6 |\n| +REPA | 4M | 5.9 | 157.8 |\n\n## Citation\n\n```bibtex\n@article{yu2024representation,\n  title={Representation Alignment for Generation: Training Diffusion Transformers is Easier Than You Think},\n  author={Yu, Sihyun and Kwak, Sangkyung and Jang, Huiwon and Jeong, Jongheon and Huang, Jonathan and Shin, Jinwoo and Xie, Saining},\n  journal={arXiv preprint arXiv:2410.06940},\n  year={2024}\n}\n```\n\n## Acknowledgments\n\nThis implementation uses code from:\n- [DINOv2](https://github.com/facebookresearch/dinov2)\n\n## License\n\nThis project is released under the MIT License.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloneofsimo%2Frepa-rf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloneofsimo%2Frepa-rf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloneofsimo%2Frepa-rf/lists"}