{"id":17719835,"url":"https://github.com/zjysteven/controlnet_tile","last_synced_at":"2026-05-02T19:35:43.096Z","repository":{"id":237781347,"uuid":"795224617","full_name":"zjysteven/controlnet_tile","owner":"zjysteven","description":"Workable training script for ControlNet tile","archived":false,"fork":false,"pushed_at":"2024-05-02T21:04:44.000Z","size":408,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-05-03T06:20:03.571Z","etag":null,"topics":["controlnet","controlnet-tile","stable-diffusion"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zjysteven.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-02T20:40:50.000Z","updated_at":"2024-05-02T21:22:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"1d75746b-14e2-417a-8b09-c0416f1b9064","html_url":"https://github.com/zjysteven/controlnet_tile","commit_stats":null,"previous_names":["zjysteven/controlnet_tile"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zjysteven/controlnet_tile","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjysteven%2Fcontrolnet_tile","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjysteven%2Fcontrolnet_tile/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjysteven%2Fcontrolnet_tile/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjysteven%2Fcontrolnet_tile/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zjysteven","download_url":"https://codeload.github.com/zjysteven/controlnet_tile/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjysteven%2Fcontrolnet_tile/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32547650,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-02T19:18:06.202Z","status":"ssl_error","status_checked_at":"2026-05-02T19:16:21.335Z","response_time":132,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["controlnet","controlnet-tile","stable-diffusion"],"created_at":"2024-10-25T15:09:39.318Z","updated_at":"2026-05-02T19:35:43.080Z","avatar_url":"https://github.com/zjysteven.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Training ControlNet tile\n\nCode for training ControlNet tile, adapted from [diffusers controlnet training example](https://github.com/huggingface/diffusers/blob/main/examples/controlnet/train_controlnet.py) by using WebDataset instead of huggingface style dataset specified in that example. Follow the [instructions](https://github.com/huggingface/diffusers/tree/main/examples/controlnet#installing-the-dependencies) for setup.\n\n```bash\nMAX_STEPS=10000\nLR=1e-5\nBS=32\nPROMPT_DROPOUT=0.05\nOUTPUT_DIR=\"controlnet/minisd_${BS}_${LR}_${MAX_STEPS}_dropout${PROMPT_DROPOUT}\"\nMODEL_NAME=\"lambdalabs/miniSD-diffusers\"\n\naccelerate launch --multi_gpu train_controlnet.py \\\n    --pretrained_model_name_or_path=$MODEL_NAME \\\n    --output_dir=$OUTPUT_DIR \\\n    --condition_resolution=64 \\\n    --resolution=256 \\\n    --learning_rate=${LR} \\\n    --max_train_steps=${MAX_STEPS} \\\n    --max_train_samples=80000000 \\\n    --dataloader_num_workers=8 \\\n    --train_shards_path_or_url=\"/data/laion400m-data/{00000..10200}.tar\" \\\n    --validation_image \"./conditioning_image_1.png\" \"./conditioning_image_2.jpeg\" \\\n    --validation_prompt \"a dog sitting on the grass\" \"home office\" \\\n    --validation_steps=100 \\\n    --checkpointing_steps=1000 --checkpoints_total_limit=10 \\\n    --train_batch_size=${BS} \\\n    --gradient_checkpointing --enable_xformers_memory_efficient_attention \\\n    --gradient_accumulation_steps=1 \\\n    --use_8bit_adam \\\n    --resume_from_checkpoint=latest \\\n    --mixed_precision=\"fp16\" \\\n    --tracker_project_name=\"controlnet\" \\\n    --report_to=wandb \\\n    --proportion_empty_prompts ${PROMPT_DROPOUT}\n```\n\nAbove is the exact training script that I used to train a controlnet tile w.r.t. [lambdalabs/miniSD-diffusers](https://huggingface.co/lambdalabs/miniSD-diffusers), a 256x256 SD model. So in my case I was doing 64x64 -\u003e 256x256 upsampling. The key data preprocessing part in the code is at [here](https://github.com/zjysteven/controlnet_tile/blob/d92aa059c3281e81e84b94de17a044e575c5852a/train_controlnet.py#L744-L766), where you could see I was using low-resolution images as conditions and high-resolution images as targets.\n\n\n\nI was using laion400m as training data. The training was done on 8x A5000 (24G) GPUs within 5 or 6 hours (with 10k training steps) if I remember correctly. The training essentially worked, which can be seen in the below screenshot. Interestingly, I did observe the [\"sudden convergence\" phenomenon](https://github.com/lllyasviel/ControlNet/blob/main/docs/train.md#more-consideration-sudden-converge-phenomenon-and-gradient-accumulation) mentioned by the author of ControlNet, where in the first 3k steps the output images didn't follow conditions at all, yet suddenly after that it started to follow the conditions.\n\n![test_screenshot](./test_screenshot.png)\n\nFor your own case you probably need to change batch size and learning rate. I didn't play with learning rate in my case as `1e-5` just worked.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjysteven%2Fcontrolnet_tile","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzjysteven%2Fcontrolnet_tile","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjysteven%2Fcontrolnet_tile/lists"}