{"id":14887007,"url":"https://github.com/RedAIGC/StoryMaker","last_synced_at":"2025-09-21T11:30:41.098Z","repository":{"id":257789630,"uuid":"851056638","full_name":"RedAIGC/StoryMaker","owner":"RedAIGC","description":"StoryMaker: Towards consistent characters in text-to-image generation","archived":false,"fork":false,"pushed_at":"2024-12-02T11:46:51.000Z","size":24143,"stargazers_count":619,"open_issues_count":18,"forks_count":53,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-01-09T11:06:58.902Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RedAIGC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-02T10:48:19.000Z","updated_at":"2025-01-08T08:30:19.000Z","dependencies_parsed_at":"2024-12-02T12:37:22.519Z","dependency_job_id":null,"html_url":"https://github.com/RedAIGC/StoryMaker","commit_stats":null,"previous_names":["redaigc/storymaker"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedAIGC%2FStoryMaker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedAIGC%2FStoryMaker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedAIGC%2FStoryMaker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedAIGC%2FStoryMaker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RedAIGC","download_url":"https://codeload.github.com/RedAIGC/StoryMaker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233748111,"owners_count":18724007,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-21T18:01:15.171Z","updated_at":"2025-09-21T11:30:34.995Z","avatar_url":"https://github.com/RedAIGC.png","language":"Python","funding_links":[],"categories":["\u003cspan id=\"image\"\u003eImage\u003c/span\u003e","Personalized Restoration"],"sub_categories":["\u003cspan id=\"tool\"\u003eLLM (LLM \u0026 Tool)\u003c/span\u003e"],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003eStoryMaker: Towards consistent characters in text-to-image generation\u003c/h1\u003e\n\n\u003ca href='https://arxiv.org/pdf/2409.12576'\u003e\u003cimg src='https://img.shields.io/badge/Technique-Report-red'\u003e\u003c/a\u003e\n\u003ca href='https://huggingface.co/RED-AIGC/StoryMaker'\u003e\u003cimg src='https://img.shields.io/static/v1?label=Paper\u0026message=Huggingface\u0026color=orange'\u003e\u003c/a\u003e \n\n\u003c/div\u003e\nStoryMaker is a personalization solution preserves not only the consistency of faces but also clothing, hairstyles and bodies in the multiple characters scene, enabling the potential to make a story consisting of a series of images.\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/day1.png\"\u003e\n  Visualization of generated images by StoryMaker. First three rows tell a story about a day in the life of a \"office worker\" and the last two rows tell a story about a movie of \"Before Sunrise\".\n\u003c/p\u003e\n\n## News\n- [2024/11/09] 🔥 We release the training code.\n- [2024/09/20] 🔥 We release the [technical report](https://arxiv.org/pdf/2409.12576).\n- [2024/09/02] 🔥 We release the [model weights](https://huggingface.co/RED-AIGC/StoryMaker).\n\n## Demos\n\n### Two Portraits Synthesis\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/two.png\"\u003e\n\u003c/p\u003e\n\n### Diverse application\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/diverse.png\"\u003e\n\u003c/p\u003e\n\n## Download\n\nYou can directly download the model from [Huggingface](https://huggingface.co/RED-AIGC/StoryMaker).\n\nIf you cannot access to Huggingface, you can use [hf-mirror](https://hf-mirror.com/) to download models.\n```python\nexport HF_ENDPOINT=https://hf-mirror.com\nhuggingface-cli download --resume-download RED-AIGC/StoryMaker --local-dir checkpoints --local-dir-use-symlinks False\n```\n\nFor face encoder, you need to manually download via this [URL](https://github.com/deepinsight/insightface/issues/1896#issuecomment-1023867304) to `models/buffalo_l` as the default link is invalid. Once you have prepared all models, the folder tree should be like:\n\n```\n  .\n  ├── models\n  ├── checkpoints/mask.bin\n  ├── pipeline_sdxl_storymaker.py\n  └── README.md\n```\n\n## Usage\n\n```python\n# !pip install opencv-python transformers accelerate insightface\nimport diffusers\n\nimport cv2\nimport torch\nimport numpy as np\nfrom PIL import Image\n\nfrom insightface.app import FaceAnalysis\nfrom diffusers import UniPCMultistepScheduler\nfrom pipeline_sdxl_storymaker import StableDiffusionXLStoryMakerPipeline\n\n# prepare 'buffalo_l' under ./models\napp = FaceAnalysis(name='buffalo_l', root='./', providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])\napp.prepare(ctx_id=0, det_size=(640, 640))\n\n# prepare models under ./checkpoints\nface_adapter = f'./checkpoints/mask.bin'\nimage_encoder_path = 'laion/CLIP-ViT-H-14-laion2B-s32B-b79K'  #  from https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K\n\nbase_model = 'huaquan/YamerMIX_v11'  # from https://huggingface.co/huaquan/YamerMIX_v11\npipe = StableDiffusionXLStoryMakerPipeline.from_pretrained(\n    base_model,\n    torch_dtype=torch.float16\n)\npipe.cuda()\n\n# load adapter\npipe.load_storymaker_adapter(image_encoder_path, face_adapter, scale=0.8, lora_scale=0.8)\npipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)\n```\n\nThen, you can customized your own images\n\n```python\n# load an image and mask\nface_image = Image.open(\"examples/ldh.png\").convert('RGB')\nmask_image = Image.open(\"examples/ldh_mask.png\").convert('RGB')\n    \nface_info = app.get(cv2.cvtColor(np.array(face_image), cv2.COLOR_RGB2BGR))\nface_info = sorted(face_info, key=lambda x:(x['bbox'][2]-x['bbox'][0])*(x['bbox'][3]-x['bbox'][1]))[-1] # only use the maximum face\n\nprompt = \"a person is taking a selfie, the person is wearing a red hat, and a volcano is in the distance\"\nn_prompt = \"bad quality, NSFW, low quality, ugly, disfigured, deformed\"\n\ngenerator = torch.Generator(device='cuda').manual_seed(666)\nfor i in range(4):\n    output = pipe(\n        image=face_image, mask_image=mask_image, face_info=face_info,\n        prompt=prompt,\n        negative_prompt=n_prompt,\n        ip_adapter_scale=0.8, lora_scale=0.8,\n        num_inference_steps=25,\n        guidance_scale=7.5,\n        height=1280, width=960,\n        generator=generator,\n    ).images[0]\n    output.save(f'examples/results/ldh666_new_{i}.jpg')\n```\n\n\n## Acknowledgements\n- Our work is highly inspired by [IP-Adapter](https://github.com/tencent-ailab/IP-Adapter) and [InstantID](https://github.com/instantX-research/InstantID). Thanks for their great works!\n- Thanks [Yamer](https://civitai.com/user/Yamer) for developing [YamerMIX](https://civitai.com/models/84040?modelVersionId=309729), we use it as base model in our demo.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRedAIGC%2FStoryMaker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FRedAIGC%2FStoryMaker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRedAIGC%2FStoryMaker/lists"}