{"id":20715671,"url":"https://github.com/ovshake/replicate-img2img","last_synced_at":"2025-04-23T10:44:43.622Z","repository":{"id":184873148,"uuid":"672067203","full_name":"ovshake/replicate-img2img","owner":"ovshake","description":"Van Gogh style diffusion model on replicate","archived":false,"fork":false,"pushed_at":"2023-07-30T17:43:29.000Z","size":514,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-29T23:22:49.641Z","etag":null,"topics":["computer-vision","diffusers","pytorch","replicate-api","stable-diffusion"],"latest_commit_sha":null,"homepage":"https://replicate.com/ovshake/replicate-ghibli-vangogh","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ovshake.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-07-28T20:47:20.000Z","updated_at":"2023-10-23T06:09:43.000Z","dependencies_parsed_at":null,"dependency_job_id":"5c100829-5fbf-49c2-95f5-ed52d649b100","html_url":"https://github.com/ovshake/replicate-img2img","commit_stats":null,"previous_names":["ovshake/replicate-img2img"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovshake%2Freplicate-img2img","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovshake%2Freplicate-img2img/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovshake%2Freplicate-img2img/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovshake%2Freplicate-img2img/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ovshake","download_url":"https://codeload.github.com/ovshake/replicate-img2img/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250419417,"owners_count":21427596,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","diffusers","pytorch","replicate-api","stable-diffusion"],"created_at":"2024-11-17T02:39:29.930Z","updated_at":"2025-04-23T10:44:43.600Z","avatar_url":"https://github.com/ovshake.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Image2VanGogh\n\n[![Replicate](https://replicate.com/stability-ai/stable-diffusion/badge)](https://replicate.com/ovshake/replicate-ghibli-vangogh)\n\n\u003e The code is running on CPU, so it will be extremly slow. It takes ~7 minutes for 50 inference steps on replicate.com. You can use GPU to speed up the process. The script supports both.\n\nThis Python script allows you to generate images using the Ghibli-Diffusion model trained with PyTorch. The generated images are based on an input image and a text prompt, and the output is a transformed version of the initial image guided by the provided text prompt.\n\n\n## Image Transformation Example\n\nHere is an example of how this model transforms an input image based on a text prompt.\n\nThe original image:\n\n![Original Image](assets/sketch-mountains-input.jpg)\n\nThe transformed image with the prompt \"A van gogh painting of a starry night\":\n\n![Transformed Image](assets/output.png)\n\nAs you can see, the transformed image retains the basic structure of the original image but modifies the colors and details to match the style suggested by the text prompt.\n## Installation\n\nEnsure that you have Python 3.11 installed. You can download it from the official Python website if necessary.\n\nNext, you need to install the necessary dependencies. Clone the repository and navigate to its directory. Then, use pip to install the requirements from the provided `requirements.txt` file:\n\n```bash\ngit clone https://github.com/ovshake/replicate-img2img\ncd replicate-img2img\npip install -r requirements.txt\n```\n\n## Usage\n\nOnce you have installed all the necessary dependencies, you can run the script using the following command:\n\n```bash\npython generate_image.py\n```\n\nThe script requires an image file and a text prompt as input. The image file can either be a local file or a URL. If you use a URL, the script will download the image before processing it.\n\nThe `initialize_pipeline` function initializes the Ghibli-Diffusion model pipeline from a pretrained model. You can specify the model and the device (CPU or GPU) to use.\n\nThe `generate_image` function takes in the initialized pipeline, the image file path or URL, and the text prompt. It generates a new image based on the input image and the text prompt. You can optionally specify the strength, guidance scale, and seed for the generation process.\n\nThe main section of the script initializes the pipeline and generates a new image using an example input image and text prompt. The generated image is saved as a `.png` file.\n\n## Note\n\nThis script assumes that you have a GPU available for use. If you do not have a GPU, you can modify the `device` parameter to `\"cpu\"` in both the `initialize_pipeline` and `generate_image` function calls.\n\n## Example\n\nThis is an example of how you can use the script to generate a new image:\n\n```python\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\npipe = initialize_pipeline(pipeline_model=\"nitrosocke/Ghibli-Diffusion\", device=device)\nimage = generate_image(pipe, device=device, url_or_filepath=\"assets/sketch-mountains-input.jpg\", prompt=\"A van gogh painting of a starry night.\")\nimage.save(\"assets/output.png\")\n```\n\nThis code generates a new image based on the `sketch-mountains-input.jpg` image and the text prompt \"A van gogh painting of a starry night\". The generated image is saved as `output.png` in the `assets` directory.\n\n## Run on Replicate\n\nThis model is also available on replicate.com. You can access the model and run predictions using the web interface. You do not need to install any dependencies or write any code to use the model on replicate.com. Simply upload your image, enter your text prompt, and run the model to generate a new image.\n\nAccess the model on replicate.com at the following URL: [https://replicate.com/ovshake/replicate-ghibli-vangogh](https://replicate.com/ovshake/replicate-ghibli-vangogh)\n\nTo build this model, you can use `cog.yaml` config file. You can use replicate to run prediction using\n```bash\ncog predict -i image=@assets/sketch-mountains-input.jpg -i prompt=\"A van gogh painting\"\n```\nYou can also use `curl` to use the endpoint as an API.\n```bash\n(sd) ➜  replicate-img2img git:(main) curl -s -X POST \\\n  -d '{\"version\": \"d991ed58d1254817879017e676da96d1c2c99976cac210395d14b17e1fb2b496\", \"input\": {\"image\": \"https://raw.githubusercontent.com/open-mmlab/mmdetection/main/resources/mmdet-logo.png\", \"prompt\": \"A van gogh style painting\"}}' \\\n  -H \"Authorization: Token $REPLICATE_API_TOKEN\" \\\n  -H 'Content-Type: application/json' \\\n  \"https://api.replicate.com/v1/predictions\" | jq\n\n```\nand the output will look like this\n```json\n{\n  \"id\": \"ovbandjba7tcg2ffgi3hk2p2lm\",\n  \"version\": \"d991ed58d1254817879017e676da96d1c2c99976cac210395d14b17e1fb2b496\",\n  \"input\": {\n    \"image\": \"https://raw.githubusercontent.com/open-mmlab/mmdetection/main/resources/mmdet-logo.png\",\n    \"prompt\": \"A van gogh style painting\"\n  },\n  \"logs\": \"\",\n  \"error\": null,\n  \"status\": \"starting\",\n  \"created_at\": \"2023-07-30T15:59:23.763416844Z\",\n  \"urls\": {\n    \"cancel\": \"https://api.replicate.com/v1/predictions/ovbandjba7tcg2ffgi3hk2p2lm/cancel\",\n    \"get\": \"https://api.replicate.com/v1/predictions/ovbandjba7tcg2ffgi3hk2p2lm\"\n  }\n}\n```\nSince this takes a long time, you need to keep polling to check when it completes. More details can be found [here](https://replicate.com/ovshake/replicate-ghibli-vangogh/api) on polling.\nDocs on cog can be found [here](https://github.com/replicate/cog).\n\n## Things to Improve\n\n1. We can further finetune the prompt and play around with hyperparameteres to get better results.\n2. We can use other custom checkpoints such as [RealisticVision](https://civitai.com/models/91) to enhance our results.\n3. We can add a super resolution model to enhance the image quality.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovshake%2Freplicate-img2img","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fovshake%2Freplicate-img2img","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovshake%2Freplicate-img2img/lists"}