{"id":13456964,"url":"https://github.com/victordibia/peacasso","last_synced_at":"2025-04-04T18:06:59.486Z","repository":{"id":58600673,"uuid":"530072220","full_name":"victordibia/peacasso","owner":"victordibia","description":"UI interface for experimenting with multimodal (text, image) models (stable diffusion).","archived":false,"fork":false,"pushed_at":"2023-08-16T01:12:55.000Z","size":105561,"stargazers_count":368,"open_issues_count":11,"forks_count":41,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-28T17:08:37.420Z","etag":null,"topics":["deeplearning","machinelearning","stable-diffusion","stablediffusion"],"latest_commit_sha":null,"homepage":"https://victordibia.com/peacasso/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/victordibia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-29T05:01:57.000Z","updated_at":"2025-03-12T23:15:02.000Z","dependencies_parsed_at":"2024-07-31T08:25:02.195Z","dependency_job_id":null,"html_url":"https://github.com/victordibia/peacasso","commit_stats":{"total_commits":104,"total_committers":1,"mean_commits":104.0,"dds":0.0,"last_synced_commit":"9c876bd093126046093bcbc61dacf178ba2d4d3c"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victordibia%2Fpeacasso","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victordibia%2Fpeacasso/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victordibia%2Fpeacasso/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victordibia%2Fpeacasso/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/victordibia","download_url":"https://codeload.github.com/victordibia/peacasso/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247226213,"owners_count":20904465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deeplearning","machinelearning","stable-diffusion","stablediffusion"],"created_at":"2024-07-31T08:01:30.948Z","updated_at":"2025-04-04T18:06:59.467Z","avatar_url":"https://github.com/victordibia.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook"],"sub_categories":[],"readme":"# Peacasso\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/victordibia/peacasso/blob/master/notebooks/tutorial.ipynb)\n\nPeacasso [Beta] is a UI tool to help you generate art (and experiment) with multimodal (text, image) AI models (stable diffusion). This project is still in development (see roadmap below).\n\n![](https://github.com/victordibia/peacasso/blob/master/docs/images/screenpc.png?raw=true)\n\n## Why Use Peacasso?\n\n\u003cimg width=\"100%\" src=\"https://github.com/victordibia/peacasso/blob/master/docs/images/peacasso.gif?raw=true\" /\u003e\n\nBecause you deserve a nice UI and great workflow that makes exploring stable diffusion models fun! But seriously, here are a few things that make Peacasson interesting:\n\n- **Easy installation**. Instead of cobbling together command line scripts, Peacasso provides a `pip install` flow and a UI that supports a set of curated default operations.\n- **UI with good defaults**. The current implementation of Peacasso provides a UI for basic operations - text and image based prompting (+ inpainting), remixing generated images as prompts, model parameter selection, image download. Also covers the little things .. like light and dark mode.\n- **Python API**. While the UI is the focus here, there is an underlying python api which will bake in experimentation features (e.g. saving intermediate images in the sampling loop, exploring model explanations etc. . see roadmap below).\n\nClearly, Peacasso (UI) might not be for those interested in low level code.\n\n## Requirements and Installation\n\n- Step 1: **Verify Environment - Pythong 3.7+ and CUDA**\n  Setup and verify that your python environment is `python 3.7` or higher (preferably, use Conda). Also verify that you have CUDA installed correctly (`torch.cuda.is_available()` is true) and your GPU has about [7GB of VRAM memory](https://stability.ai/blog/stable-diffusion-public-release).\n\nOnce requirements are met, run the following command to install the library:\n\n```bash\npip install peacasso\n```\n\nWant to stay on the bleeding edge of updates (which might be buggy)? Install directly from the repo:\n\n```bash\ngit clone https://github.com/victordibia/peacasso.git\ncd  peacasso\npip install -e .\n```\n\nDon't have a GPU, you can still use the python api and UI in a colab notebook. See this [colab notebook](https://colab.research.google.com/github/victordibia/peacasso/blob/master/notebooks/tutorial.ipynb) for more details.\n\n## Usage - UI and Python API\n\nYou can use the library from the ui by running the following command:\n\n```bash\npeacasso ui  --port=8080\n```\n\nThen navigate to http://localhost:8080/ in your browser. Note that you can also pass flags like `--model` (huggingface model) and `--device` (cuda, cpu, mps) to the command above to specify the model and device to use.\n\nYou can also use the python api by running the following command.\nNote that each generation is parameterized by a [`GeneratorConfig`](peacasso/datamodel.py) object.\n\n```python\n\nfrom peacasso.generator import ImageGenerator\nfrom peacasso.datamodel import GeneratorConfig, ModelConfig\n\n# model configuration\nmodel_config: ModelConfig = ModelConfig(\n    device=\"cuda:0\" , # device ..cpu, cuda, cuda:0\n    model=\"nitrosocke/mo-di-diffusion\",\n    revision=\"main\", # HF model branch\n    token=None, # HF_TOKEN here if needed\n)\n\nprompt = \"victorian ampitheater of sand, pillars with statues on top, lamps on ground, by peter mohrbacher dan mumford craig mullins nekro, cgsociety, pixiv, volumetric light, 3 d render\"\nprompt_config = GeneratorConfig(\n    prompt=prompt,\n    num_images=3,\n    width=512,\n    height=512,\n    guidance_scale=7.5,\n    num_inference_steps=20,\n    return_intermediates=True, # return intermediate images during diffusion sampling\n    seed=6010691039\n)\nresult = gen.generate(prompt_config)\nresult = gen.generate(prompt_config)\nfor i, image in enumerate(result[\"images\"]):\n    image.save(f\"image_{i}.png\")\n\n# result[\"intermediates\"] contains the intermediate images\n```\n\n![](https://github.com/victordibia/peacasso/blob/master/docs/images/prompt_result.png?raw=true)\n\nVisualizing intermediate images during the diffusion loop.\n![](https://github.com/victordibia/peacasso/blob/master/docs/images/intermediates.png?raw=true)\n\nYou can also interpolate between along several dimensions e.g., `latents` used in the diffusion process (based on seed, image, mask or combinations) and `text embeddings` used to guide the diffusion process. You can do this by modifying the `application` field in `GeneratorConfig`. See the [colab notebook](https://colab.research.google.com/github/victordibia/peacasso/blob/master/notebooks/interpolation.ipynb) for more details.\n\nThe config below shows how to interpolate between two prompts and two latents (based on provided start and end images).\n\n```python\ninterpolation_config = GeneratorConfig(\n    prompt=start_prompt,\n    num_inference_steps=40,\n    seed=3167288399,\n    strength=1,\n    application={\n            \"type\": \"interpolate\",\n            \"config\": {\"num_steps\": 60,\n            \"image\": {\"start\": start_image, \"end\":end_image} ,\n            \"prompt\": {\"start\": start_prompt, \"end\": end_prompt}\n            }\n})\n```\n\n![](https://github.com/victordibia/peacasso/blob/master/docs/images/image_interpolation.png?raw=true)\n\n## Design Philosophy\n\nFeatures in `Peacasso` are being designed based on insights from communication theory [^1] and also research on Human-AI interaction design [^2]. Learn more about the design and components in peacasso in the paper draft [here](https://github.com/victordibia/peacasso/blob/master/docs/images/paper.pdf).\n\n\u003cimg width=\"100%\" src=\"https://github.com/victordibia/peacasso/blob/master/docs/images/mrt.png?raw=true\" /\u003e\n\nA general vision for the Peacasso architecture is shown below (parts of this are still being implemented):\n\n\u003cimg width=\"100%\" src=\"https://github.com/victordibia/peacasso/blob/master/docs/images/peacasso_arch.png?raw=true\" /\u003e\n\n## Features and Road Map\n\n- [x] Command line interface\n- [x] UI Features. Query models with multiple parametrs\n  - [x] Prompting: Text prompting (text2img), Image based prompting (img2img), Inpainting (img2img)\n  - [x] Latent space interpolation (see [notebook](notebooks/interpolation.ipynb)).\n  - [ ] Full Editor (for outpainting) (see tutorial [notebook](https://github.com/victordibia/peacasso/blob/master/notebooks/tutorial.ipynb) )\n- [ ] Experimentation tools\n  - [x] Save intermediate images in the sampling loop\n  - [x] Weighted prompt mixing\n  - [ ] Prompt recommendation\n  - [ ] Curation/sharing experiment results\n  - [ ] Defined Workflows (e.g., tiles, composition etc.)\n  - [ ] Model explanations\n\n## Acknowledgement\n\nThis work builds on the stable diffusion model and code is adapted from the HuggingFace [implementation](https://huggingface.co/blog/stable_diffusion). Please note the - [CreativeML Open RAIL-M](https://huggingface.co/spaces/CompVis/stable-diffusion-license) license associated with the stable diffusion model.\n\n## Citation\n\nIf you use `peacasso` for in your research or adopt the design guidelines used to build `peacasso`, please consider citing the [paper](https://github.com/victordibia/peacasso/blob/master/docs/images/paper.pdf) as follows:\n\n```bibtex\n@misc{dibia2022peacasso,\n      title={Interaction Design for Systems that Integrate Image Generation Models: A Case Study with Peacasso},\n      author={Victor Dibia},\n      year={2022},\n      publisher={GitHub},\n      journal={GitHub repository},\n      primaryClass={cs.CV}\n}\n```\n\n## References\n\n[^1]:\n    Richard L Daft and Robert H Lengel. 1986. Organizational information require-\n    ments, media richness and structural design. Management science\n\n[^2]:\n    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi,\n    Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019.\n    Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on\n    human factors in computing systems.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictordibia%2Fpeacasso","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvictordibia%2Fpeacasso","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictordibia%2Fpeacasso/lists"}