{"id":15671746,"url":"https://github.com/defgsus/clipig","last_synced_at":"2025-07-28T16:08:25.695Z","repository":{"id":51121002,"uuid":"363739928","full_name":"defgsus/clipig","owner":"defgsus","description":"OpenAI CLIP based image generator with complex config file controlled transformation and training pipelines","archived":false,"fork":false,"pushed_at":"2022-01-04T13:53:32.000Z","size":12579,"stargazers_count":19,"open_issues_count":1,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-05-30T06:26:34.900Z","etag":null,"topics":["ai","artifical-fantasy","artificial-neural-networks","clip","image-generation","openai","yaml"],"latest_commit_sha":null,"homepage":"https://defgsus.github.io/clipig/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/defgsus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-02T20:06:13.000Z","updated_at":"2025-01-28T23:59:18.000Z","dependencies_parsed_at":"2022-09-01T04:50:57.666Z","dependency_job_id":null,"html_url":"https://github.com/defgsus/clipig","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/defgsus/clipig","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defgsus%2Fclipig","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defgsus%2Fclipig/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defgsus%2Fclipig/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defgsus%2Fclipig/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/defgsus","download_url":"https://codeload.github.com/defgsus/clipig/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defgsus%2Fclipig/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267543513,"owners_count":24104589,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-28T02:00:09.689Z","response_time":68,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","artifical-fantasy","artificial-neural-networks","clip","image-generation","openai","yaml"],"created_at":"2024-10-03T15:04:49.372Z","updated_at":"2025-07-28T16:08:25.663Z","avatar_url":"https://github.com/defgsus.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# (Yet another) CLIP Image Generator\n\nDear seeker!\n\nThis is probably one of many generative gradient-descent engines using OpenAI's \n[CLIP](https://github.com/openai/CLIP/) network to rate the similarity \nbetween generated pixels and a CLIP-feature, typically given in the \nform of natural language:\n\n![kissing cthuluhu](docs/static/img/kissing-cthulhu.png)\n*\"kissing cthulhu\"*\n\nCLIPig uses no sophisticated image generation network, just ordinary RGB pixel planes.\n\nThe outstanding thing, maybe, and the reason for developing it, is it's configuration interface.\nI got pretty tired of constantly changing actual code during various experiments \nand then loosing stuff i did before so i started\nthis new project which fulfills *most* desires through yaml configuration files. \n\nPlease read the **[documentation](https://defgsus.github.io/clipig/)** \n(or as [markdown](https://defgsus.github.io/clipig/docs/README.md))\nor gather everything from this example:\n\n### example experiment\n\n```yaml\nresolution: 512\nepochs: 200\n\nlearnrate: 1.5\nlearnrate_scale: 1. - .95 * pow(t, 5.)   # 't' is training epoch in [0, 1] range\n\ntargets:\n\n  - name: random sampler\n    batch_size: 10\n\n    transforms:\n      - noise: 0.1 0.2 0.3\n      - random_scale: .1 1.\n      - random_crop: 224\n      - blur:\n          kernel_size: 13\n\n    features:\n      - text: some text feature to match\n      - text: some text feature to avoid\n        weight: -1.\n      - image: /path/to/image.png\n        loss: L2  # 'cosine' is default, 'L1' and 'L2' are also possible\n\n  - name: image adjustments\n    end: 30%\n    constraints:\n      - mean:\n          above: .1 .2 .3\n      \n# post-processing is applied after each back-propagation step\npostproc:\n  - blur: \n      kernel_size: 3\n    end: 10%    \n```\n\nAs you can see, it supports \n- [expressions](docs/expressions.md)\n- multiple [targets](docs/reference.md#targets)\n- multiple [features](docs/reference.md#targetsfeatures) per target\n- CLIP features from [texts](docs/reference.md#targetsfeaturestext) or [images](docs/reference.md#targetsfeaturesimage)\n- negative weights!\n- arbitrary pixel [transformation](docs/transforms.md) chains\n- a couple of other [constraints](docs/constraints.md) that can be added to the loss\n- a couple of image [post-processing](docs/reference.md#postproc) effects \n- scheduling via `start` and `end` parameters\n\n---\n\nCurrently, to get started switch to a virtual environment \nthat contains the `torch` library matching your CUDA drivers \nand then\n\n```bash\npython clipig.py examples/strawberries.yaml -o ./images/\n```\n\nIt will save an image to `./images/strawberries.png`. If such a file exists, \na number is attached to the filename or incremented automatically. **It will never\noverwrite an existing file.**\n\nThe **GUI** version for **interactive fun** but it's even more hacked together\nthan the rest. Install `pyqt5` and start it with:\n\n```bash\npython clipig-gui.py examples/strawberries.yaml\n```\n\nIt's possible to change the configuration during training which is funny and horrible alike. \nIt will not save snapshots by default, instead they are immediately displayed for your\npleasure and can be saved with `CTRL-S`. \n\n--- \n\nHere's a motivating [article](https://defgsus.github.io/blog/2021/04/28/malazan-clip-features.html)\nwhose images where created with some former code and \n[another one](https://defgsus.github.io/blog/2021/06/09/clipig-animal-transfer.html)\nwhere the excellent [DALL-E](https://openai.com/blog/dall-e/) examples have been\ntried to reproduce.\n\n\n## Known issues\n\nThe author does not have enough time!\n\nThen there are bugs in there and stuff that is not yet tested \nor even implemented.\n\nDocs is missing an installation section. \n\nBut the major concern is the image quality. Just adjusting pixels so that CLIP does like them\ngenerally creates horrible grids and artifacts. A good first step to avoid them is \na relatively small learning rate and random rotation like:\n\n```yaml\n- random_rotate:\n    degree: -10 10\n    center: 0 1\n```\n\nStill it's not eye-friendly to look at without a little gaussian blur. It's possible to use\na gaussian blur as a training constraint and add it to the loss function of the \nparticular target window. That leads to much better results already. And after a few hours of\nstaring at those images they get better and better anyways.\n\nThe other problem is the uniform *depth* of images. CLIP seems to be good with object \nboundaries and textures. But that does not automatically help to divide an image into \nsignificant foreground and less significant background. Generally, the image often tends\nto be a uniform something with some objects carved out. Although, there are counter \nexamples that show promise:\n\n![2d-platform with ](docs/static/img/2d-platformer.png)\n\n*\"2d platformer\"*\n\nIf you feel you can help or want to discuss things, please \n[open an issue](https://github.com/defgsus/clipig/issues).\n\n\n---\n\n![CLIP-generated strawberry image](docs/static/img/strawberry.png)\n\n\"*strawberry*\" - well, i did not make this up. things are sometimes strangely \ninterconnected in CLIP\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefgsus%2Fclipig","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdefgsus%2Fclipig","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefgsus%2Fclipig/lists"}