{"id":27957642,"url":"https://github.com/laion-ai/conditioned-prior","last_synced_at":"2025-05-07T18:13:56.875Z","repository":{"id":44330262,"uuid":"510680603","full_name":"LAION-AI/conditioned-prior","owner":"LAION-AI","description":"(wip) Use LAION-AI's CLIP \"conditoned prior\" to generate CLIP image embeds from CLIP text embeds.","archived":false,"fork":false,"pushed_at":"2022-07-14T15:55:12.000Z","size":27,"stargazers_count":27,"open_issues_count":0,"forks_count":2,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-05-07T18:13:52.744Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LAION-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-05T10:02:17.000Z","updated_at":"2024-12-04T07:41:49.000Z","dependencies_parsed_at":"2022-09-05T09:31:51.268Z","dependency_job_id":null,"html_url":"https://github.com/LAION-AI/conditioned-prior","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LAION-AI%2Fconditioned-prior","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LAION-AI%2Fconditioned-prior/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LAION-AI%2Fconditioned-prior/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LAION-AI%2Fconditioned-prior/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LAION-AI","download_url":"https://codeload.github.com/LAION-AI/conditioned-prior/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252931550,"owners_count":21827112,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T18:13:56.360Z","updated_at":"2025-05-07T18:13:56.864Z","avatar_url":"https://github.com/LAION-AI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Conditioned Prior (WIP)\n\n\u003ca href=\"https://replicate.com/laion-ai/conditioned-prior\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Replicate\u0026message=Demo and API\u0026color=pink\"\u003e\u003c/a\u003e\n\nWeights and code by [@nousr](https://twitter.com/nousr_)\n\n\nPredict a CLIP image embedding from its text embedding using a diffusion prior.\n\nThis code is part of an effort to replicate the models laid out in [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://arxiv.org/abs/2204.06125).\n\n## Requirements\n\n* nvidia GPU\n* [docker](https://docs.docker.com/get-docker/)\n* [cog](https://github.com/replicate/cog/)\n\n## Quick start\n\n```sh\ncog predict r8.im/laion-ai/conditioned-prior \\\n    -i prompt=\"...\" \\\n    -i candidates=2 \\\n    -i cond_scale=1.0 \\\n    -i overwrite=False\n```\n\n### Parameters\n\n* `prompt` - Text to invert to a CLIP image embed (Required)\n\n* `cond_scale` - How much prior guidance to use. (Default 1.0)\n\n* `candidates` - Number of image embeds to draw from in the prior. Increasing may improve performance.  (Default: 2)\n\n* `overwrite` - Recomputes all embeds from scratch, even if they already exist on local storage. (Default: False)\n\n### Output\n\nA `PriorOutput` - typed dictionary containing the text tokens, text embed and the image embed.\n\n* `text_embedding: List[float]` - CLIP embed of your text, included as convenience for cosine similarity.\n\n* `image_embedding: List[float]` - CLIP image embedding inverted from the text embedding using the conditoned-prior model.\n\nOutputs are also stored as numpy arrays in the current directory at `f\"./.embed_cache/text_embedding_{prompt}.npy\"`\nand `f\"./.embed_cache/image_embedding_{prompt}.npy\"`\n\n## Intended use\n\nAnytime you need a CLIP image embed but only have a text description. For instance:\n\n* Use as input to models that accept CLIP embeds such as CLIP-guided VQGAN, diffusion to improve generations.\n\n* Use to improve performance on lookup tasks\n\n## Special Thanks\n\n* [LAION](https://discord.gg/uPMftTmrvS) for support, resources, and community\n\n* [Stability AI](https://stability.ai/) for compute which makes these models possible\n\n* [lucidrains](https://github.com/lucidrains) for spearheading the open-source replication of DALLE 2\n\n## Caveats and recommendations\n\nJust to avoid any confusion, this research is a recreation of (one part of) OpenAI's DALLE2 paper. It is _not_, \"DALLE2\", the product/service from OpenAI you may have seen on the web.\n\n## Contribute\n\n* Install [docker](https://docs.docker.com/get-docker/).\n* Install [cog](https://github.com/replicate/cog/).\n\n```sh\ngit clone https://github.com/laion-ai/conditoned-prior.git \u0026\u0026 cd conditioned-prior\n```\n\n### Build the docker image from scratch\n\nDownload the \"slim\" weights:\n\n```sh\nwget https://huggingface.co/laion/DALLE2-PyTorch/resolve/main/prior_ema_fp16.pth\n```\n\nThen, run:\n\n```sh\ncog build -t \"my-custom-conditioned-prior\"\n```\n\n### Local prediction flask endpoint\n\n```sh\ndocker run -d -p 5000:5000 --gpus=all 'my-custom-conditioned-prior'\n```\n\nA `POST` route `/predictions` will now trigger the model to be run. Weights are only loaded into GPU memory once upon running `docker run`, making repeated API calls faster.\n\n```sh\ncurl http://localhost:5000/predictions -X POST -H \"Content-Type: application/json\" \\\n  -d '{\"input\": {\n    \"prompt\": \"...\",\n    \"candidates\": \"2\",\n    \"cond_scale\": \"1.0\"\n  }}'\n```\n\n### Push a fork to your own Replicate account\n\nFirst, edit the `image` property in [cog.yaml](/cog.yaml)\n\n```yaml\n# ...\nimage: \"\" # TODO put your own url here after creating a model on Replicate.\nbuild:\n  gpu: true\n  python_version: \"3.8\"\n# ...\n```\n\nMake sure you are logged in:\n\n```sh\ncog login\n```\n\nand push your docker image to Replicate:\n\n```sh\ncog push\n```\n\n### Update the official laion-ai Replicate demo/api\nIf you need to change the Replicate demo uploaded to `replicate.com/laion-ai/conditioned-prior`, you will need to be invited to be part of the laion-ai org on Replicate. Reach out to @afiaka87, @robvanvolt, @christophschuhmann, or @rom1504 if you need to.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaion-ai%2Fconditioned-prior","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flaion-ai%2Fconditioned-prior","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaion-ai%2Fconditioned-prior/lists"}