{"id":23069353,"url":"https://github.com/zer0int/clip-gradient-ascent-embeddings","last_synced_at":"2025-04-30T05:04:30.944Z","repository":{"id":266818452,"uuid":"899451387","full_name":"zer0int/CLIP-gradient-ascent-embeddings","owner":"zer0int","description":"Use CLIP to create matching texts + embeddings for given images; useful for XAI, adversarial training","archived":false,"fork":false,"pushed_at":"2024-12-09T02:24:03.000Z","size":5913,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"CLIP-vision","last_synced_at":"2025-04-30T05:04:21.722Z","etag":null,"topics":["adversarial-attacks","adversarial-examples","clip","contrastive-language-image-pretraining","embeddings","gradient-ascent","text-embeddings","text-image","typographic-attack"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zer0int.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-06T09:45:57.000Z","updated_at":"2025-01-21T03:18:44.000Z","dependencies_parsed_at":"2024-12-06T10:41:22.112Z","dependency_job_id":null,"html_url":"https://github.com/zer0int/CLIP-gradient-ascent-embeddings","commit_stats":null,"previous_names":["zer0int/clip-gradient-ascent-embeddings"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zer0int%2FCLIP-gradient-ascent-embeddings","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zer0int%2FCLIP-gradient-ascent-embeddings/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zer0int%2FCLIP-gradient-ascent-embeddings/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zer0int%2FCLIP-gradient-ascent-embeddings/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zer0int","download_url":"https://codeload.github.com/zer0int/CLIP-gradient-ascent-embeddings/tar.gz/refs/heads/CLIP-vision","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251644841,"owners_count":21620632,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-attacks","adversarial-examples","clip","contrastive-language-image-pretraining","embeddings","gradient-ascent","text-embeddings","text-image","typographic-attack"],"created_at":"2024-12-16T06:14:49.132Z","updated_at":"2025-04-30T05:04:30.927Z","avatar_url":"https://github.com/zer0int.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"### 🚀🆙 CLIP-gradient-ascent-embeddings\n\n- ❗ Requires [OpenAI/CLIP](https://github.com/openai/CLIP)\n- Generates matching text embeddings / a 'CLIP opinion' about images\n- Uses gradient ascent to optimize text embeds for cosine similarity with image embeds\n- Saves 'CLIP opinion' as .txt files [best tokens]\n- Saves text-embeds.pt with [batch_size] number of embeds\n- Can be used to create an adversarial text-image aligned dataset\n- For XAI, adversarial training, etc; see 'attack' folder for example images\n- Usage: Single image: `python gradient-ascent.py --use_image attack/024_attack.png`\n- Usage: Batch process: `python gradient-ascent.py --img_folder attack`\n- 🆕 Load custom model: `python gradient-ascent-unproj_flux1.py --model_name \"path/to/myCLIP.safetensors\"`\n-----\n## Changes 07/DEC/2024\n\n- Args `--model_name` now accepts name (default `ViT-L/14`), *OR* a `\"/path/to/model.pt\"`\n- If it ends on `.safetensors`, will assume 'ViT-L/14' (CLIP-L) and load state_dict. ✅\n- ⚠️ Must be nevertheless in original \"OpenAI/CLIP\" format. HuggingFace converted models will NOT work.\n- My [HF: zer0int/CLIP-GmP-ViT-L-14](https://huggingface.co/zer0int/CLIP-GmP-ViT-L-14/tree/main) `model.safetensors` will NOT work (it's for diffusers / HF).\n- Instead, download the full model .safetensors [text encoder AND vision encoder]; direkt link:\n- My [GmP-BEST-smooth](https://huggingface.co/zer0int/CLIP-GmP-ViT-L-14/resolve/main/ViT-L-14-BEST-smooth-GmP-HF-format.safetensors?download=true) and [GmP-Text-detail](https://huggingface.co/zer0int/CLIP-GmP-ViT-L-14/resolve/main/ViT-L-14-TEXT-detail-improved-hiT-GmP-HF.safetensors?download=true) and 🆕 [SAE-GmP](https://huggingface.co/zer0int/CLIP-SAE-ViT-L-14/resolve/main/ViT-L-14-GmP-SAE-FULL-model.safetensors?download=true) will work with this code.\n----\n- 🆕 Added `gradient-ascent-unproj_flux1.py`. Usage is the same; however, in addition to projected embeddings:\n- Saves `pinv` and `inv` version of pre-projection embeddings.\n- 👉 `Flux.1-dev` uses these embeddings (`pinv` seems best for Flux.1-dev).\n- Recommended samplers: _HEUN_, Euler.\n\nExample \"worst portrait ever\" generated by Flux.1-dev with pure CLIP guidance (no T5!) as CLIP apparently tried to encode the facial expression of the cat 😂; plus, the usual CLIP text gibberish of something 'cat' and 'shoe' mashed-up:\n\n\n![worst](https://github.com/user-attachments/assets/8523f4bc-32f5-42f2-9854-faa1db0f30f8)\n\n\n-----\n![gradient-ascent](https://github.com/user-attachments/assets/386645d8-5ed1-4799-9511-4ebe9746241c)\n-----\n\nCommand-line arguments:\n\n```\n--batch_size, default=13, type=int, help=\"Reduce batch_size if you have OOM issues\"\n--model_name, default='ViT-L/14', help=\"CLIP model to use\"\n--tokens_to, default=\"texts\", help=\"Save CLIP opinion texts path\"\n--embeds_to, default=\"embeds\", help=\"Save CLIP embeddings path\"\n--use_best, default=\"True\", help=\"If True, use best embeds (loss); if False, just saves last step (not recommended)\"\n--img_folder, default=None, help=\"Path to folder with images, for batch embeddings generation\"\n--use_image, default=None, help=\"Path to a single image\"\n```\n\n\nFurther processing example code snippets:\n\n```\ntext_embeddings = torch.load(\"path/to/embeds.pt\").to(device)\n\n# loop over all batches of embeds and do a thing\nnum_embeddings = text_embeddings.size(0) # e.g. batch_size 13 -\u003e idx 0 to 12\nfor selected_embedding_idx in range(num_embeddings):\n    print(f\"Processing embedding index: {selected_embedding_idx}\")\n    # do your thing here!\n\n\n# select a random batch from embedding and do a thing\nselected_embedding_idx = torch.randint(0, text_embeddings.size(0), (1,)).item()\nselected_embedding = text_embeddings[selected_embedding_idx:selected_embedding_idx + 1]\n\n# or just manually select one\nselected_embedding_idx = 3\nselected_embedding = text_embeddings[selected_embedding_idx:selected_embedding_idx + 1]\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzer0int%2Fclip-gradient-ascent-embeddings","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzer0int%2Fclip-gradient-ascent-embeddings","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzer0int%2Fclip-gradient-ascent-embeddings/lists"}