{"id":14043876,"url":"https://github.com/ebagdasa/adversarial_illusions","last_synced_at":"2025-07-27T15:31:50.233Z","repository":{"id":190394907,"uuid":"681410547","full_name":"ebagdasa/adversarial_illusions","owner":"ebagdasa","description":"Code for \"Adversarial Illusions in Multi-Modal Embeddings\"","archived":false,"fork":false,"pushed_at":"2024-08-04T19:41:54.000Z","size":25013,"stargazers_count":4,"open_issues_count":1,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-08-12T08:13:04.444Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2308.11804","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ebagdasa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-22T00:50:36.000Z","updated_at":"2024-08-04T19:37:56.000Z","dependencies_parsed_at":null,"dependency_job_id":"cb844a27-9402-4e24-98fd-cc58ee15e54f","html_url":"https://github.com/ebagdasa/adversarial_illusions","commit_stats":null,"previous_names":["ebagdasa/adversarial_illusions"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebagdasa%2Fadversarial_illusions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebagdasa%2Fadversarial_illusions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebagdasa%2Fadversarial_illusions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ebagdasa%2Fadversarial_illusions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ebagdasa","download_url":"https://codeload.github.com/ebagdasa/adversarial_illusions/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227814471,"owners_count":17823909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-12T08:06:36.579Z","updated_at":"2024-12-02T22:31:46.890Z","avatar_url":"https://github.com/ebagdasa.png","language":"Jupyter Notebook","funding_links":[],"categories":["Adversarial-Attack"],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e \nAdversarial Illusions in Multi-Modal Embeddings \u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e \u003ci\u003eTingwei Zhang*, Rishi Jha*, Eugene Bagdasaryan, and Vitaly Shmatikov\u003c/i\u003e\u003c/p\u003e\n\nMulti-modal embeddings encode texts, images, sounds, videos, etc., into a single embedding space, aligning representations across different modalities (e.g., associate an image of a dog with a barking sound). In this paper, we show that multi-modal embeddings can be vulnerable to an attack we call \"adversarial illusions.\" Given an image or a sound, an adversary can perturb it to make its embedding close to an arbitrary, adversary-chosen input in another modality.\n\nThese attacks are cross-modal and targeted: the adversary is free to align any image and any sound with any target of his choice. Adversarial illusions exploit proximity in the embedding space and are thus agnostic to downstream tasks and modalities, enabling a wholesale compromise of current and future downstream tasks and modalities not available to the adversary. Using ImageBind and AudioCLIP embeddings, we demonstrate how adversarially aligned inputs, generated without knowledge of specific downstream tasks, mislead image generation, text generation, zero-shot classification, and audio retrieval.\n\nWe investigate transferability of illusions across different embeddings and develop a black-box version of our method that we use to demonstrate the first adversarial alignment attack on Amazon's commercial, proprietary Titan embedding. Finally, we analyze countermeasures and evasion attacks.\n\nPaper link:\n[https://arxiv.org/abs/2308.11804](https://arxiv.org/abs/2308.11804)\n\n\u003cimg src=\"image/illusion.png\" alt=\"drawing\" width=\"600\"/\u003e\n\nMost experiments run on a single NVIDIA 2080ti GPU.\n\n# Installation\n1. **Setup Environment:** run `conda env create -f environment.yml`.\n2. **Download Data:** We run experiments on ImageNet [1], AudioSet [2], and LLVIP [3]. For ease of reproduction, we provide necessary config files for all datasets and 100-example subsets of the latter two datasets as a [release](https://github.com/rjha18/adversarial_illusions/releases). To install, please download the `data.zip` file and, from root, run `unzip /path/to/data.zip -d .`.\n    - For imagenet, we only use the validation set. As required by PyTorch, we also require `ILSVRC2012_devkit_t12.tar.gz` and `ILSVRC2012_img_val.tar` to be located in `data/imagenet/`. Please follow the instructions in [the note on PyTorch's page](https://pytorch.org/vision/main/generated/torchvision.datasets.ImageNet.html) to acquire those two files.\n3. **AudioCLIP Checkpoints:** To conduct any experiments on AudioCLIP, we require pretraining checkpoints.\n    - For the full checkpoint, run:\n      ```\n      wget https://github.com/AndreyGuzhov/AudioCLIP/releases/download/v0.1/AudioCLIP-Full-Training.pt -P bpe/\n      ```\n    - For the partial checkpoint (used for transfer attacks):\n      ```\n      wget https://github.com/AndreyGuzhov/AudioCLIP/releases/download/v0.1/AudioCLIP-Partial-Training.pt -P bpe/\n      ```\n4. **Submodule Setup:** This includes lightly adapted code from [ImageBind](https://github.com/facebookresearch/ImageBind), [AudioCLIP](https://github.com/AndreyGuzhov/AudioCLIP), and [DiffJPEG](https://github.com/mlomnitz/DiffJPEG/) and directly employs two submodules: [PandaGPT](https://github.com/yxuansu/PandaGPT) and [BindDiffusion](https://github.com/sail-sg/BindDiffusion). To initialize the two submodules (if desired), run the following and download the checkpoints as described below:\n    ```\n    git submodule update --init\n    scp image_text_generation/image_generation.py BindDiffusion\n    scp image_text_generation/text_generation_demo.ipynb PandaGPT/code\n    scp image_text_generation/text_generation.py PandaGPT/code\n    ```\n    - **PandaGPT Checkpoints:** To conduct any experiments with PandaGPT, place the [PandaGPT checkpoints](https://github.com/yxuansu/PandaGPT#2-running-pandagpt-demo-back-to-top) into `PandaGPT/pretrained_ckpt` by following [these instructions](PandaGPT/pretrained_ckpt/README.md).\n    - **BindDiffusion Checkpoints:** To conduct any experiments with BindDiffusion, place the [BindDiffusion checkpoints](https://github.com/sail-sg/BindDiffusion) into `BindDiffusion/checkpoints` by following [these instructions](BindDiffusion/README.md).\n\n\n# Image Illusion Demo on Text Generation\n1. Run the `image_illusion_demo.ipynb` notebook.\n2. Replace the existing image and aligned text with your own choices to generate an image illusion.\n3. Run `text_generation_demo.ipynb` to see a quick demonstration of image illusions comprising text generation task.\n\n# Experiments\n\n## White Box\nOur white box experiments (save for thermal and baseline experiments) are run using the `adversarial_illusions.py` script and configured by a `{EXPERIMENT_NAME}.toml` file in the `configs/` folder. An explanation of each of the hyperparameters can be found here: `configs/explanations/illusions.toml`. Some examples are provided in the rest of the `configs/` folder:\n\n- **Image Classification:** \n  - `python adversarial_illusions.py imagenet/whitebox/{MODEL_NAME}.toml`\n- **Audio Classification:**\n  - `python adversarial_illusions.py audiocaps/whitebox/{MODEL_NAME}.toml`\n- **Audio Retrieval:**\n  - `python adversarial_illusions.py audioset/whitebox/{MODEL_NAME}.toml`\n- **Thermal Image Classification:**\n  - `python thermal_illusion_classification.py \u003e outputs/thermal/result.txt` \n\nOur baseline numbers are run using the `evaluate_illusions.py` file and configured in the `configs/baseline/` directory. The `.toml` files have a slightly different structure, and descriptions can be found here: `configs/explanations/baseline.toml`. The two white-box baselines as described in the paper can be run as follows:\n  - `python evaluate_illusions.py baseline/organic/{TASK_NAME}.toml`\n  - `python evaluate_illusions.py baseline/adversarial/{TASK_NAME}.toml`\n\n## Black Box\n  - **Run Transfer Attack Experiments:** Our transfer numbers are produced similarly to our baselines, but require an additional flag `adv_file`. This parameter should point to a `.npy` file containing the adversarial images to evaluate. See `configs/explanations/transfer.toml` for a description. An example:\n    - `python adversarial_illusions.py imagenet/transfer/ensemble.toml`\n    - `python evaluate_illusions.py imagenet/transfer/ensemble_eval.toml`\n\n  - **Run Query-based Attack Experiments:**\n     - `python query_attack.py imagebind`\n     - `python query_attack.py audioclip`\n\n  \u003c!-- - **Run Hybrid Attack Experiments:**\n     - `python query_attack.py imagenet/hybrid/imagebind`\n     - `python query_attack.py imagenet/hybrid/audioclip` --\u003e\n\n## Defense\n  - **Certification:** `certification.ipynb`\n  - **Feature Distillation:** \n  \n    ```\n    python adversarial_illusions.py imagenet/whitebox/imagebind\n    python adversarial_illusions.py imagenet/whitebox/audioclip\n    python adversarial_illusions.py imagenet/whitebox/imagebind_jpeg\n    python adversarial_illusions.py imagenet/whitebox/audioclip_jpeg\n    python evaluate_jpeg.py\n    ```\n  - **Anomaly Detection:** `python anomaly_detection.py`\n\n\nPlease feel free to email: [tz362@cornell.edu](mailto:tz362@cornell.edu) or raise an issue.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Febagdasa%2Fadversarial_illusions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Febagdasa%2Fadversarial_illusions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Febagdasa%2Fadversarial_illusions/lists"}