{"id":14043873,"url":"https://github.com/yunqing-me/AttackVLM","last_synced_at":"2025-07-27T15:31:50.119Z","repository":{"id":170339912,"uuid":"645725108","full_name":"yunqing-me/AttackVLM","owner":"yunqing-me","description":"[NeurIPS-2023] Annual Conference on Neural Information Processing Systems","archived":false,"fork":false,"pushed_at":"2023-10-30T05:31:45.000Z","size":24917,"stargazers_count":143,"open_issues_count":6,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-08-12T08:13:04.161Z","etag":null,"topics":["adversarial-attack","deep-generative-model","foundation-models","generative-ai","image-to-text-generation","large-language-models","text-to-image-generation","trustworthy-ai","vision-language-model"],"latest_commit_sha":null,"homepage":"https://arxiv.org/pdf/2305.16934.pdf","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yunqing-me.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-05-26T09:44:08.000Z","updated_at":"2024-08-06T02:53:11.000Z","dependencies_parsed_at":"2023-10-26T11:34:12.196Z","dependency_job_id":"deb1a5e0-6eae-428c-a66a-7f799feb68ea","html_url":"https://github.com/yunqing-me/AttackVLM","commit_stats":null,"previous_names":["yunqing-me/attackvlm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yunqing-me%2FAttackVLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yunqing-me%2FAttackVLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yunqing-me%2FAttackVLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yunqing-me%2FAttackVLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yunqing-me","download_url":"https://codeload.github.com/yunqing-me/AttackVLM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227814455,"owners_count":17823909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-attack","deep-generative-model","foundation-models","generative-ai","image-to-text-generation","large-language-models","text-to-image-generation","trustworthy-ai","vision-language-model"],"created_at":"2024-08-12T08:06:36.511Z","updated_at":"2025-07-27T15:31:50.112Z","avatar_url":"https://github.com/yunqing-me.png","language":"Python","funding_links":[],"categories":["Adversarial-Attack"],"sub_categories":[],"readme":"\u003ch1 align='center' style=\"text-align:center; font-weight:bold; font-size:2.0em;letter-spacing:2.0px;\"\u003e\n                On Evaluating Adversarial Robustness of \u003c/br\u003e Large Vision-Language Models \u003c/h1\u003e\n\n\u003c!-- \u003cp align='center' style=font-size:1.2em;\u003e\n\u003cb\u003e\n\u003cem\u003earXiv-Preprint, 2023\u003c/em\u003e \u003cbr\u003e\n\u003c/b\u003e\n\u003c/p\u003e --\u003e\n\n\u003cp align='left' style=\"text-align:left;font-size:1.2em;\"\u003e\n\u003cb\u003e\n    [\u003ca href=\"https://yunqing-me.github.io/AttackVLM/\" target=\"_blank\" style=\"text-decoration: none;\"\u003eProject Page\u003c/a\u003e] |\n    [\u003ca href=\"https://yunqing-me.github.io/AttackVLM/\" target=\"_blank\" style=\"text-decoration: none;\"\u003eSlides\u003c/a\u003e] |\n    [\u003ca href=\"https://arxiv.org/pdf/2305.16934.pdf\" target=\"_blank\" style=\"text-decoration: none;\"\u003earXiv\u003c/a\u003e] | \n    [\u003ca href=\"https://drive.google.com/drive/folders/118MTDLEw0YefC-Z0eGllKNAx_aavBrFP?usp=sharing\" target=\"_blank\" style=\"text-decoration: none;\"\u003eData Repository\u003c/a\u003e]\u0026nbsp;\n\u003c/b\u003e\n\u003c/p\u003e\n\n\n----------------------------------------------------------------------\n\n### TL, DR: \n```\nIn this research, we evaluate the adversarial robustness of recent large vision-language (generative) models (VLMs), under the most realistic and challenging setting with threat model of black-box access and targeted goal.\n\nOur proposed method aims for the targeted response generation over large VLMs such as MiniGPT-4, LLaVA, Unidiffuser, BLIP/2, Img2Prompt, etc.\n\nIn other words, we mislead and let the VLMs say what you want, regardless of the content of the input image query.\n```\n\n![Teaser image](./assets/teaser_1.jpg)\n![Teaser image](./assets/teaser_2.jpg)\n\n# Requirements\n\n- Platform: Linux\n- Hardware: A100 PCIe 40G\n- lmdb, tqdm\n- wandb, torchvision, etc.\n\nIn our work, we used DALL-E, Midjourney and Stable Diffusion for the target image generation and demonstration. For the large-scale experiments, we apply [Stable Diffusion](https://github.com/CompVis/stable-diffusion) for target image generation. To install Stable Diffusion, we init our [conda](https://docs.conda.io/en/latest/) environment following [Latent Diffusion Models](https://github.com/CompVis/latent-diffusion). A suitable base conda environment named `ldm` can be created and activated with:\n```\nconda env create -f environment.yaml\nconda activate ldm\n```\n\nNote that for different victim models, we will follow their official implementations and conda environments.\n\n\n# Targeted Image Generation\n![Teaser image](./assets/teaser_3.jpg)\nAs discussed in our paper, to achieve a flexible targeted attack, we leverage a pretrained text-to-image model to generate an targetd image given a single caption as the targeted text. Consequently, in this way you can specify the targeted caption for attack by yourself! \n\nWe use [Stable Diffusion](https://github.com/CompVis/stable-diffusion), [DALL-E](https://openai.com/blog/dall-e-now-available-without-waitlist) or [Midjourney](https://www.midjourney.com/app/) as the text-to-image generators in our experiments. Here, we use Stable Diffusion for demonstration (thanks for open-sourcing!). \n\n## Prepare the scripts\n\n```\ngit clone https://github.com/CompVis/stable-diffusion.git\ncd stable-diffusion\n```\nthen, prepare the full targeted captions from [MS-COCO](https://cocodataset.org/#home), or download our processed and cleaned version:\n```\nhttps://drive.google.com/file/d/19tT036LBvqYonzI7PfU9qVi3jVGApKrg/view?usp=sharing\n```\nand move it to ```./stable-diffusion/```. In experiments, one can randomly sample a subset of COCO captions (e.g., `10`, `100`, `1K`) for the adversarial attack. For example, lets assume we sampled some COCO captions from the following file as our targeted text c_tar:\n```\nhttps://drive.google.com/file/d/1e5W3Yim7ZJRw3_C64yqVZg_Na7dOawaF/view?usp=sharing\n```\n\n## Generate the targeted images\nThe targeted images h_ξ(c_tar) can be obtained via Stable Diffusion by reading text prompt from the sampled COCO captions, with the script below and [`txt2img_coco.py`](https://drive.google.com/file/d/1hTHxlgdx97_uEL3g9AmVx-qGNgssJeIy/view?usp=sharing) (please move `txt2img_coco.py` to ```./stable-diffusion/```, note that hyperparameters can be adjusted with your preference):\n\u003c!-- $\\boldsymbol{h}_\\xi(\\boldsymbol{c}_\\text{tar})$ --\u003e\n\n```\npython txt2img_coco.py \\\n        --ddim_eta 0.0 \\\n        --n_samples 10 \\\n        --n_iter 1 \\\n        --scale 7.5 \\\n        --ddim_steps 50 \\\n        --plms \\\n        --skip_grid \\\n        --ckpt ./_model_pool/sd-v1-4-full-ema.ckpt \\\n        --from-file './name_of_your_coco_captions_file.txt' \\\n        --outdir './path_of_your_targeted_images' \\\n```\nwhere the ckpt is provided by [Stable Diffusion v1](https://github.com/CompVis/stable-diffusion#weights:~:text=The%20weights%20are%20available%20via) and can be downloaded here: [sd-v1-4-full-ema.ckpt](https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/resolve/main/sd-v1-4-full-ema.ckpt).\n\nAdditional implementation details of text-to-image generation by Stable Diffusion can be found [HERE](https://github.com/CompVis/stable-diffusion#:~:text=active%20community%20development.-,Reference%20Sampling%20Script,-We%20provide%20a).\n\n# Adversarial Attack \u0026 Black-box Query\n\n## Overview of our AttackVLM strategy\n![Teaser image](./assets/teaser_4.jpg)\n\n## Prepare the VLM scripts\n\nThere are two steps of adversarial attack for VLMs: (1) transfer-based attacking strategy and (2) query-based attacking strategy using (1) as initialization. For BLIP/BLIP-2/Img2Prompt Models, please refer to ```./LAVIS_tool```. Here, we use [Unidiffuser](https://github.com/thu-ml/unidiffuser) for an example. \n\n### \u003cb\u003e Example: Unidiffuser \u003c/b\u003e\n- Installation\n```\ngit clone https://github.com/thu-ml/unidiffuser.git\ncd unidiffuser\ncp ../unidff_tool/* ./\n```\nthen, create a suitable conda environment named `unidiffuser` following the steps [HERE](https://github.com/thu-ml/unidiffuser#:~:text=to%2Dimage%20generation\\).-,Dependency,-conda%20create%20%2Dn), and prepare the corresponding model weights (we use `uvit_v1.pth` as the weight of U-ViT).\n\n- Transfer-based attacking strategy\n\n```\nconda activate unidiffuser\n\nbash _train_adv_img_trans.sh\n```\nthe crafted adv images x_trans will be stored in `dir of white-box transfer images` specified in `--output`. Then, we perform image-to-text and store the generated response of x_trans. This can be achieved by:\n\n```\npython _eval_i2t_dataset.py \\\n        --batch_size 100 \\\n        --mode i2t \\\n        --img_path 'dir of white-box transfer images' \\\n        --output 'dir of white-box transfer captions' \\\n```\n\nwhere the generated responses will be stored in `dir of white-box transfer captions` in `.txt` format. We will use them for pseudo-gradient estimation via RGF-estimator.\n\n- Query-based attacking strategy (via RGF-estimator): assume we use **fixed perturbation budget** for `MF-ii + MF-tt` (e.g., 8 px)\n\n```\nbash _train_trans_and_query_fixed_budget.sh\n```\n\nOn the other hand, if you want to conduct transfer+query - based attack with **separate perturbation budget**, we additionally provide a script:\n\n```\nbash _train_trans_and_query_more_budget.sh\n```\n\n# Evaluation\nHere, we use [`wandb`](https://wandb.ai/site) to dynamically monitor the moving average of the CLIP score (e.g., RN50, ViT-B/32, ViT-L/14, etc.) to evaluate the similarity between (a) the generated response (of trans/query images) and (b) the predefined targeted text `c_tar`.\n\nAn example shown as below, where the dotted line denotes the moving average of the CLIP score (of image captions) after query:\n![Teaser image](./assets/example.png)\n\nMeanwhile, the image caption after query will be stored and the directory can be specified by `--output`.\n\n\n# Visualization\nThe effect of the attack would be visualized upon the success of the crafted adversarial samples. If you want to visualize the heat map similar to those demonstrated in the paper or the following figure:\n![Teaser image](./assets/IMG3425_703_vis.jpg)\nplease refer to the guidlines in [this tutorial](https://colab.research.google.com/github/salesforce/LAVIS/blob/main/projects/img2llm-vqa/img2llm_vqa.ipynb) or the original work of [GradCAM](https://github.com/jacobgil/pytorch-grad-cam). Please nota that there are [known limitations](https://arxiv.org/abs/1710.11063) of GradCAM.\n\n\n# Bibtex\nIf you find this project useful in your research, please consider citing our paper:\n\n```\n@inproceedings{zhao2023evaluate,\n  title={On Evaluating Adversarial Robustness of Large Vision-Language Models},\n  author={Zhao, Yunqing and Pang, Tianyu and Du, Chao and Yang, Xiao and Li, Chongxuan and Cheung, Ngai-Man and Lin, Min},\n  booktitle={Thirty-seventh Conference on Neural Information Processing Systems},\n  year={2023}\n}\n```\n\nMeanwhile, a relevant research that aims to protect your generated assets or copyright by [Embedding a Watermark to (multi-modal) Diffusion Models](https://github.com/yunqing-me/WatermarkDM):\n```\n@article{zhao2023recipe,\n  title={A Recipe for Watermarking Diffusion Models},\n  author={Zhao, Yunqing and Pang, Tianyu and Du, Chao and Yang, Xiao and Cheung, Ngai-Man and Lin, Min},\n  journal={arXiv preprint arXiv:2303.10137},\n  year={2023}\n}\n```\n```\n@misc{pang2024device,\n  title={Device and method for watermarking a diffusion model},\n  author={Pang, Tianyu and Du, Chao and Lin, Min and Zhao, Yunqing},\n  year={2024},\n  month=sep # \"~12\",\n  publisher={Google Patents},\n  note={US Patent App. 18/597,640}\n}\n```\n\n\n# Acknowledgement: \n\nWe appreciate the wonderful base implementation of [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4), [LLaVA](https://llava-vl.github.io/), [Unidiffuser](https://github.com/thu-ml/unidiffuser), [LAVIS](https://github.com/salesforce/LAVIS) and [CLIP](https://openai.com/research/clip). \nWe also thank [@MetaAI](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) for open-sourcing their LLaMA checkponts. We thank SiSi for providing some enjoyable and visual-pleasant images generated by [@Midjourney](https://www.midjourney.com/app/) in our research.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyunqing-me%2FAttackVLM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyunqing-me%2FAttackVLM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyunqing-me%2FAttackVLM/lists"}