{"id":27948356,"url":"https://github.com/pku-alignment/safevla","last_synced_at":"2025-05-07T14:57:51.374Z","repository":{"id":279897820,"uuid":"939851266","full_name":"PKU-Alignment/SafeVLA","owner":"PKU-Alignment","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-18T13:20:57.000Z","size":102758,"stargazers_count":32,"open_issues_count":3,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-07T14:57:26.979Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PKU-Alignment.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-27T08:01:28.000Z","updated_at":"2025-05-07T13:55:03.000Z","dependencies_parsed_at":"2025-02-28T11:36:45.536Z","dependency_job_id":null,"html_url":"https://github.com/PKU-Alignment/SafeVLA","commit_stats":null,"previous_names":["pku-alignment/safevla"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PKU-Alignment%2FSafeVLA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PKU-Alignment%2FSafeVLA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PKU-Alignment%2FSafeVLA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PKU-Alignment%2FSafeVLA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PKU-Alignment","download_url":"https://codeload.github.com/PKU-Alignment/SafeVLA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252902623,"owners_count":21822257,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T14:57:50.685Z","updated_at":"2025-05-07T14:57:51.350Z","avatar_url":"https://github.com/PKU-Alignment.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Safe Reinforcement Learning\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://arxiv.org/abs/2503.03480\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-2503.03480-red?style=for-the-badge\u0026link=https%3A%2F%2Farxiv.org%2Fabs%2F2503.03480\" alt=\"arXiv\"\u003e\u003c/a\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-%20Apache%202.0-green?style=for-the-badge\u0026link=https%3A%2F%2Farxiv.org%2Fabs%2F2503.03480\" alt=\"License\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://sites.google.com/view/pku-safevla\"\u003e\u003cimg src=\"https://img.shields.io/badge/website-%20google%20sites%20-purple?style=for-the-badge\u0026link=https%3A%2F%2Farxiv.org%2Fabs%2F2503.03480\" alt=\"Website\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\n\u003cdiv style=\"text-align: center;\"\u003e\n    \u003cimg src=\"assets/fig_1.png\" alt=\"safevla_fig_1\"\u003e\n\nThe left figure illustrates three typical unsafe behaviors exhibited by traditional VLA models in grasping tasks: 1) severe damage to irrelevant objects, 2) misidentification of target objects leading to the misuse of hazardous items, and 3) interactions with dangerous objects while executing instructions. The right figure further demonstrates three unsafe behaviors of traditional VLA models during navigation through examples of navigation paths.\n\u003c/div\u003e\n\n\u003e **The overview of SafeVLA pipeline.** **Top-Left**: Three typical unsafe behaviors of the standard VLA during grasping, including 1) severe damage to irrelevant objects, 2) misidentification of the target leading to the abuse of hazardous objects, and 3) interaction with dangerous objects while executing the instruction. **Bottom-Left**: An example of a navigation route illustrating three typical unsafe behaviors of standard VLAs during the navigation process. **Middle**: A comparison between SafeVLA and the standard VLA, showing how SafeVLA’s aligned objective balances safety and task performance. **Right**: (a) SafeVLA is significantly safer than baseline methods and achieves state-of-the-art task performance. (b) The cost distribution in one test environment shows that SafeVLA significantly improves the model’s safety across the entire room compared to the baseline.\n\n|| \u003cdetails\u003e\u003csummary\u003eprompt\u003c/summary\u003enavigate to a basketball\u003c/details\u003e | \u003cdetails\u003e\u003csummary\u003eprompt\u003c/summary\u003efind to a basketball\u003c/details\u003e  | \u003cdetails\u003e\u003csummary\u003eprompt\u003c/summary\u003elocate a vase.\u003c/details\u003e |\u003cdetails\u003e\u003csummary\u003eprompt\u003c/summary\u003efind a spray bottle and pick up that spray bottle\u003c/details\u003e|\n|---| ---------------------------------- | --- | --- | --- |\n|Baseline| \u003cimg src=\"assets/unsafevideo_1.gif\" alt=\"Image 8\" style=\"max-width: 100%; height: auto;\"\u003e| \u003cimg src=\"assets/unsafevideo_2.gif\" alt=\"Image 8\" style=\"max-width: 100%; height: auto;\"\u003e | \u003cimg src=\"assets/unsafevideo_3.gif\" alt=\"Image 8\" style=\"max-width: 100%; height: auto;\"\u003e  | \u003cimg src=\"assets/unsafevideo_4.gif\" alt=\"Image 8\" style=\"max-width: 100%; height: auto;\"\u003e|\n|**SafeVLA**| \u003cimg src=\"assets/safevideo_1.gif\" alt=\"Image 8\" style=\"max-width: 100%; height: auto;\"\u003e | \u003cimg src=\"assets/safevideo_2.gif\" alt=\"Image 8\" style=\"max-width: 100%; height: auto;\"\u003e | \u003cimg src=\"assets/safevideo_3.gif\" alt=\"Image 8\" style=\"max-width: 100%; height: auto;\"\u003e  | \u003cimg src=\"assets/safevideo_4.gif\" alt=\"Image 8\" style=\"max-width: 100%; height: auto;\"\u003e|\n\u003e Several demos demonstrate how SafeVLA can ensure safety while optimizing task performance.\n\u003chr style=\"border: 2px solid gray;\"\u003e\u003c/hr\u003e\n\n## Latest Updates\n- [2025-03-06] Paper released: [SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Safe Reinforcement Learning](https://arxiv.org/abs/2503.03480)\n- [2025-02-28] Initial release\n\n\u003chr style=\"border: 2px solid gray;\"\u003e\u003c/hr\u003e\n\n\n\n## Quick Start\n\n### Setting up the Python environment\n\nPlease use the pre-built image from Docker Hub:\n\n```bash\ndocker pull safevla/safevla:v0\n```\n\nThen\n\n```bash\nexport CODE_PATH=/path/to/this/repo\nexport DATA_PATH=/path/to/training_data\nexport DOCKER_IMAGE=safevla/safevla:v0\ndocker run \\\n    --gpus all \\\n    --device /dev/dri \\\n    --mount type=bind,source=${CODE_PATH},target=/root/spoc \\\n    --mount type=bind,source=${DATA_PATH},target=/root/data \\\n    --shm-size 50G \\\n    --runtime=nvidia \\\n    -it ${DOCKER_IMAGE}:latest\n```\n\nand use the following conda environment:\n\n```bash\nconda activate spoc\n```\nThe ``Safety-CHORES`` task we proposed has been integrated into [Safety-Gymnasium](https://github.com/PKU-Alignment/safety-gymnasium/tree/main/safety_gymnasium/tasks/safe_vla)\nThen please clone ``Safety-gymnasium`` and install it:\n```bash\ngit clone https://github.com/PKU-Alignment/safety-gymnasium.git\ncd safety-gymnasium\npip install -e .\n``` \n\n## Training\n\n\nIn order to run training and evaluation you'll need:\n\n\u003e1. The processed/optimized Objaverse assets along with their annotations.\n\u003e2. The set of ProcTHOR-Objaverse houses you'd like to train/evaluate on.\n\u003e3. For evaluation only, a trained model checkpoint.\n\nBelow we describe how to download the assets, annotations, and the ProcTHOR-Objaverse houses. We also describe how you can use one of our pre-trained models to run evaluation.\n\n### Downloading assets, annotations, and houses\n\n#### Downloading optimized Objaverse assets and annotations\n\nPick a directory `/path/to/objaverse_assets` where you'd like to save the assets and annotations. Then run the following commands:\n\n```bash\npython -m objathor.dataset.download_annotations --version 2023_07_28 --path /path/to/objaverse_assets\npython -m objathor.dataset.download_assets --version 2023_07_28 --path /path/to/objaverse_assets\n```\n\nThese will create the directory structure:\n\n```\n/path/to/objaverse_assets\n    2023_07_28\n        annotations.json.gz                              # The annotations for each object\n        assets\n            000074a334c541878360457c672b6c2e             # asset id\n                000074a334c541878360457c672b6c2e.pkl.gz\n                albedo.jpg\n                emission.jpg\n                normal.jpg\n                thor_metadata.json\n            ... #  39663 more asset directories\n```\n\n#### Downloading ProcTHOR-Objaverse houses\n\nPick a directory `/path/to/objaverse_houses` where you'd like to save ProcTHOR-Objaverse houses. Then run: \n\n```bash\npython -m scripts.download_objaverse_houses --save_dir /path/to/objaverse_houses --subset val\n```\n\nto download the validation set of houses as `/path/to/objaverse_houses/val.jsonl.gz`.\nYou can also change `val` to `train` to download the training set of houses.\n\n### Setting environment variables\n\nNext you need to set the following environment variables:\n\n```bash\nexport PYTHONPATH=/path/to/code_in_docker\nexport OBJAVERSE_HOUSES_DIR=/path/to/objaverse_houses\nexport OBJAVERSE_DATA_DIR=/path/to/objaverse_assets\n```\n\nFor training, we recommend to set two more environment variables to avoid timeout issues from [AllenAct](https://allenact.org/):\n\n```bash\nexport ALLENACT_DEBUG=True\nexport ALLENACT_DEBUG_VST_TIMEOUT=2000\n```\n\n### Running Safe RL finetuning\n\nDownload pretrained IL ckpt:\n\n```bash\npython scripts/download_il_ckpt.py --ckpt_ids spoc_IL --save_dir PATH_TO_SAVE_DIR\n```\n\nRun Safe RL training:\n\n```bash\npython training/online/dinov2_vits_tsfm_rgb_augment_objectnav.py train --il_ckpt_path IL_CKPT_PATH --num_train_processes NUM_OF_TRAIN_PROCESSES --output_dir PATH_TO_RESULT --dataset_dir PATH_TO_DATASET --cost_limit COST_LIMIT --tag EXP_NAME\n```\n\nFor example,\n\n```bash\npython training/online/dinov2_vits_tsfm_rgb_augment_objectnav.py train --il_ckpt_path /root/data/il_ckpt/spoc_IL/model.ckpt --num_train_processes 32 --output_dir results --dataset_dir /root/data/data/astar/ObjectNavType --cost_limit 2.31964 --tag SafeVLA2.31964-ObjectNavType-RL-DinoV2-ViTS-TSFM\n```\n\n## Evaluation\n\n\n#### Downloading the trained model ckpt and evaluation results\n\n```bash\npython scripts/download_trained_ckpt.py --save_dir ckpt\ncd ckpt\ncat safevla_* | tar -xz\n```\n\n```bash\nbash scripts/objnav.bash\n```\n---\n\n## Citation\nIf you find our code or models useful in your work, please cite [our paper](https://arxiv.org/abs/2503.03480):\n```bash\n@article{zhang25safevla,\n    title={SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Safe Reinforcement Learning},\n    author={Borong Zhang and Yuhao Zhang and Jiaming Ji and Yingshan Lei and Josef Dai and Yuanpei Chen and Yaodong Yang},\n    journal = {arXiv preprint arXiv:2503.03480},\n    year={2025}\n} \n```\n\n## Acknowledgment\n\nThis repository benefits from [AllenAct](https://github.com/allenai/allenact), [AI2THOR](https://github.com/allenai/ai2thor), [ProcTHOR](https://github.com/allenai/procthor), [SPOC](https://github.com/allenai/spoc-robot-training), [FLaRe](https://github.com/JiahengHu/FLaRe) and [Align-Anything](https://github.com/PKU-Alignment/Align-Anything).\n\nThanks for their wonderful works and their efforts to further promote VLA research.\nSafeVLA and its related assets are built and open-sourced with love and respect ❤️.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpku-alignment%2Fsafevla","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpku-alignment%2Fsafevla","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpku-alignment%2Fsafevla/lists"}