{"id":31908963,"url":"https://github.com/allenai/robustnav","last_synced_at":"2025-10-13T15:59:56.166Z","repository":{"id":48297466,"uuid":"370939315","full_name":"allenai/robustnav","owner":"allenai","description":"Evaluating pre-trained navigation agents under corruptions","archived":false,"fork":false,"pushed_at":"2021-09-07T05:00:42.000Z","size":16386,"stargazers_count":25,"open_issues_count":1,"forks_count":3,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-04-14T07:50:02.299Z","etag":null,"topics":["ai2thor-environment","computer-vision","deep-reinforcement-learning","embodied-ai","embodied-vision","visual-navigation"],"latest_commit_sha":null,"homepage":"https://prior.allenai.org/projects/robustnav","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/allenai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-26T07:07:42.000Z","updated_at":"2024-01-04T16:57:59.000Z","dependencies_parsed_at":"2022-09-19T02:42:48.832Z","dependency_job_id":null,"html_url":"https://github.com/allenai/robustnav","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/allenai/robustnav","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Frobustnav","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Frobustnav/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Frobustnav/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Frobustnav/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/allenai","download_url":"https://codeload.github.com/allenai/robustnav/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Frobustnav/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279015953,"owners_count":26085777,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai2thor-environment","computer-vision","deep-reinforcement-learning","embodied-ai","embodied-vision","visual-navigation"],"created_at":"2025-10-13T15:59:45.541Z","updated_at":"2025-10-13T15:59:56.160Z","avatar_url":"https://github.com/allenai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003ca href=\"https://arxiv.org/abs/2106.04531\"\u003eRobustNav: Towards Benchmarking Robustness in Embodied Navigation\u003c/a\u003e\n\n#### Prithvijit Chattopadhyay, Judy Hoffman, Roozbeh Mottaghi, Ani Kembhavi\n\n#### \u003ca href=\"https://prior.allenai.org/projects/robustnav\"\u003e(Project Page)\u003c/a\u003e\n\nAs an attempt towards assessing the robustness of embodied navigation agents, we propose \u003cb\u003eRobustNav\u003c/b\u003e, a\nframework to quantify the performance of embodied navigation agents when exposed to a wide variety of visual\n– affecting RGB inputs – and dynamics – affecting transition dynamics – corruptions. Most recent efforts in visual\nnavigation have typically focused on generalizing to novel\ntarget environments with similar appearance and dynamics characteristics. With \u003cb\u003eRobustNav\u003c/b\u003e, we find that some\nstandard embodied navigation agents significantly underperform (or fail) in the presence of visual or dynamics corruptions. We systematically analyze the kind of idiosyncrasies that emerge in the behavior of such agents when operating under corruptions. Finally, for visual corruptions\nin \u003cb\u003eRobustNav\u003c/b\u003e, we show that while standard techniques\nto improve robustness such as data-augmentation and self-supervised adaptation offer some zero-shot resistance and\nimprovements in navigation performance, there is still a\nlong way to go in terms of recovering lost performance relative to clean “non-corrupt” settings, warranting more research in this direction.\n\nRobustNav is based on \u003ca href=https://allenact.org/\u003eAllenAct\u003c/a\u003e framework and the majority of the core training algorithms and pipelines are borrowed from a specific commit of the \u003ca href=https://github.com/allenai/allenact\u003eAllenAct code base\u003c/a\u003e.\n\n### Citation\n\nIf you find this project useful in your research, please consider citing:\n\n```\n@article{2021RobustNav, \n  author = {Chattopadhyay, Prithvijit and Hoffman, Judy and Mottaghi, Roozbeh and Kembhavi, Ani},\n  title = {RobustNav: Towards Benchmarking Robustness in Embodied Navigation},\n  year = 2021,\n  journal = {International Conference in Computer Vision (ICCV)}\n}\n```\n\n### Contents\n\n\u003cdiv class=\"toc\"\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#-installation\"\u003e💻 Installation\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#-robustnav-description\"\u003e📝 RobustNav Description\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#-visual-corruptions\"\u003e🖼️ Visual Corruptions\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#-dynamics-corruptions\"\u003e🏃 Dynamics Corruptions\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#-dataset\"\u003e📊 Dataset\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#-training-an-agent\"\u003e🏋 Training an Agent\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#-evaluating-a-pre-trained-agent\"\u003e💪 Evaluating a Pre-Trained Agent\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/div\u003e\n\n## 💻 Installation\n\n1. First install [Anaconda](https://docs.anaconda.com/anaconda/install/linux/)\n\n2. To begin, clone this repository locally\n\n```bash\ngit clone https://github.com/allenai/robustnav.git\n```\n\n3. You can then install requirements inside a conda environment by running\n\n```bash\nconda env create --file robustnav.yml --name \u003cENV-NAME\u003e\n```\n\nwhere `ENV-NAME` is the name of the environment. The default environment name is `robustnav` if `ENV-NAME` is not specified.\n\n4. Activate the conda environment by running\n\n```bash\nconda activate \u003cENV-NAME\u003e\n```\n\nor\n\n```bash\nconda activate robustnav\n```\n\n5. \u003cb\u003e[Optional]\u003c/b\u003e In case the CUDA version on your machine (if you are using linux, you might find this version by running `/usr/local/cuda/bin/nvcc --version`) is different from the one used in the repository, you can install your desired version by running:\n\n```bash\nconda env update --file ./conda/environment-\u003cCUDA-VERSION\u003e.yml --name \u003cENV-NAME\u003e\n```\n\nwhere `CUDA-VERSION` is the specified CUDA version (this repo right now supports 9.1, 10.1 and 10.2 under the folder `conda/`) and `ENV-NAME` is the name of the project environment (`robustnav` or the specified name in step 3)\n\nAfter installing the requirements, you should start the xserver by running [this script](scripts/startx.py) in the background.\n\n## 📝 RobustNav Description\n\n\u003cimg src=\"media/robustnav_teaser.png\" alt=\"\" width=\"80%\"\u003e\n\nRobustNav is a framework to evaluate pre-trained navigation agents in the presence of a wide variety of visual and dynamics corruptions. As such, this codebase supports training (in line with AllenAct) and evaluating PointNav\nand ObjectNav agents (either RGB and / or Depth observations) in unseen environments where visual and dynamics corruptions have been applied. For instance, in the above example, a (LoCoBot based) agent trained in a set\nof clean environments (see (a)), is evaluated in environments with Camera-Crack and Motion Drift as the visual and dynamics corruptions respectively (see (b) and (c)).\n\n## 🖼️ Visual Corruptions\n\n\u003cimg src=\"media/vis_corr.png\" alt=\"\" width=\"80%\"\u003e\n\nAs highlighted in the figure above, RobustNav supports 7 visual corruptions --\nDefocus Blur, Motion Blur, Spatter, Camera Crack, Low Lighting, Lower FOV and Speckle Noise. The above figure represents \"clean\" and \"corrupt\" egocentric RGB frames as viewed by the agent. Among these corruptions, Defocus Blur, Motion Blur, Spatter, Low Lighting and Speckle Noise are supported at 5 progressively increasing levels of severity. Defocus Blur, Spatter and Speckle Noise are akin to the visual corruptions introduced \u003ca href=\"https://arxiv.org/abs/1903.12261\"\u003ehere\u003c/a\u003e. Note that these corruptions are applied only on the RGB observations (not Depth).\n\n## 🏃 Dynamics Corruptions\n\n\u003cimg src=\"media/dyn_corr.png\" alt=\"\" width=\"80%\"\u003e\n\nAs highlighted above, RobustNav includes 4 dynamics corruptions -- (a) Motion Bias (Constant), (b) Motion Bias (Stochastic), (c) Motion Drift and (d) Motor Failure. Motion Bias (Constant \u0026 Stochastic) are intended to model scene-level friction and high and low friction zones in the environment. Motion Drift models a setting where translation (forward) has a slight bias towards turning left or right. In Motor Failure, one of the rotation actions fail.\n\nPointNav agents have the following actions available to them -- `MoveAhead` (0.25m), `RotateLeft` (30\u0026deg;), `RotateRight` (30\u0026deg;) and `End` (end an episode). ObjectNav agents additionally have access to `LookUp` and `LookDown` actions -- indicating change in agent's view above or below the horizon.\n\n## 📊 Dataset\n\nAgents in RobustNav are trained and evaluated on the RoboTHOR set of scenes. RoboTHOR consists of 60 training and 15 validation environments. Agents are trained on the train set of environments are evaluated on the validation set of environments. Evaluation episodes are present in `datasets/robothor-pointnav/robustnav_eval/` for PointNav and `datasets/robothor-objectnav/robustnav_eval/` for ObjectNav. For ObjectNav, we consider 12 object categories -- AlarmClock, Apple, BaseBallBat, BasketBall, Bowl, GarbageCan, HousePlant, Laptop, Mug, SprayBottle, Television and Vase.\n\n## 🏋 Training An Agent\n\nTo train an agent, we rely on the experiment configs located [here](projects/robustnav_baselines/experiments/robustnav_train/). Upon activating the robustnav anaconda environment, run the following command to train an agent:\n\n```\npython main.py \\\n    -o storage/\u003cEXPERIMENT-STORAGE-DIRECTORY\u003e \\\n    -b projects/robustnav_baselines/experiments/robustnav_train \u003cEXPERIMENT-CONFIG\u003e \\\n    -s \u003cRANDOM-SEED\u003e \\\n    -et \u003cEXPERIMENT-STRING-TAG-IDENTIFIER\u003e\n```\n\nFor instance, to train a PointNav RGB agent, the command is:\n\n```\npython main.py \\\n    -o storage/robothor-pointnav-rgb-resnetgru-ddppo \\\n    -b projects/robustnav_baselines/experiments/robustnav_train pointnav_robothor_vanilla_rgb_resnet_ddppo \\\n    -s 12345 \\\n    -et rnav_pointnav_vanilla_rgb_resnet_ddppo_clean\n```\n\nCheckpoints over the course of training will be stored under `storage/robothor-pointnav-rgb-resnetgru-ddppo/checkpoints/` and tensorboard logs will be stored under `storage/robothor-pointnav-rgb-resnetgru-ddppo/tb/`.\n\nIf you have Tensorboard installed, you can track training progress with\n\n```bash\ntensorboard --logdir storage/robothor-pointnav-rgb-resnetgru-ddppo/tb/\n```\n\nwhich will default to the URL [http://localhost:6006/](http://localhost:6006/).\n\nTo see commands used to train the navigation agents considered in the paper, see `train_navigation_agents.sh`\n\n## 💪 Evaluating A Pre-Trained Agent\n\nTo evaluate pre-trained navigation agents, first run [this](rnav_checkpoints/download_checkpoints.sh) script to download all the checkpoints.\n\nFor evaluation, we will consider evaluating an agent under (1) clean settings, (2) under visual corruptions, (3) under dynamics corruptions and (4)under visual + dynamics corruptions. We will consider evaluating a pre-trained PointNav RGB agent as the running example.\n\n### Clean Settings\n\nTo evaluate under clean settings, run the following command:\n\n```\npython main.py \\\n    -o \u003cMETRIC-STORAGE-DIRECTORY\u003e \\\n    -b projects/robustnav_baselines/experiments/robustnav_eval \u003cEXPERIMENT-CONFIG\u003e \\\n    -c \u003cCHECKPOINT-PATH\u003e \\\n    -t \u003cCHECKPOINT-TIMESTAMP\u003e \\\n    -et \u003cEXPERIMENT-STRING-TAG-IDENTIFIER\u003e \\\n    -s \u003cRANDOM-SEED\u003e \\\n    -e \\\n    -tsg \u003cGPU-ID\u003e\n```\n\nFor a PointNav RGB agent, this is equivalent to:\n\n```\npython main.py \\\n    -o storage/robothor-pointnav-rgb-resnetgru-ddppo-eval \\\n    -b projects/robustnav_baselines/experiments/robustnav_eval pointnav_robothor_vanilla_rgb_resnet_ddppo \\\n    -c rnav_checkpoints/pnav_rgb_agent.pt \\\n    -t 2021-03-02_05-58-58 \\\n    -et rnav_pointnav_vanilla_rgb_resnet_ddppo_clean \\\n    -s 12345 \\\n    -e \\\n    -tsg 0\n```\n\nThe output trajectories and associated metrics per episode will be stored under `storage/robothor-pointnav-rgb-resnetgru-ddppo-eval/metrics/`.\n\n### Visual Corruptions\n\nFor visual corruptions, when evaluating under Defocus Blur, Motion Blur, Spatter, Low Lighting and Speckle Noise, we have two additional command line arguments `-vc` and `-vs` identifying the kind and severity of the corruptions. For instance, to evaluate under Defocus Blur, the command is:\n\n```\npython main.py \\\n    -o storage/robothor-pointnav-rgb-resnetgru-ddppo-eval \\\n    -b projects/robustnav_baselines/experiments/robustnav_eval pointnav_robothor_vanilla_rgb_resnet_ddppo \\\n    -c rnav_checkpoints/pnav_rgb_agent.pt \\\n    -t 2021-03-02_05-58-58 \\\n    -et rnav_pointnav_vanilla_rgb_resnet_ddppo_Defocus_Blur_s5 \\\n    -s 12345 \\\n    -e \\\n    -tsg 0 \\\n    -vc Defocus_Blur \\\n    -vs 5\n```\n\nWhen evaluating under Lower-FOV, run:\n\n```\npython main.py \\\n    -o storage/robothor-pointnav-rgb-resnetgru-ddppo-eval \\\n    -b projects/robustnav_baselines/experiments/robustnav_eval pointnav_robothor_vanilla_rgb_resnet_ddppo_fov \\\n    -c rnav_checkpoints/pnav_rgb_agent.pt \\\n    -t 2021-03-02_05-58-58 \\\n    -et rnav_pointnav_vanilla_rgb_resnet_ddppo_fov \\\n    -s 12345 \\\n    -e \\\n    -tsg 0\n```\n\nWhen evaluating under Camera-Crack, run:\n\n```\npython main.py \\\n    -o storage/robothor-pointnav-rgb-resnetgru-ddppo-eval \\\n    -b projects/robustnav_baselines/experiments/robustnav_eval pointnav_robothor_vanilla_rgb_resnet_ddppo_cam_crack \\\n    -c rnav_checkpoints/pnav_rgb_agent.pt \\\n    -t 2021-03-02_05-58-58 \\\n    -et rnav_pointnav_vanilla_rgb_resnet_ddppo_cam_crack \\\n    -s 12345 \\\n    -e \\\n    -tsg 0\n```\n\n### Dynamics Corruptions\n\nWhen evaluating under Motion Bias (Constant), run:\n\n```\npython main.py \\\n    -o storage/robothor-pointnav-rgb-resnetgru-ddppo-eval \\\n    -b projects/robustnav_baselines/experiments/robustnav_eval pointnav_robothor_vanilla_rgb_resnet_ddppo_dyn \\\n    -c rnav_checkpoints/pnav_rgb_agent.pt \\\n    -t 2021-03-02_05-58-58 \\\n    -et rnav_pointnav_vanilla_rgb_resnet_ddppo_clean_mb_const \\\n    -s 12345 \\\n    -e \\\n    -tsg 0 \\\n    -dcr \\\n    -ctr \\\n    -crt\n```\n\n`-dcr` is set to `True` for Motion Bias, Motion Drift and Motor Failure. `-ctr` is set to `True` for constant translation bias and `-crt` is set to `True` for constant rotation bias.\n\nWhen evaluating under Motion Bias (Stochastic), run:\n\n```\npython main.py \\\n    -o storage/robothor-pointnav-rgb-resnetgru-ddppo-eval \\\n    -b projects/robustnav_baselines/experiments/robustnav_eval pointnav_robothor_vanilla_rgb_resnet_ddppo_dyn \\\n    -c rnav_checkpoints/pnav_rgb_agent.pt \\\n    -t 2021-03-02_05-58-58 \\\n    -et rnav_pointnav_vanilla_rgb_resnet_ddppo_clean_mb_stoch \\\n    -s 12345 \\\n    -e \\\n    -tsg 0 \\\n    -dcr \\\n    -str \\\n    -srt\n```\n\n`-str` is set to `True` for stochastic translation bias and `-srt` is set to `True` for stochastic rotation bias.\n\nWhen evaluating under Motion Drift, run:\n\n```\npython main.py \\\n    -o storage/robothor-pointnav-rgb-resnetgru-ddppo-eval \\\n    -b projects/robustnav_baselines/experiments/robustnav_eval pointnav_robothor_vanilla_rgb_resnet_ddppo_dyn \\\n    -c rnav_checkpoints/pnav_rgb_agent.pt \\\n    -t 2021-03-02_05-58-58 \\\n    -et rnav_pointnav_vanilla_rgb_resnet_ddppo_clean_drift_deg_10 \\\n    -s 12345 \\\n    -e \\\n    -tsg 0 \\\n    -dcr \\\n    -dr \\\n    -dr_deg 10.0\n```\n\n`-dr` is set to `True` for motion drift and `-dr_deg` is set to `10.0` as the drift angle.\n\nWhen evaluating under Motor Failure, run:\n\n```\npython main.py \\\n    -o storage/robothor-pointnav-rgb-resnetgru-ddppo-eval \\\n    -b projects/robustnav_baselines/experiments/robustnav_eval pointnav_robothor_vanilla_rgb_resnet_ddppo_dyn \\\n    -c rnav_checkpoints/pnav_rgb_agent.pt \\\n    -t 2021-03-02_05-58-58 \\\n    -et rnav_pointnav_vanilla_rgb_resnet_ddppo_clean_motfail \\\n    -s 12345 \\\n    -e \\\n    -tsg 0 \\\n    -dcr \\\n    -mf\n```\n\n`-mf` is set to `True` for motor failure.\n\nWhen evaluating under PyRobot noise models from \u003ca href=\"https://arxiv.org/abs/1906.08236\"\u003ehere\u003c/a\u003e, run:\n\n```\npython main.py \\\n    -o storage/robothor-pointnav-rgb-resnetgru-ddppo-eval \\\n    -b projects/robustnav_baselines/experiments/robustnav_eval pointnav_robothor_vanilla_rgb_resnet_ddppo_pyrobot_dyn \\\n    -c rnav_checkpoints/pnav_rgb_agent.pt \\\n    -t 2021-03-02_05-58-58 \\\n    -et rnav_pointnav_vanilla_rgb_resnet_ddppo_clean_pyrobot_ilqr_1 \\\n    -s 12345 \\\n    -e \\\n    -tsg 0\n```\n\nNote that the experiment config used for PyRobot settings is different from the ones used for Motion Bias, Motion Drift and Motor Failure. To evaluate other agents which differ either in terms of the task (PointNav\u0026#8594;ObjectNav) or sensors (RGB\u0026#8594;RGB-D), just pick the corresponding experiment configs and checkpoints under `projects/robustnav_baselines/experiments/robustnav_eval/` and `rnav_checkpoints/` respectively. These instructions are also highlighted in `eval_navigation_agents.sh`.\n\nIf you use the PyRobot noise models, please cite:\n\n```\n@article{murali2019pyrobot,\n  title={Pyrobot: An open-source robotics framework for research and benchmarking},\n  author={Murali, Adithyavairavan and Chen, Tao and Alwala, Kalyan Vasudev and Gandhi, Dhiraj and Pinto, Lerrel and Gupta, Saurabh and Gupta, Abhinav},\n  journal={arXiv preprint arXiv:1906.08236},\n  year={2019}\n}\n\n```\n\n### Visual + Dynamics Corruptions\n\nTo evaluate in the presence of visual and dynamics corruptions, use the additional `-vc` and `-vs` arguments to highlight the kind and severity of visual corruption in the commands used to evaluate under dynamics corruptions (above). For instance, to evaluate under Defocus Blur + Motion Drift, the command is:\n\n```\npython main.py \\\n    -o storage/robothor-pointnav-rgb-resnetgru-ddppo-eval \\\n    -b projects/robustnav_baselines/experiments/robustnav_eval pointnav_robothor_vanilla_rgb_resnet_ddppo_dyn \\\n    -c rnav_checkpoints/pnav_rgb_agent.pt \\\n    -t 2021-03-02_05-58-58 \\\n    -et rnav_pointnav_vanilla_rgb_resnet_ddppo_clean_drift_deg_10 \\\n    -s 12345 \\\n    -e \\\n    -tsg 0 \\\n    -dcr \\\n    -dr \\\n    -dr_deg 10.0 \\\n    -vc Defocus_Blur \\\n    -vs 5\n```\n\nTo find the time-stamps associated with all the checkpoints, refer to `eval_navigation_agents.sh`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallenai%2Frobustnav","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fallenai%2Frobustnav","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallenai%2Frobustnav/lists"}