{"id":17154821,"url":"https://github.com/xiaoming-zhao/pointnav-vo","last_synced_at":"2025-04-13T12:52:28.516Z","repository":{"id":45190001,"uuid":"398901643","full_name":"Xiaoming-Zhao/PointNav-VO","owner":"Xiaoming-Zhao","description":"[ICCV 2021] Official implementation of \"The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation\"","archived":false,"fork":false,"pushed_at":"2024-02-23T06:26:51.000Z","size":4890,"stargazers_count":78,"open_issues_count":1,"forks_count":11,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-13T12:52:23.040Z","etag":null,"topics":["computer-vision","deep-learning","embodied-ai","iccv2021","indoor-navigation","visual-odometry"],"latest_commit_sha":null,"homepage":"https://xiaoming-zhao.github.io/projects/pointnav-vo/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Xiaoming-Zhao.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-08-22T21:07:48.000Z","updated_at":"2025-04-02T09:09:59.000Z","dependencies_parsed_at":"2022-08-04T02:00:15.758Z","dependency_job_id":null,"html_url":"https://github.com/Xiaoming-Zhao/PointNav-VO","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xiaoming-Zhao%2FPointNav-VO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xiaoming-Zhao%2FPointNav-VO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xiaoming-Zhao%2FPointNav-VO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xiaoming-Zhao%2FPointNav-VO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Xiaoming-Zhao","download_url":"https://codeload.github.com/Xiaoming-Zhao/PointNav-VO/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248717259,"owners_count":21150388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","embodied-ai","iccv2021","indoor-navigation","visual-odometry"],"created_at":"2024-10-14T21:50:03.421Z","updated_at":"2025-04-13T12:52:28.495Z","avatar_url":"https://github.com/Xiaoming-Zhao.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\n\u003ch1 align=\"center\"\u003ePointNav-VO\u003c/h1\u003e\n\u003cp align=\"center\"\u003eThe Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003cb\u003e\u003ca href=\"https://xiaoming-zhao.github.io/projects/pointnav-vo/\"\u003eProject Page\u003c/a\u003e | \u003ca href=\"https://arxiv.org/abs/2108.11550\"\u003ePaper\u003c/a\u003e\u003c/b\u003e\u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"100%\" src=\"media/nav.gif\"/\u003e\n\u003c/p\u003e\n\n## Table of Contents\n\n- [Setup](#setup)\n- [Reproduction](#reproduce)\n- [Plug-and-play](#use-vo-as-a-drop-in-module)\n- [Train](#train-your-own-vo)\n- [Citation](#citation)\n\n## Setup\n\n### Install Dependencies\n\n```bash\nconda env create -f environment.yml\n```\n\n### Install Habitat\n\nThe repo is tested under the following commits of [habitat-lab](https://github.com/facebookresearch/habitat-lab) and [habitat-sim](https://github.com/facebookresearch/habitat-sim).\n```bash\nhabitat-lab == d0db1b55be57abbacc5563dca2ca14654c545552\nhabitat-sim == 020041d75eaf3c70378a9ed0774b5c67b9d3ce99\n```\n\nNote, to align with Habitat Challenge 2020 settings (see Step 36 in [the Dockerfile](https://hub.docker.com/layers/fairembodied/habitat-challenge/testing_2020_habitat_base_docker/images/sha256-761ca2230667add6ab241a0eaff16984dc271486ec659984ae13ccab57a9c52b?context=explore)), when installing `habitat-sim`, we compiled without CUDA support as\n```bash\npython setup.py install --headless\n```\n\nThere was a discrepancy between noises models in CPU and CPU versions which has now been fixed, see [this issue](https://github.com/facebookresearch/habitat-sim/pull/987). Therefore, to reproduce the results in the paper with our pre-trained weights, you need to use noises model of CPU-version.\n\n### Download Data\n\nWe need two datasets to enable running of this repo:\n1. [Gibson scene dataset](https://github.com/StanfordVL/GibsonEnv/blob/f474d9e/README.md#database)\n2. [PointGoal Navigation splits](https://github.com/facebookresearch/habitat-lab/blob/d0db1b5/README.md#task-datasets), we need `pointnav_gibson_v2.zip`.\n\nPlease follow [Habitat's instruction](https://github.com/facebookresearch/habitat-lab/blob/d0db1b5/README.md#task-datasets) to download them. We assume all data is put under `./dataset` with structure:\n```\n.\n+-- dataset\n|  +-- Gibson\n|  |  +-- gibson\n|  |  |  +-- Adrian.glb\n|  |  |  +-- Adrian.navmesh\n|  |  |  ...\n|  +-- habitat_datasets\n|  |  +-- pointnav\n|  |  |  +-- gibson\n|  |  |  |  +-- v2\n|  |  |  |  |  +-- train\n|  |  |  |  |  +-- val\n|  |  |  |  |  +-- valmini\n```\n\n## Reproduce\n\nDownload pretrained checkpoints of RL navigation policy and VO from [the release page](https://github.com/Xiaoming-Zhao/PointNav-VO/releases) or [this link](https://drive.google.com/drive/folders/1HG_d-PydxBBiDSnqG_GXAuG78Iq3uGdr?usp=sharing). Put them under `pretrained_ckpts` with the following structure:\n```\n.\n+-- pretrained_ckpts\n|  +-- rl\n|  |  +-- no_tune\n|  |  |  +-- rl_no_tune.pth\n|  |  +-- tune_vo\n|  |  |  +-- rl_tune_vo.pth\n|  +-- vo\n|  |  +-- act_forward.pth\n|  |  +-- act_left_right_inv_joint.pth\n```\n\nRun the following command to reproduce navigation results. On Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz and a Nvidia GeForce GTX 1080 Ti, it takes around 4.5 hours to complete evaluation on all 994 episodes with navigation policy tuned with VO.\n```bash\ncd /path/to/this/repo\nexport POINTNAV_VO_ROOT=$PWD\n\nexport NUMBA_NUM_THREADS=1 \u0026\u0026 \\\nexport NUMBA_THREADING_LAYER=workqueue \u0026\u0026 \\\nconda activate pointnav-vo \u0026\u0026 \\\npython ${POINTNAV_VO_ROOT}/launch.py \\\n--repo-path ${POINTNAV_VO_ROOT} \\\n--n_gpus 1 \\\n--task-type rl \\\n--noise 1 \\\n--run-type eval \\\n--addr 127.0.1.1 \\\n--port 8338\n```\n\n## Use VO as a Drop-in Module\n\nWe provide a class `BaseRLTrainerWithVO` that contains all necessary functions to compute odometry in [base_trainer_with_vo.py](./pointnav_vo/rl/common/base_trainer_with_vo.py). Specifically, you can use `_compute_local_delta_states_from_vo` to compute odometry based on adjacent observations. The code sturcture will be something like:\n```python\nlocal_delta_states = _compute_local_delta_states_from_vo(prev_obs, cur_obs, action)\ncur_goal = compute_goal_pos(prev_goal, local_delta_states)\n```\n\nTo get more sense about how to use this function, please refer to [challenge2020_agent.py](./challenge_2020/challenge2020_agent.py), which is the agent we used in [HabitatChallenge 2020](https://eval.ai/web/challenges/challenge-page/580/leaderboard/1631#leaderboardrank-1).\n\n## Train Your Own VO\n\nSee details in [TRAIN.md](./TRAIN.md)\n\n## Citation\n\nPlease cite the following papers if you found our model useful. Thanks!\n\n\u003eXiaoming Zhao, Harsh Agrawal, Dhruv Batra, and Alexander Schwing. The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation. ICCV 2021.\n```\n@inproceedings{ZhaoICCV2021,\n  title={{The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation}},\n  author={Xiaoming Zhao and Harsh Agrawal and Dhruv Batra and Alexander Schwing},\n  booktitle={Proc. ICCV},\n  year={2021},\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiaoming-zhao%2Fpointnav-vo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxiaoming-zhao%2Fpointnav-vo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiaoming-zhao%2Fpointnav-vo/lists"}