{"id":20337928,"url":"https://github.com/gt-ripl/robo-vln","last_synced_at":"2025-04-22T16:58:44.717Z","repository":{"id":104529725,"uuid":"360163581","full_name":"GT-RIPL/robo-vln","owner":"GT-RIPL","description":"Pytorch code for ICRA'21 paper: \"Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation\"","archived":false,"fork":false,"pushed_at":"2024-06-27T18:20:32.000Z","size":34876,"stargazers_count":78,"open_issues_count":1,"forks_count":8,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-29T16:51:15.707Z","etag":null,"topics":["artificial-intelligence","bert","computer-vision","deep-learning","deep-neural-networks","habitat-api","habitat-sim","language","navigation","python","pytorch","robotics","supervised-learning","transformers","vision-and-language","vision-and-language-navigation"],"latest_commit_sha":null,"homepage":"https://zubair-irshad.github.io/projects/robo-vln.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GT-RIPL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-21T12:33:55.000Z","updated_at":"2025-03-15T06:36:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"a6b71522-a018-4643-99f5-26617751efab","html_url":"https://github.com/GT-RIPL/robo-vln","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GT-RIPL%2Frobo-vln","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GT-RIPL%2Frobo-vln/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GT-RIPL%2Frobo-vln/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GT-RIPL%2Frobo-vln/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GT-RIPL","download_url":"https://codeload.github.com/GT-RIPL/robo-vln/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250284768,"owners_count":21405295,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","bert","computer-vision","deep-learning","deep-neural-networks","habitat-api","habitat-sim","language","navigation","python","pytorch","robotics","supervised-learning","transformers","vision-and-language","vision-and-language-navigation"],"created_at":"2024-11-14T21:10:53.366Z","updated_at":"2025-04-22T16:58:44.696Z","avatar_url":"https://github.com/GT-RIPL.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation\n\u003cimg src=\"demo/Pytorch_logo.png\" width=\"10%\"\u003e [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/hierarchical-cross-modal-agent-for-robotics-1/vision-and-language-navigation-on-robo-vln)](https://paperswithcode.com/sota/vision-and-language-navigation-on-robo-vln?p=hierarchical-cross-modal-agent-for-robotics-1)\n\nThis repository is the pytorch implementation of our paper:\n\n**Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation**\u003cbr\u003e\n[__***Muhammad Zubair Irshad***__](https://zubairirshad.com), [Chih-Yao Ma](https://chihyaoma.github.io/), [Zsolt Kira](https://www.cc.gatech.edu/~zk15/) \u003cbr\u003e\nInternational Conference on Robotics and Automation (ICRA), 2021\u003cbr\u003e\n\n[[Project Page](https://zubair-irshad.github.io/projects/robo-vln.html)] [[arXiv](https://arxiv.org/abs/2104.10674)] [[GitHub](https://github.com/GT-RIPL/robo-vln)] \n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"demo/ACMI_final.jpg\" width=\"100%\"\u003e\n\u003c/p\u003e\n\n## Citation\n\nIf you find this repository useful, please cite our paper:\n\n```\n@inproceedings{irshad2021hierarchical,\n  title={Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation},\n  author={Muhammad Zubair Irshad and Chih-Yao Ma and Zsolt Kira},\n  booktitle={Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)},\n  year={2021}\n  }\n\n@INPROCEEDINGS{irshad2022sasra,\n        author={Irshad, Muhammad Zubair and Chowdhury Mithun, Niluthpol and Seymour, Zachary and Chiu, Han-Pang and Samarasekera, Supun and Kumar, Rakesh},\n        booktitle={2022 26th International Conference on Pattern Recognition (ICPR)}, \n        title={Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environments}, \n        year={2022},\n        volume={},\n        number={},\n        pages={4065-4071},\n        keywords={Visualization;Three-dimensional displays;Navigation;Semantics;Natural languages;Transformers;Feature extraction},\n        doi={10.1109/ICPR56361.2022.9956561}}\n```\n\n## Installation \n\nClone the current repository and required submodules:\n\n```bash\ngit clone https://github.com/GT-RIPL/robo-vln\ncd robo-vln\n  \nexport robovln_rootdir=$PWD\n    \ngit submodule init \ngit submodule update\n```\n\n### Habitat and Other Dependencies\n\nInstall `robo-vln` dependencies as follows:\n```bash\nconda create -n habitat python=3.6 cmake=3.14.0\ncd $robovln_rootdir\npython -m pip install -r requirements.txt\n```\n\nWe use modified versions of [Habitat-Sim](https://github.com/facebookresearch/habitat-sim) and [Habitat-API](https://github.com/facebookresearch/habitat-lab) to support continuous control/action-spaces in Habitat Simulator. The details regarding continuous action spaces and converting discrete VLN dataset into continuous control formulation can be found in our [paper](https://arxiv.org/pdf/2104.10674.pdf). The specific commits of our modified [Habitat-Sim](https://github.com/facebookresearch/habitat-sim) and [Habitat-API](https://github.com/facebookresearch/habitat-lab) versions are mentioned below.\n\n```bash\t\n# installs both habitat-api and habitat_baselines\ncd $robovln_rootdir/environments/habitat-lab\npython -m pip install -r requirements.txt\npython -m pip install -r habitat_baselines/rl/requirements.txt\npython -m pip install -r habitat_baselines/rl/ddppo/requirements.txt\npython setup.py develop --all\n\t\n# Install habitat-sim\ncd $robovln_rootdir/environments/habitat-sim\npython setup.py install --headless --with-cuda\n```\n\n### Data\n\nSimilar to [Habitat-API](https://github.com/facebookresearch/habitat-lab), we expect a `data` folder (or symlink) with a particular structure in the top-level directory of this project.\n\n#### Matterport3D\n\nWe utilize Matterport3D (MP3D) photo-realistic scene reconstructions to train and evaluate our agent. A total of 90 Matterport3D scenes are used for `robo-vln`. Here is the official Matterport3D Dataset download link and associated instructions: [project webpage](https://niessner.github.io/Matterport/). To download the scenes needed for `robo-vln`, run the following commands:\n\n```bash\n# requires running with python 2.7\npython download_mp.py --task habitat -o data/scene_datasets/mp3d/\n```\nExtract this data to `data/scene_datasets/mp3d` such that it has the form `data/scene_datasets/mp3d/{scene}/{scene}.glb`. \n\n#### Dataset\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"demo/GIF.gif\" width=\"100%\"\u003e\n\u003c/p\u003e\n\nThe Robo-VLN dataset is a continuous control formualtion of the VLN-CE dataset by [Krantz et al](https://arxiv.org/pdf/2004.02857.pdf) ported over from Room-to-Room (R2R) dataset created by [Anderson et al](http://openaccess.thecvf.com/content_cvpr_2018/papers/Anderson_Vision-and-Language_Navigation_Interpreting_CVPR_2018_paper.pdf). The details regarding converting discrete VLN dataset into continuous control formulation can be found in our [paper](https://arxiv.org/pdf/2104.10674.pdf). \n\n| Dataset \t| Path to extract              \t| Size  \t|\n|--------------\t|----------------------------\t|-------\t|\n| [robo_vln_v1.zip](https://www.dropbox.com/s/1e339avr84tfrbt/robo_vln_v1.zip?dl=1) \t| `data/datasets/robo_vln_v1`          \t| 76.9 MB \t|\n\n#### Robo-VLN Dataset\n\nThe dataset `robo_vln_v1` contains the `train`, `val_seen`, and `val_unseen` splits. \n\n* train: 7739 episodes\n* val_seen: 570 episodes\n* val_unseen: 1224 episodes\n\nFormat of `{split}.json.gz`\n\n```\n{\n    'episodes' = [\n        {\n            'episode_id': 4991,\n            'trajectory_id': 3279,\n            'scene_id': 'mp3d/JeFG25nYj2p/JeFG25nYj2p.glb',\n            'instruction': {\n                'instruction_text': 'Walk past the striped area rug...',\n                'instruction_tokens': [2384, 1589, 2202, 2118, 133, 1856, 9]\n            },\n            'start_position': [10.257800102233887, 0.09358400106430054, -2.379739999771118],\n            'start_rotation': [0, 0.3332950713608026, 0, 0.9428225683587541],\n            'goals': [\n                {\n                    'position': [3.360340118408203, 0.09358400106430054, 3.07817006111145], \n                    'radius': 3.0\n                }\n            ],\n            'reference_path': [\n                [10.257800102233887, 0.09358400106430054, -2.379739999771118], \n                [9.434900283813477, 0.09358400106430054, -1.3061100244522095]\n                ...\n                [3.360340118408203, 0.09358400106430054, 3.07817006111145],\n            ],\n            'info': {'geodesic_distance': 9.65537166595459},\n        },\n        ...\n    ],\n    'instruction_vocab': [\n        'word_list': [..., 'orchids', 'order', 'orient', ...],\n        'word2idx_dict': {\n            ...,\n            'orchids': 1505,\n            'order': 1506,\n            'orient': 1507,\n            ...\n        },\n        'itos': [..., 'orchids', 'order', 'orient', ...],\n        'stoi': {\n            ...,\n            'orchids': 1505,\n            'order': 1506,\n            'orient': 1507,\n            ...\n        },\n        'num_vocab': 2504,\n        'UNK_INDEX': 1,\n        'PAD_INDEX': 0,\n    ]\n}\n```\n* Format of `{split}_gt.json.gz`\n\n```\n{\n    '4991': {\n        'actions': [\n          ...\n          [-0.999969482421875, 1.0],\n          [-0.9999847412109375, 0.15731772780418396],\n          ...\n          ],\n        'forward_steps': 325,\n        'locations': [\n            [10.257800102233887, 0.09358400106430054, -2.379739999771118],\n            [10.257800102233887, 0.09358400106430054, -2.379739999771118],\n            ...\n            [-12.644463539123535, 0.1518409252166748, 4.2241311073303220]\n        ]\n    }\n    ...\n}\n```\n#### Depth Encoder Weights\nSimilar to [VLN-CE](https://arxiv.org/pdf/2004.02857.pdf), our learning-based models utilizes a depth encoder pretained on a large-scale point-goal navigation task i.e. [DDPPO](https://arxiv.org/abs/1911.00357). We utilize depth pretraining by using the DDPPO features from the ResNet50 from the original paper. The pretrained network can be downloaded [here](https://dl.fbaipublicfiles.com/habitat/data/baselines/v1/ddppo/ddppo-models.zip). Extract the contents of `ddppo-models.zip` to `data/ddppo-models/{model}.pth`.\n\n## Training and reproducing results\n\n### \nWe use `run.py` script to train and evaluate all of our baseline models. Use `run.py` along with a configuration file and a run type (either `train` or `eval`) to train or evaluate:\n```bash\npython run.py --exp-config path/to/config.yaml --run-type {train | eval}\n```\nFor lists of modifiable configuration options, see the default [task config](habitat_extensions/config/default.py) and [experiment config](robo_vln_baselines/config/default.py) files.\n\n### Evaluating Models\nAll models can be evaluated using `python run.py --exp-config path/to/config.yaml --run-type eval`. The relevant config entries for evaluation are:\n\n```bash\nEVAL_CKPT_PATH_DIR  # path to a checkpoint or a directory of checkpoints\nEVAL.USE_CKPT_CONFIG  # if True, use the config saved in the checkpoint file\nEVAL.SPLIT  # which dataset split to evaluate on (typically val_seen or val_unseen)\nEVAL.EPISODE_COUNT  # how many episodes to evaluate\n```\nIf `EVAL.EPISODE_COUNT` is equal to or greater than the number of episodes in the evaluation dataset, all episodes will be evaluated. If `EVAL_CKPT_PATH_DIR` is a directory, one checkpoint will be evaluated at a time. If there are no more checkpoints to evaluate, the script will poll the directory every few seconds looking for a new one. Each config file listed in the next section is capable of both training and evaluating the model it is accompanied by.\n\n\n### Off-line Data Buffer\nAll our models require an off-line data buffer for training. To collect the continuous control dataset for both `train` and `val_seen` splits, run the following commands before training (Please note that it would take some time on a single GPU to store data. Please also make sure to dedicate around ~1.5 TB of hard-disk space for data collection):\n\n`Collect data buffer for train split:`\n```bash\npython run.py --exp-config robo_vln_baselines/config/paper_configs/robovln_data_train.yaml --run-type train\n```\n\n`Collect data buffer for val_seen split:`\n```bash\npython run.py --exp-config robo_vln_baselines/config/paper_configs/robovln_data_val.yaml --run-type train \n```\n\n### CUDA\nWe use 2 GPUs to train our Hierarchical Model [hierarchical_cma.yaml](robo_vln_baselines/config/paper_configs/hierarchical_cma.yaml). To train the hierarchical model, dedicate 2 GPUs for training as follows:\n\n```bash\nCUDA_VISIBLE_DEVICES=0,1 python run.py --exp-config robo_vln_baselines/config/paper_configs/hierarchical_cma.yaml --run-type train\n```\n\n## Models/Results From the Paper\n\n| Model              | val_seen SPL | val_unseen SPL | Config                                                                                                                                                                                   |\n|--------------------|--------------|----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Seq2Seq            | 0.34         | 0.30           | [seq2seq_robo.yaml](robo_vln_baselines/config/paper_configs/seq2seq_robo.yaml)                                                                                                                        |\n| PM                 | 0.27         | 0.24           | [seq2seq_robo_pm.yaml](robo_vln_baselines/config/paper_configs/seq2seq_robo_pm.yaml)                                                                                                                  |\n| CMA                | 0.25         | 0.25           | [cma.yaml](robo_vln_baselines/config/paper_configs/cma.yaml)                                                                                                                  |\n| **HCM (Ours)**     | 0.43         | 0.40           | [hierarchical_cma.yaml](robo_vln_baselines/config/paper_configs/hierarchical_cma.yaml)                                                                                                                  |\n\n\n|         |  Legend                                                                                                                                               |\n|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Seq2Seq | [Sequence-to-Sequence](https://github.com/jacobkrantz/VLN-CE). Please see our [paper](https://arxiv.org/pdf/2104.10674.pdf) on modification made to the model to match the continuous action spaces in robo-vln                                                                                                                                                          |\n| PM      | [Progress monitor](https://github.com/chihyaoma/selfmonitoring-agent)                                                                                 |\n| CMA     | [Cross-Modal Attention model](https://github.com/jacobkrantz/VLN-CE). Please see our [paper](https://arxiv.org/pdf/2104.10674.pdf) on modification made to the model to match the continuous action spaces in robo-vln                                                                                                                                                          |\n| **HCM**     | Hierarchical Cross-Modal Agent Module (The proposed hierarchical VLN model from our [paper](https://arxiv.org/pdf/2104.10674.pdf)).                                                                                                                                                                 |\n\n### Pretrained Model\n\nWe provide pretrained model for our best Hierarchical Cross-Modal Agent ([HCM](https://www.dropbox.com/s/4v8bqks7a8jgjzz/HCM_Agent.pth?dl=1)). Pre-trained Model can be downloaded as follows:\n\n| Pre-trained Model \t| Size  \t|\n|--------------\t|-------\t|\n| [HCM_Agent.pth](https://www.dropbox.com/s/4v8bqks7a8jgjzz/HCM_Agent.pth?dl=1) \t| 691 MB \t|\n\n## Acknowledgments\n* This code is built upon the implementation from [VLN-CE](https://github.com/jacobkrantz/VLN-CE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgt-ripl%2Frobo-vln","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgt-ripl%2Frobo-vln","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgt-ripl%2Frobo-vln/lists"}