{"id":20789489,"url":"https://github.com/real-stanford/sscnav","last_synced_at":"2025-05-05T19:44:41.482Z","repository":{"id":67469580,"uuid":"345273507","full_name":"real-stanford/SSCNav","owner":"real-stanford","description":"[ICRA 2021] SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation","archived":false,"fork":false,"pushed_at":"2021-03-09T09:37:36.000Z","size":640,"stargazers_count":43,"open_issues_count":4,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-30T23:11:10.736Z","etag":null,"topics":["computer-vision","navigation","object-goal-navigation","robotics"],"latest_commit_sha":null,"homepage":"https://sscnav.cs.columbia.edu/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/real-stanford.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-03-07T06:18:35.000Z","updated_at":"2025-01-17T19:37:52.000Z","dependencies_parsed_at":"2023-04-15T22:03:29.081Z","dependency_job_id":null,"html_url":"https://github.com/real-stanford/SSCNav","commit_stats":null,"previous_names":["real-stanford/sscnav"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2FSSCNav","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2FSSCNav/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2FSSCNav/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2FSSCNav/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/real-stanford","download_url":"https://codeload.github.com/real-stanford/SSCNav/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252564192,"owners_count":21768609,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","navigation","object-goal-navigation","robotics"],"created_at":"2024-11-17T15:24:25.358Z","updated_at":"2025-05-05T19:44:41.462Z","avatar_url":"https://github.com/real-stanford.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation\n\n[Yiqing Liang](https://yiqing-liang.netlify.app/)\u003csup\u003e1\u003c/sup\u003e,\n[Boyuan Chen](http://www.cs.columbia.edu/~bchen/)\u003csup\u003e1\u003c/sup\u003e,\n[Shuran Song](https://www.cs.columbia.edu/~shurans/)\u003csup\u003e1\u003c/sup\u003e\n\u003cbr\u003e\n\u003csup\u003e1\u003c/sup\u003eColumbia University\n\u003cbr\u003e\n[ICRA 2021](http://www.icra2021.org/)\n\n### [Project Page](https://sscnav.cs.columbia.edu/) | [Video](https://youtu.be/tfBbdGS72zg) | [arXiv](https://arxiv.org/pdf/2012.04512)\n\n## Overview\nPyTorch implementation for paper \"SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation\".\n\n![teaser](https://sscnav.cs.columbia.edu/images/teaser.gif)\n\n## Content\n\n- [Installation](#installation)\n- [Data Preparation](#data-preparation)\n- [Pretrained Models](#pretrained-models)\n- [Usage](#usage)\n\n## Installation\nRecommend working on conda environment, Python 3.7+\n1. Download Matterport3d dataset following instruction [here](https://github.com/niessner/Matterport).\n2. ```conda create -p /path/to/your/env python=3.7```\n3. Install [PyTorch](https://pytorch.org/).\n4. Install Habitat environment following instruction [here](https://github.com/facebookresearch/habitat-lab).\n5. Clone and navigate to this repository.\n6. ```conda install --yes --file requirements.txt --channel default --channel menpo --channel conda-forge```\n\n## Data Preparation\nFor training and testing Semantic Scene Completion Model and Confidence Model, download [data](https://sscnav.cs.columbia.edu/download/SceneCompletionData.zip) (634M):\n```\nSceneCompletionData.zip\n    |-- data_gen_1 \\\n    |-- data_gen_2 - training data for groundtruth semantic segmentation\n    |-- data_gen_3 /\n    |-- test_data_gen_1 \\\n    |-- test_data_gen_2 - testing data for groundtruth semantic segmentation\n    |-- test_data_gen_3 /\n    |-- train - training data for ACNet semantic segmentation output\n    |-- test - testing data for ACNet semantic segmentation output\n```\n## Pretrained Models\nDownload our pretrained models [here](https://sscnav.cs.columbia.edu/download/pretrained.zip) (1.2G).\nModel | Description\n--- | --- \n```cmplt.pth``` | semantic scene completion model weight for groundtruth semantic segmentation\n```conf.pth``` | confidence model weight for groundtruth semantic segmentation\n```cmplt_seg.pth``` | semantic scene completion model weight for ACNet semantic segmentation output\n```conf_seg.pth``` | confidence model weight for ACNet semantic segmentation output\n```final_seg.pth``` | ACNet semantic segmentation model weight\n```final_Q.pth``` | SSCNav navigation model weight\n\n## Usage\n\n### Semantic Scene Completion\n\nThe experiment settings are controlled by file ```configs/config.json```. Each experiment is assigned a unique key in the file. Check the JSON file for details.\n\nMost Important JSON keys:\n* **root_dirs:** determine which training dataset to use.\n* **test_dirs:** determine which dataset is used for evaluation.\n* **layer_infos:** configuration of model structure.\n* **cd_path:** pretrained weight for semantic scene completion model.\n\n\n**Note:** data are provided in the form of list.\nFor example:\n```\n\"root_dirs\": [\"../resized_buffer/data_gen_1\", \"../resized_buffer/data_gen_2\", \"../resized_buffer/data_gen_3\"]    \n\"test_dirs\": [\"../resized_buffer/test_data_gen_1\", \"../resized_buffer/test_data_gen_2\", \"../resized_buffer/test_data_gen_3\"]\n```\n\n#### Training \u0026 Evaluating a Semantic Scene Completion Model\nThe command to train the provided model for groundtruth semantic segmentation:\n```\npython train_cmplt.py train_cmplt_resized\n```\nThe command to train the provided model for ACNet semantic segmentation output:\n```\npython train_cmplt.py train_cmplt_resized_seg\n```\n#### Training \u0026 Evaluating a Confidence Model\nThe command to train the provided model for groundtruth semantic segmentation:\n```\npython train_conf.py train_conf_4\n```\nThe command to train the provided model for ACNet semantic segmentation output:\n```\npython train_conf.py train_conf_seg_6\n```\n### Navigation\n\nNavigation Experiments are controlled by argparse.\n\nHere are some fields that matter most (feel free to check the rest in the code though!):\n* **title:** experiment title, determines where to save the model \u0026 visualization \u0026 log.\n* **config_paths:** configuration file for Habitat, ./configs/agent_train.yaml for training and ./configs/agent_test.yaml for testing.\n* **user_semantics:** True -\u003e use ACNet segmentation, groundtruth otherwise.\n* **cmplt \u0026 cmplt_pretrained:** if semantic completion model is used, must set former to be True and optionally assign initial weight for the model.\n* **conf \u0026 conf_pretrained:** if confidence model is used, must set former to be True and optionally assign initial weight for the model.\n* **save_dir:** place to store model and visualization.\n* **log_dir:** place to store Tensorboard log.\n* **load_json:** dir to the JSON file which stores the evaluation episodes.\n```\nusage: train_agent.py [-h] --title TITLE [--seed SEED] [--device DEVICE]\n                      [--config_paths CONFIG_PATHS] [--flip FLIP]\n                      [--seg_threshold SEG_THRESHOLD] [--pano]\n                      [--user_semantics] [--seg_pretrained SEG_PRETRAINED]\n                      [--cmplt] [--cmplt_pretrained CMPLT_PRETRAINED] [--conf]\n                      [--conf_pretrained CONF_PRETRAINED] [--targets TARGETS]\n                      [--aggregate] [--memory_size MEMORY_SIZE]\n                      [--num_channel NUM_CHANNEL]\n                      [--success_threshold SUCCESS_THRESHOLD]\n                      [--collision_threshold COLLISION_THRESHOLD]\n                      [--ignore IGNORE] [--Q_pretrained Q_PRETRAINED]\n                      [--offset OFFSET] [--floor_threshold FLOOR_THRESHOLD]\n                      [--lr LR] [--momentum MOMENTUM]\n                      [--weight_decay WEIGHT_DECAY] [--gamma GAMMA]\n                      [--batch_size BATCH_SIZE] [--buffer_size BUFFER_SIZE]\n                      [--height HEIGHT] [--area_x AREA_X] [--area_z AREA_Z]\n                      [--h H] [--w W] [--h_new H_NEW] [--w_new W_NEW]\n                      [--max_step MAX_STEP] [--navigable_base NAVIGABLE_BASE]\n                      [--max_transition MAX_TRANSITION]\n                      [--start_replay START_REPLAY]\n                      [--update_target UPDATE_TARGET] [--start_eps START_EPS]\n                      [--end_eps END_EPS] [--fix_transition FIX_TRANSITION]\n                      [--success_reward SUCCESS_REWARD]\n                      [--step_penalty STEP_PENALTY]\n                      [--approach_reward APPROACH_REWARD]\n                      [--collision_penalty COLLISION_PENALTY]\n                      [--save_dir SAVE_DIR] [--save_interval SAVE_INTERVAL]\n                      [--log_dir LOG_DIR] [--train_thin TRAIN_THIN]\n                      [--loss_thin LOSS_THIN] [--train_vis TRAIN_VIS]\n                      [--scene_types SCENE_TYPES] [--double_dqn] [--TAU TAU]\n                      [--soft_update] [--count COUNT] [--preconf]\n                      [--load_json LOAD_JSON] [--checkpoint CHECKPOINT]\n                      [--shortest] [--tsplit TSPLIT] [--new_eval]\n                      [--fake_conf] [--discrete] [--att] [--rc] [--unconf]\n                      [--full_map]\n\n\n```\n#### Training a Navigation Model\nThe command to train the provided Q model:\n```\npython train_agent.py --cmplt \n    --cmplt_pretrained /local/crv/yiqing/result/train_cmplt_resized/17_cd.pth\n    --conf --conf_pretrained /local/crv/yiqing/result/train_conf_4/14_fd.pth               \n```\n#### Evaluating a Navigation Model\nThe command to evaluate the provided Q model with groundtruth semantic segmentation:\n```\npython test_agent.py --Q_pretrained /local/crv/yiqing/result/s_cu_chal_4/60000.pth --cmplt --cmplt_pretrained /local/crv/yiqing/result/train_cmplt_resized/17_cd.pth --conf --conf_pretrained /local/crv/yiqing/result/train_conf_4/14_fd.pth               \n```\nThe command to evaluate the provided Q model with ACNet semantic segmentation output:\n```\npython test_agent.py --user_semantics --seg_pretrained /local/crv/yiqing/result/train_seg/10.pth --Q_pretrained /local/crv/yiqing/result/s_cu_chal_4/60000.pth --cmplt --cmplt_pretrained /local/crv/yiqing/result/train_cmplt_resized_seg/16_cd.pth --conf --conf_pretrained /local/crv/yiqing/result/train_conf_seg_6/6_fd.pth               \n```\n\n## BibTeX\n```\n@inproceedings{liang2021sscnav,\n    title={SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation},\n    author={Liang, Yiqing and Chen, Boyuan and Song, Shuran},\n    booktitle = {Proc. of The International Conference in Robotics and Automation (ICRA)},\n    year={2021}\n}\n```\n\n## License\n\nThis repository is released under the MIT license. See [LICENSE](LICENSE) for additional details.\n\n\n## Acknowledgement\n\nThis work was supported in part by the Amazon Research Award and Columbia School of Engineering.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freal-stanford%2Fsscnav","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freal-stanford%2Fsscnav","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freal-stanford%2Fsscnav/lists"}