{"id":13422320,"url":"https://github.com/allenai/savn","last_synced_at":"2025-10-13T15:58:40.566Z","repository":{"id":50352271,"uuid":"177212861","full_name":"allenai/savn","owner":"allenai","description":"Learning to Learn how to Learn: Self-Adaptive Visual Navigation using Meta-Learning (https://arxiv.org/abs/1812.00971)","archived":false,"fork":false,"pushed_at":"2019-10-22T22:27:22.000Z","size":19226,"stargazers_count":184,"open_issues_count":20,"forks_count":56,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-10-27T23:57:51.929Z","etag":null,"topics":["ai2thor-environment","meta-learning","reinforcement-learning","self-adaptive"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/allenai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-22T21:43:52.000Z","updated_at":"2024-10-16T03:17:49.000Z","dependencies_parsed_at":"2022-08-29T20:21:10.974Z","dependency_job_id":null,"html_url":"https://github.com/allenai/savn","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Fsavn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Fsavn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Fsavn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Fsavn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/allenai","download_url":"https://codeload.github.com/allenai/savn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243730938,"owners_count":20338742,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai2thor-environment","meta-learning","reinforcement-learning","self-adaptive"],"created_at":"2024-07-30T23:00:41.562Z","updated_at":"2025-10-13T15:58:40.475Z","avatar_url":"https://github.com/allenai.png","language":"Python","funding_links":[],"categories":["Learning to Learn How to Learn: Self-Adaptive Visual Navigation using Meta-Learning. CVPR 2019"],"sub_categories":["[few-shot 知乎](https://zhuanlan.zhihu.com/p/58298920)"],"readme":"# Learning to Learn how to Learn: Self-Adaptive Visual Navigation using Meta-Learning\n\nBy Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi and Roozbeh Mottaghi (Oral Presentation at CVPR 2019).\n\n\n[CVPR 2019 Paper ](https://arxiv.org/abs/1812.00971) | [Video](https://www.youtube.com/watch?v=-Ba6ZRMcxEE\u0026feature=youtu.be) | [BibTex](#citing)\n\nIntuition            |  Examples\n:-------------------------:|:-------------------------:\n![](figs/abstract_figure.jpg)  |  ![](figs/qualitative.jpg)\n\nThere is a lot to learn about a task by actually attempting it! Learning is continuous, i.e. we learn as we perform.\nTraditional navigation approaches freeze the model during inference (top row in the intuition figure above). \nIn  this  paper,  we  propose a self-addaptive agent for visual navigation that learns via self-supervised\ninteraction with the environment (bottom row in the intuition figure above).\n\n\n## Citing\n\nIf you find this project useful in your research, please consider citing:\n\n```\n@InProceedings{Wortsman_2019_CVPR,\n  author={Mitchell Wortsman and Kiana Ehsani and Mohammad Rastegari and Ali Farhadi and Roozbeh Mottaghi},\n  title={Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning},\n  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},\n  month = {June},\n  year = {2019}\n}\n```\n\n## Results\n\n\n| Model  | SPL  \u0026geq; 1 | Success  \u0026geq; 1 | SPL   \u0026geq; 5 | Success  \u0026geq; 5 |\n| :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |\n| [SAVN](#SAVN)  |  16.15  \u0026pm; 0.5 | 40.86  \u0026pm; 1.2 | 13.91  \u0026pm; 0.5 | 28.70  \u0026pm; 1.5 |\n| [Scene Priors](https://arxiv.org/abs/1810.06543)  | 15.47  \u0026pm; 1.1 | 35.13  \u0026pm; 1.3 | 11.37  \u0026pm; 1.6 | 22.25  \u0026pm; 2.7 |\n| [Non-Adaptive A3C](#Non-Adaptvie-A3C)  | 14.68  \u0026pm; 1.8 | 33.04  \u0026pm; 3.5 | 11.69  \u0026pm; 1.9 | 21.44  \u0026pm; 3.0 |\n\n\n## Setup\n\n- Clone the repository with `git clone https://github.com/allenai/savn.git \u0026\u0026 cd savn`.\n\n- Install the necessary packages. If you are using pip then simply run `pip install -r requirements.txt`.\n\n- Download the [pretrained models](https://prior-datasets.s3.us-east-2.amazonaws.com/savn/pretrained_models.tar.gz) and\n[data](https://prior-datasets.s3.us-east-2.amazonaws.com/savn/data.tar.gz) to the `savn` directory. Untar with\n```bash\ntar -xzf pretrained_models.tar.gz\ntar -xzf data.tar.gz\n```\n\nThe `data` folder contains:\n\n- `thor_offline_data` which is organized into sub-folders, each of which corresponds to a scene in [AI2-THOR](https://ai2thor.allenai.org/). For each room we have scraped the [ResNet](https://arxiv.org/abs/1512.03385) features of all possible locations in addition to a metadata and [NetworkX](https://networkx.github.io/) graph of possible navigations in the scene.\n- `thor_glove` which contains the [GloVe](https://nlp.stanford.edu/projects/glove/) embeddings for the navigation targets.\n- `gcn` which contains the necessary data for the [Graph Convolutional Network (GCN)](https://arxiv.org/abs/1609.02907) in [Scene Priors](https://arxiv.org/abs/1810.06543), including the adjacency matrix.\n\nNote that the starting positions and scenes for the test and validation set may be found in `test_val_split`.\n\nIf you wish to access the RGB images in addition to the ResNet features, replace `thor_offline_data` with [thor_offlline_data_with_images](https://prior-datasets.s3.us-east-2.amazonaws.com/savn/offline_data_with_images.tar.gz). If you wish to run your model on the image files,\nadd the command line argument `--images_file_name images.hdf5`. \n\n## Evaluation using Pretrained Models\n\nUse the following code to run the pretrained models on the test set. Add the argument `--gpu-ids 0 1` to speed up the evaluation by using GPUs.\n\n#### SAVN\n```bash\npython main.py --eval \\\n    --test_or_val test \\\n    --episode_type TestValEpisode \\\n    --load_model pretrained_models/savn_pretrained.dat \\\n    --model SAVN \\\n    --results_json savn_test.json \n\ncat savn_test.json \n```\n\n#### Scene Priors\n```bash\npython main.py --eval \\\n    --test_or_val test \\\n    --episode_type TestValEpisode \\\n    --load_model pretrained_models/gcn_pretrained.dat \\\n    --model GCN \\\n    --glove_dir ./data/gcn \\\n    --results_json scene_priors_test.json\n\ncat scene_priors_test.json \n```\n\n\n#### Non-Adaptvie-A3C\n```bash\npython main.py --eval \\\n    --test_or_val test \\\n    --episode_type TestValEpisode \\\n    --load_model pretrained_models/nonadaptivea3c_pretrained.dat \\\n    --results_json nonadaptivea3c_test.json\n\ncat nonadaptivea3c_test.json\n```\n\nThe result may vary depending on system and set-up though we obtain:\n\n| Model  | SPL  \u0026geq; 1 | Success  \u0026geq; 1 | SPL   \u0026geq; 5 | Success  \u0026geq; 5 |\n| :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |\n| [SAVN](#SAVN)  |  16.13 | 42.20 | 14.30 | 30.09 |\n| [Scene Priors](https://arxiv.org/abs/1810.06543)  |  14.86 | 36.90 | 11.49 | 24.70 |\n| [Non-Adaptive A3C](#Non-Adaptvie-A3C)  | 14.10 | 32.40 | 10.73 | 19.16 |\n\nThe results in the [initial submission](https://arxiv.org/abs/1812.00971v1) (shown below) were the best (in terms of success on the validation set). After the initial submission, we trained the model 5 times from scratch to obtain error bars, which you may find in [results](#results).\n\n| Model  | SPL  \u0026geq; 1 | Success  \u0026geq; 1 | SPL   \u0026geq; 5 | Success  \u0026geq; 5 |\n| :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |\n| [SAVN](#SAVN)  |  16.13 | 42.10 | 13.19 | 30.54 |\n| [Non-Adaptive A3C](#Non-Adaptvie-A3C)  | 13.73 | 32.90 | 10.88 | 20.66 |\n\n## How to Train your SAVN\n\nYou may train your own models by using the commands below.\n\n#### Training SAVN\n```bash\npython main.py \\\n    --title savn_train \\\n    --model SAVN \\\n    --gpu-ids 0 1 \\\n    --workers 12\n```\n\n\n#### Training Non-Adaptvie A3C\n```bash\npython main.py \\\n    --title nonadaptivea3c_train \\\n    --gpu-ids 0 1 \\\n    --workers 12\n```\n\n\n## How to Evaluate your Trained Model\n\nYou may use the following commands for evaluating models you have trained.\n\n#### SAVN\n```bash\npython full_eval.py \\\n    --title savn \\\n    --model SAVN \\\n    --results_json savn_results.json \\\n    --gpu-ids 0 1\n    \ncat savn_results.json\n```\n\n#### Non-Adaptive A3C\n```bash\npython full_eval.py \\\n    --title nonadaptivea3c \\\n    --results_json nonadaptivea3c_results.json \\\n    --gpu-ids 0 1\n    \ncat nonadaptivea3c_results.json\n```\n\n####  Random Agent\n```bash\npython main.py \\\n    --eval \\\n    --test_or_val test \\\n    --episode_type TestValEpisode \\\n    --title random_test \\\n    --agent_type RandomNavigationAgent \\\n    --results_json random_results.json\n    \ncat random_results.json\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallenai%2Fsavn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fallenai%2Fsavn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallenai%2Fsavn/lists"}