{"id":28753229,"url":"https://github.com/google-deepmind/streetlearn","last_synced_at":"2025-07-16T12:40:51.249Z","repository":{"id":60256676,"uuid":"161509173","full_name":"google-deepmind/streetlearn","owner":"google-deepmind","description":"A C++/Python implementation of the StreetLearn environment based on images from Street View, as well as a TensorFlow implementation of goal-driven navigation agents solving the task published in “Learning to Navigate in Cities Without a Map”, NeurIPS 2018","archived":false,"fork":false,"pushed_at":"2020-07-21T12:18:52.000Z","size":328,"stargazers_count":310,"open_issues_count":4,"forks_count":63,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-06-16T21:48:32.424Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://streetlearn.cc","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-deepmind.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-12T15:40:55.000Z","updated_at":"2025-06-13T21:24:08.000Z","dependencies_parsed_at":"2022-09-27T09:00:22.996Z","dependency_job_id":null,"html_url":"https://github.com/google-deepmind/streetlearn","commit_stats":null,"previous_names":["google-deepmind/streetlearn","deepmind/streetlearn"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/google-deepmind/streetlearn","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fstreetlearn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fstreetlearn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fstreetlearn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fstreetlearn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-deepmind","download_url":"https://codeload.github.com/google-deepmind/streetlearn/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fstreetlearn/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260268635,"owners_count":22983601,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-17T00:39:14.046Z","updated_at":"2025-06-17T00:39:14.974Z","avatar_url":"https://github.com/google-deepmind.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# StreetLearn\n\n## Overview\n\nThis repository contains an implementation of the\n[**StreetLearn**](http://streetlearn.cc) C++ engine and Python environment for training navigation agents in real-world photographic street environments, as well as code for implementing the agents used in [1] [\"Learning to Navigate in Cities Without a Map\"](http://papers.nips.cc/paper/7509-learning-to-navigate-in-cities-without-a-map) (NeurIPS 2018). This environment was also used in two follow-up papers: [2] [\"Cross-View Policy Learning for Street Navigation\"](https://arxiv.org/pdf/1906.05930) (ICCV 2019) and [3] [\"Learning to follow directions in Street View\"](https://arxiv.org/pdf/1903.00401) (AAAI 2020), as well as in technical report [4] [\"The StreetLearn Environment and Dataset\"](https://arxiv.org/pdf/1903.01292). The StreetLearn environment relies on panorama images from\n[Google Street View](https://maps.google.com) and provides an interface for\nmoving a first-person view agent inside the Street View graph. This is not an\nofficially supported Google product. Please cite papers [1] and [4] if you use the code from this repository in your work.\n\nOur papers [1], [2] and [3] also provide a detailed description of how to train and implement navigation agents in the StreetLearn environment by using a TensorFlow implementation of \"Importance Weighted Actor-Learner Architectures\", published in Espeholt, Soyer, Munos et al. (2018) [\"IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures\"](https://arxiv.org/abs/1802.01561)). The generic agent and trainer code have been published by Lasse Espeholt under an Apache license at:\n[https://github.com/deepmind/scalable_agent](https://github.com/deepmind/scalable_agent).\n\n### Bibtex\n\n```\n@inproceedings{mirowski2018learning,\n  title={Learning to Navigate in Cities Without a Map},\n  author={Mirowski, Piotr and Grimes, Matthew Koichi and Malinowski, Mateusz and Hermann, Karl Moritz and Anderson, Keith and Teplyashin, Denis and Simonyan, Karen and Kavukcuoglu, Koray and Zisserman, Andrew and Hadsell, Raia},\n  booktitle={Neural Information Processing Systems (NeurIPS)},\n  year={2018}\n}\n\n@article{mirowski2019streetlearn,\n  title={The StreetLearn Environment and Dataset},\n  author={Mirowski, Piotr and Banki-Horvath, Andras and Anderson, Keith and Teplyashin, Denis and Hermann, Karl Moritz and Malinowski, Mateusz and Grimes, Matthew Koichi and Simonyan, Karen and Kavukcuoglu, Koray and Zisserman, Andrew and others},\n  journal={arXiv preprint arXiv:1903.01292},\n  year={2019}\n}\n```\n\n### Code structure\n\nThis environment code contains:\n\n*   **streetlearn/engine** Our C++ StreetLearn engine for loading, caching and\n    serving Google Street View panoramas by projecting them from a\n    equirectangular representation to first-person projected view at a given\n    yaw, pitch and field of view, and for handling navigation (moving from one\n    panorama to another) depending on the city street graph and the current\n    orientation.\n*   **streetlearn/proto** The message\n    [protocol buffer](https://developers.google.com/protocol-buffers/) used to\n    store panoramas and street graph.\n*   **streetlearn/python/environment** A Python-based interface for calling the\n    StreetLearn environment with custom action spaces. Within the Python\n    StreetLearn interface, several games are defined in individual files whose\n    names end with **game.py**. A second interface, called\n    **BatchedStreetLearn**, can be used to instantiate multiple StreetLearn\n    environments that share the same action specs, observation specs, and\n    panorama cache, and return observations in batched format.\n*   **streetlearn/python/ui** A simple interactive **human_agent** and an\n    **oracle_agent** and **instruction_following_oracle_agent** for courier and\n    instruction-following tasks respectively; all agents are implemented in\n    Python using pygame and instantiate the StreetLearn environment on the\n    requested map, along with a simple user interface. The interactive\n    **human_agent** enables a user to play various games. The **oracle_agent**\n    and **instruction_following_oracle_agent** are similar to the human agent\n    and automatically navigate towards the goal (courier game) or towards the\n    goal via waypoints, following instructions (instruction-following game) and\n    they report oracle performance on these tasks. A batched version of th\n    oracle agent can be started using **batched_oracle_agent**.\n\n## Compilation from source\n\n[Bazel](http://bazel.build) is the official build system for StreetLearn. The\nbuild has only been tested running on Ubuntu 18.04.\n\n### Install build prerequisites\n\n```shell\nsudo apt-get install autoconf automake libtool curl make g++ unzip virtualenv python-virtualenv cmake subversion pkg-config libpython-dev libcairo2-dev libboost-all-dev python-pip libssl-dev\npip install setuptools\npip install pyparsing\n```\n\n### Install Protocol Buffers\n\nFor detailed information see:\nhttps://github.com/protocolbuffers/protobuf/blob/master/src/README.md\n\n```shell\ngit clone https://github.com/protocolbuffers/protobuf.git\ncd protobuf\ngit submodule update --init --recursive\n./autogen.sh\n./configure\nmake -j7\nsudo make install\nsudo ldconfig\ncd python\npython setup.py build\nsudo python setup.py install\ncd ../..\n```\n\n### Install CLIF\n\n```shell\ngit clone https://github.com/google/clif.git\ncd clif\n./INSTALL.sh\ncd ..\n```\n\n### Install OpenCV 2.4.13\n\n```shell\nwget https://github.com/opencv/opencv/archive/2.4.13.6.zip\nunzip 2.4.13.6.zip\ncd opencv-2.4.13.6\nmkdir build\ncd build\ncmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..\nmake -j7\nsudo make install\nsudo ldconfig\ncd ../..\n```\n\n### Install Python dependencies\n\n```shell\npip install six\npip install absl-py\npip install inflection\npip install wrapt\npip install numpy\npip install dm-sonnet\npip install tensorflow-gpu\npip install pygame\n```\n\n### Install Bazel\n\n[This page](https://bazel.build/) describes how to install the Bazel build and\ntest tool on your machine. We currently support Bazel versions up to 0.24.0,\nwhose installation instructions are listed on [this page](https://docs.bazel.build/versions/master/install-ubuntu.html) under section `Using the binary installer` (copy-pasted below):\n\n```shell\nwget https://github.com/bazelbuild/bazel/releases/download/0.24.0/bazel-0.24.0-installer-linux-x86_64.sh\nchmod +x bazel-0.24.0-installer-linux-x86_64.sh\n./bazel-0.24.0-installer-linux-x86_64.sh  --user\nexport PATH=\"$PATH:$HOME/bin\"\n```\n\n### Building StreetLearn\n\nClone this repository and install\n[Scalable Agent](https://github.com/deepmind/scalable_agent):\n\n```shell\ngit clone https://github.com/deepmind/streetlearn.git\ncd streetlearn\nsh get_scalable_agent.sh\n```\n\nTo build the StreetLearn engine only:\n\n```shell\nexport CLIF_PATH=$HOME/opt\nbazel build streetlearn:streetlearn_engine_py\n```\n\nTo build the human agent and the oracle agent in the StreetLearn environment,\nwith all the dependencies:\n\n```shell\nexport CLIF_PATH=$HOME/opt\nbazel build streetlearn/python/ui:all\n```\n\n## Running the StreetLearn human agent\n\nTo run the human agent using one of the StreetLearn datasets downloaded and\nstored at **dataset_path** (note that **dataset_path** needs to be absolute, not relative):\n\n```shell\nbazel run streetlearn/python/ui:human_agent -- --dataset_path=\u003cdataset_path\u003e\n```\n\nFor help with the options of the human_agent:\n\n```shell\nbazel run streetlearn/python/ui:human_agent -- --help\n```\n\nSimilarly, to run the oracle agent on the courier game:\n\n```shell\nbazel run streetlearn/python/ui:oracle_agent -- --dataset_path=\u003cdataset_path\u003e\n```\n\nThe human agent and the oracle agent show a **view_image** (on top) and a\n**graph_image** (on bottom).\n\nNote: you need to add the following line to your `.bashrc` script to avoid specifying the CLIF path each time you open a new terminal:\n\n```shell\nexport CLIF_PATH=$HOME/opt\n```\n\n### Actions available to an agent:\n\n*   Rotate left or right in the panorama, by a specified angle (change the yaw\n    of the agent). In the human_agent, press **a** or **d**.\n*   Rotate up or down in the panorama, by a specified angle (change the pitch of\n    the agent). In the human_agent, press **w** or **s**.\n*   Move from current panorama A forward to another panorama B if the current\n    bearing of the agent from A to B is within a tolerance angle of 30 degrees.\n    In the human_agent, press **space**.\n*   Zoom in and out in the panorama. In the human_agent, press **i** or **o**.\n\nAdditional keys for the human_agent are **escape** and **p** (to print the\ncurrent view as a bitmap image).\n\nFor training RL agents, action spaces are discretized using integers. For\ninstance, in our paper, we used 5 actions: (move forward, turn left by 22.5 deg,\nturn left by 67.5 deg, turn right by 22.5 deg, turn right by 67.5 deg).\n\n### Navigation Bar\n\nAlong the bottom of the **view_image** is the navigation bar which displays a\nsmall circle in any direction in which travel is possible:\n\n*   When within the centre range, they will turn green meaning the user can move\n    in this direction.\n*   When they are out of this range, they will turn red meaning this is\n    inaccessible.\n*   When more than one dots are within the centre range, all except the most\n    central will turn orange, meaning that there are multiple (forward)\n    directions available.\n\n### Stop signs\n\nThe graph is constructed by breadth first search to the depth specified by the\ngraph depth flags. At the maximum depth the graph will suddenly stop, generally\nin the middle of a street. Because we are trying to train agents to recognize\nstreets as navigable, and in order not to confuse the agents, red stop signs are\nshown from two panoramas away from any terminal node in the graph.\n\n### Obtaining the StreetLearn dataset\n\nYou can request the StreetLearn dataset on the [StreetLearn project website](https://sites.google.com/view/streetlearn/). The following datasets are currently distributed:\n* 56k Manhattan panoramas, used [1], [2], [3] and [4]:\n** **manhattan_highres** (size 1632 x 408)\n** **manhattan_lowres** (size 408 x 204)\n* 58k Pittsburgh panoramas, used in [2], [3] and [4]:\n** **pittsburgh_highres** (size 1632 x 408)\n** **pittsburgh_lowres** (size 408 x 204)\n* 29k Manhattan panoramas used in [5] [\"TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments\"](http://openaccess.thecvf.com/content_CVPR_2019/papers/Chen_TOUCHDOWN_Natural_Language_Navigation_and_Spatial_Reasoning_in_Visual_Street_CVPR_2019_paper.pdf) (Chen, Suhr, Misra et al, ICCV 2019), with accompanying code at [https://github.com/lil-lab/touchdown](https://github.com/lil-lab/touchdown):\n** **touchdown_manhattan_highres** (size 3000 x 1500)\n** **touchdown_manhattan_lowres** (downsampled to 500 x 250)\n\nThe downsampled version of the panoramas can be used when the RGB inputs are small (e.g., 84 x 84), to save on panorama image loading and projection time.\n\n## Using the StreetLearn environment code\n\nThe Python StreetLearn environment follows the specifications from [OpenAI Gym](https://gym.openai.com/docs/). The call to function **step(action)** returns:\n* **observation** (tuple of observations requested at construction),\n* **reward** (a float with the current reward of the agent),\n* **done** (boolean indicating whether the episode has ended)\n* and **info** (a dictionary of environment state variables).\nAfter creating the environment, it is initialised by calling function **reset()**. If the flag auto_reset is set to True at construction, **reset()** will be called automatically every time that an episode ends.\n\n### Environment Settings\n\nDefault environment settings are stored in streetlearn/python/default_config.py.\n\n*   **seed**: Random seed.\n*   **width**: Width of the streetview image.\n*   **height**: Height of the streetview image.\n*   **graph_width**: Width of the map graph image.\n*   **graph_height**: Height of the map graph image.\n*   **status_height**: Status bar height in pixels.\n*   **field_of_view**: Horizontal field of view, in degrees.\n*   **min_graph_depth**: Min bound on BFS depth for panos.\n*   **max_graph_depth**: Max bound on BFS depth for panos.\n*   **max_cache_size**: Pano cache size.\n*   **bbox_lat_min**: Minimum value for normalizing the target latitude.\n*   **bbox_lat_max**: Maximum value for normalizing the target latitude.\n*   **bbox_lng_min**: Minimum value for normalizing the target longitude.\n*   **bbox_lng_max**: Maximum value for normalizing the target longitude.\n*   **min_radius_meters**: Minimum distance from goal at which reward shaping\n    starts in the courier game.\n*   **max_radius_meters**: Maximum distance from goal at which reward shaping\n    starts in the courier game.\n*   **timestamp_start_curriculum**: Integer timestamp (UNIX time) when\n    curriculum learning starts, used in the curriculum courier game.\n*   **hours_curriculum_part_1**: Number of hours for the first part of\n    curriculum training (goal location within minimum distance), used in the\n    curriculum courier game.\n*   **hours_curriculum_part_2**: Number of hours for the second part of\n    curriculum training (goal location annealed further away), used in the\n    curriculum courier game.\n*   **min_goal_distance_curriculum**: Distance in meters of the goal location at\n    the beginning of curriculum learning, used in the curriculum courier game.\n*   **max_goal_distance_curriculum**: Distance in meters of the goal location at\n    the beginning of curriculum learning, used in the curriculum courier game.\n*   **instruction_curriculum_type**: Type of curriculum learning, used in the\n    instruction following games.\n*   **frame_cap**: Episode frame cap.\n*   **full_graph**: Boolean indicating whether to build the entire graph upon\n    episode start.\n*   **sample_graph_depth**: Boolean indicating whether to sample graph depth\n    between min_graph_depth and max_graph_depth.\n*   **start_pano**: The pano ID string to start from. The graph will be build\n    out from this point.\n*   **graph_zoom**: Initial graph zoom. Valid between 1 and 32.\n*   **graph_black_on_white**: Show the graph as black on white. Default value: false (shows the graph as white on black).\n*   **show_shortest_path**: Boolean indicator asking whether the shortest path\n    to the goal shall be shown on the graph.\n*   **calculate_ground_truth**: Boolean indicator asking whether the ground\n    truth direction to the goal should be calculated during the game (useful for\n    oracle agents, visualisation and for imitation learning).\n*   **neighbor_resolution**: Used to calculate a binary occupancy vector of\n    neighbors to the current pano.\n*   **color_for_touched_pano**: RGB color for the panos touched by the agent.\n*   **color_for_observer**: RGB color for the observer.\n*   **color_for_coin**: RGB color for the panos containing coins.\n*   **color_for_goal**: RGB color for the goal pano.\n*   **color_for_shortest_path**: RGB color for panos on the shortest path to the\n    goal.\n*   **color_for_waypoint**: RGB color for a waypoint pano.\n*   **observations**: Array containing one or more names of the observations\n    requested from the environment: ['view_image', 'graph_image', 'yaw',\n    'pitch', 'metadata', 'target_metadata', 'latlng', 'target_latlng',\n    'latlng_label', 'target_latlng_label', 'yaw_label', 'neighbors',\n    'thumbnails', 'instructions', 'ground_truth_direction']\n*   **reward_per_coin**: Coin reward for coin game.\n*   **reward_at_waypoint**: Waypoint reward for the instruction-following games.\n*   **reward_at_goal**: Goal reward for the instruction-following games.\n*   **proportion_of_panos_with_coins**: The proportion of panos with coins.\n*   **game_name**: Game name, can be: 'coin_game', 'exploration_game',\n    'courier_game', 'curriculum_courier_game', 'goal_instruction_game',\n    'incremental_instruction_game' and 'step_by_step_instruction_game'.\n*   **action_spec**: Either of 'streetlearn_default', 'streetlearn_fast_rotate',\n    'streetlearn_tilt'\n*   **rotation_speed**: Rotation speed in degrees. Used to create the action\n    spec.\n*   **auto_reset**: Boolean indicator whether games are reset automatically when\n    the max number of frames is achieved.\n\n### Observations\n\nThe following observations can be returned by the agent:\n\n*   **view_image**: RGB image for the first-person view image returned from the\n    environment and seen by the agent,\n*   **graph_image**: RGB image for the top-down street graph image, usually not\n    seen by the agent,\n*   **yaw**: Scalar value of the yaw angle of the agent, in degrees (zero\n    corresponds to North),\n*   **pitch**: Scalar value of the pitch angle of the agent, in degrees (zero\n    corresponds to horizontal),\n*   **metadata**: Message protocol buffer of type Pano with the metadata of the\n    current panorama,\n*   **target_metadata**: Message protocol buffer of type Pano with the metadata\n    of the target/goal panorama,\n*   **latlng**: Tuple of lat/lng scalar values for the current position of the\n    agent,\n*   **target_latlng**: Tuple of lat/lng scalar values for the target/goal\n    position,\n*   **latlng_label**: Integer discretized value of the current agent position\n    using 1024 bins (32 bins for latitude and 32 bins for longitude),\n*   **target_latlng_label**: Integer discretized value of the target position\n    using 1024 bins (32 bins for latitude and 32 bins for longitude),\n*   **yaw_label**: Integer discretized value of the agent yaw using 16 bins,\n*   **neighbors**: Vector of immediate neighbor egocentric traversability grid\n    around the agent, with 16 bins for the directions around the agent and bin 0\n    corresponding to the traversability straight ahead of the agent.\n*   **thumbnails**: Array of n+1 RGB images for the first-person view image\n    returned from the environment, that should be seen by the agent at specific\n    waypoints and goal locations when playing the instruction-following game\n    with n instructions,\n*   **instructions**: List of n string instructions for the agent at specific\n    waypoints and goal locations when playing the instruction-following game\n    with n instructions,\n*   **ground_truth_direction**: Scalar value of the relative ground truth\n    direction to be taken by the agent in order to follow a shortest path to the\n    next goal or waypoint. This observation should be requested only for agents\n    trained using imitation learning.\n\n### Games\n\nThe following games are available in the StreetLearn environment:\n\n*   **coin_game**: invisible coins scattered throughout the map, yielding a\n    reward of 1 for each. Once picked up, these rewards do not reappear until\n    the end of the episode.\n*   **courier_game**: the agent is given a goal destination, specified as\n    lat/long pairs. Once the goal is reached (with 100m tolerance), a new goal\n    is sampled, until the end of the episode. Rewards at a goal are proportional\n    to the number of panoramas on the shortest path from the agent's position\n    when it gets the new goal assignment to that goal position. Additional\n    reward shaping consists in early rewards when the agent gets within a range\n    of 200m of the goal. Additional coins can also be scattered throughout the\n    environment. The proportion of coins, the goal radius and the early reward\n    radius are parametrizable.\n*   **curriculum_courier_game**: same as the courier game, but with a curriculum\n    on the difficulty of the task (maximum straight-line distance from the\n    agent's position to the goal when it is assigned).\n*   **goal_instruction_game** and its variations\n    **incremental_instruction_game** and **step_by_step_instruction_game** use\n    navigation instructions to direct agents to a goal. Agents are provided with\n    a list of instructions as well as thumbnails that guide the agent from its\n    starting position to the goal location. In\n    **step_by_step_instruction_game**, agents are provided one instruction and\n    two thumbnails at a time, in the other game variants the whole list is\n    available throughout the whole game. Reward is granted upon reaching the\n    goal location (all variants), as well as when hitting individual waypoints\n    (**incremental_instruction_game** and **step_by_step_instruction_game**\n    only). During training various curriculum strategies are available to the\n    agents, and reward shaping can be employed to provide fractional rewards\n    when the agent gets within a range of 50m of a waypoint or goal.\n\n## License\n\nThe Abseil C++ library is licensed under the terms of the Apache license. See\n[LICENSE](LICENSE) for more information.\n\n## Disclaimer\n\nThis is not an official Google product.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-deepmind%2Fstreetlearn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-deepmind%2Fstreetlearn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-deepmind%2Fstreetlearn/lists"}