{"id":19401141,"url":"https://github.com/google-research/snap","last_synced_at":"2026-03-15T21:18:54.148Z","repository":{"id":203808309,"uuid":"700774632","full_name":"google-research/snap","owner":"google-research","description":"SNAP: Self-supervised Neural Maps for Visual Positioning and Semantic Understanding (NeurIPS 2023)","archived":false,"fork":false,"pushed_at":"2023-12-14T10:44:32.000Z","size":583,"stargazers_count":187,"open_issues_count":3,"forks_count":18,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-03T01:01:56.846Z","etag":null,"topics":["3d-mapping","deep-learning","pose-estimation","self-supervised-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-10-05T09:14:06.000Z","updated_at":"2025-02-05T06:40:49.000Z","dependencies_parsed_at":"2023-12-14T12:19:47.925Z","dependency_job_id":null,"html_url":"https://github.com/google-research/snap","commit_stats":null,"previous_names":["google-research/snap"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fsnap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fsnap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fsnap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fsnap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/snap/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250582783,"owners_count":21453913,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-mapping","deep-learning","pose-estimation","self-supervised-learning"],"created_at":"2024-11-10T11:17:19.472Z","updated_at":"2026-03-15T21:18:49.116Z","avatar_url":"https://github.com/google-research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ch1 align=\"center\"\u003eSNAP!\u003cbr\u003e\u003cins\u003eS\u003c/ins\u003eelf-Supervised \u003cins\u003eN\u003c/ins\u003eeural M\u003cins\u003eap\u003c/ins\u003es\u003cbr\u003efor Visual Positioning and Semantic Understanding\u003c/h1\u003e\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://psarlin.com/\"\u003ePaul-Edouard\u0026nbsp;Sarlin\u003c/a\u003e\n    ·\n    \u003ca href=\"https://etrulls.github.io/\"\u003eEduard\u0026nbsp;Trulls\u003c/a\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://www.microsoft.com/en-us/research/people/mapoll/\"\u003eMarc\u0026nbsp;Pollefeys\u003c/a\u003e\n    ·\n    \u003ca href=\"https://scholar.google.com/citations?user=qsB2vcgAAAAJ\u0026hl=en\"\u003eJan\u0026nbsp;Hosang\u003c/a\u003e\n    ·\n    \u003ca href=\"https://scholar.google.com/citations?hl=en\u0026user=RQip5VgAAAAJ\"\u003eSimon\u0026nbsp;Lynen\u003c/a\u003e\n  \u003c/p\u003e\n  \u003ch2 align=\"center\"\u003e\n    \u003ca href=\"https://arxiv.org/pdf/2306.05407.pdf\" align=\"center\"\u003eNeurIPS 2023 Paper\u003c/a\u003e\n    | \u003ca href=\"https://youtu.be/xogrwdgNQdo\" align=\"center\"\u003eVideo\u003c/a\u003e\n    | \u003ca href=\"https://psarlin.com/doc/SNAP_NeurIPS2023_poster_compressed.pdf\" align=\"center\"\u003ePoster\u003c/a\u003e\n  \u003c/h2\u003e\n  \n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://arxiv.org/pdf/2306.05407.pdf\"\u003e\u003cimg src=\"doc/teaser_web.svg\" alt=\"teaser\" width=80%\u003e\u003c/a\u003e\n    \u003cbr\u003e\n    \u003cem\u003eSNAP estimates 2D neural maps from multi-modal data like StreetView and aeral imagery.\n    \u003cbr\u003eNeural maps learn easily interpretable, high-level semantics through self-supervision alone\u003cbr\u003eand can be used for geometric and semantic tasks.\u003c/em\u003e\n\u003c/p\u003e\n\n##\n\nThis repository hosts the training and inference code for SNAP, a deep neural network that turns multi-modal imagery into rich 2D neural maps.\nSNAP was trained on a large dataset of 50M StreetView images with associated camera poses and aerial views.\n**We do not release this dataset and the trained models, so this code is provided solely as a reference and cannot be used as is to reproduce any result of the paper.**\n\n## Usage\n\nThe project requires Python \u003e= 3.10 and is based on [Jax](https://github.com/google/jax) and [Scenic](https://github.com/google-research/scenic). All dependencies are listed in [`requirements.txt`](./requirements.txt). \n\n- The data is stored as TensorFlow dataset and loaded in `snap/data/loader.py`.\n- Train SNAP with self-supervision:\n```bash\npython -m snap.train --config=snap/configs/train_localization.py \\\n    --config.batch_size=32 \\\n    --workdir=train_snap_sv+aerial\n```\n- Evaluate SNAP for visual positioning:\n```bash\npython -m snap.evaluate --config=snap/configs/eval_localization.py \\\n    --config.workdir=train_snap_sv+aerial \\\n    --workdir=.  # unused\n```\n- Fine-tune SNAP for semantic mapping:\n```bash\npython -m snap.train --config=snap/configs/train_semantics.py \\\n    --config.batch_size=32 \\\n    --config.model.bev_mapper.pretrained_path=train_snap_sv+aerial \\\n    --workdir=train_snap_sv+aerial_semantics\n```\n- Evaluate the semantic mapping:\n```bash\npython -m snap.evaluate --config=snap/configs/eval_semantics.py \\\n    --config.workdir=train_snap_sv+aerial_semantics \\\n    --workdir=.  # unused\n```\n\n## BibTeX citation\nIf you use any ideas from the paper or code from this repo, please consider citing:\n\n```bibtex\n@inproceedings{sarlin2023snap,\n  author    = {Paul-Edouard Sarlin and\n               Eduard Trulls and\n               Marc Pollefeys and\n               Jan Hosang and\n               Simon Lynen},\n  title     = {{SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding}},\n  booktitle = {NeurIPS},\n  year      = {2023}\n}\n```\n\n*This is not an officially supported Google product.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fsnap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Fsnap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fsnap/lists"}