{"id":13442998,"url":"https://github.com/bradyz/cross_view_transformers","last_synced_at":"2026-04-02T13:59:26.813Z","repository":{"id":37273257,"uuid":"475112689","full_name":"bradyz/cross_view_transformers","owner":"bradyz","description":"Cross-view Transformers for real-time Map-view Semantic Segmentation (CVPR 2022 Oral)","archived":false,"fork":false,"pushed_at":"2023-11-06T08:29:41.000Z","size":13674,"stargazers_count":539,"open_issues_count":45,"forks_count":82,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-20T15:41:16.517Z","etag":null,"topics":["cvpr2022","deep-learning","pytorch","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bradyz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-03-28T17:39:38.000Z","updated_at":"2025-03-20T09:54:11.000Z","dependencies_parsed_at":"2024-01-18T14:41:31.032Z","dependency_job_id":null,"html_url":"https://github.com/bradyz/cross_view_transformers","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bradyz/cross_view_transformers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bradyz%2Fcross_view_transformers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bradyz%2Fcross_view_transformers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bradyz%2Fcross_view_transformers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bradyz%2Fcross_view_transformers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bradyz","download_url":"https://codeload.github.com/bradyz/cross_view_transformers/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bradyz%2Fcross_view_transformers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31307435,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T12:59:32.332Z","status":"ssl_error","status_checked_at":"2026-04-02T12:54:48.875Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cvpr2022","deep-learning","pytorch","transformer"],"created_at":"2024-07-31T03:01:54.583Z","updated_at":"2026-04-02T13:59:26.795Z","avatar_url":"https://github.com/bradyz.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# \u003cdiv align=\"center\"\u003e**Cross View Transformers**\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\u003cimg src=\"docs/assets/teaser.jpg\" width=\"65%\"\u003e\u003c/div\u003e\n\u003cbr\u003e\n\nThis repository contains the source code and data for our paper:\n\n\u003e [**Cross-view Transformers for real-time Map-view Semantic Segmentation**](http://www.philkr.net/media/zhou2022crossview.pdf)  \n\u003e [Brady Zhou](https://www.bradyzhou.com/), [Philipp Kr\u0026auml;henb\u0026uuml;hl](http://www.philkr.net/)  \n\u003e [*CVPR 2022*](https://cvpr2022.thecvf.com/)\n\n## \u003cdiv align=\"center\"\u003e**Demos**\u003c/div\u003e\n\n\u003cbr\u003e\n\n\u003cdiv align=\"center\"\u003e\u003cimg src=\"docs/assets/predictions.gif\" width=\"75%\"/\u003e\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n\u003cb\u003eMap-view Segmentation:\u003c/b\u003e\nThe model uses multi-view images to produce a map-view segmentation at 45 FPS\n\u003c/div\u003e\n\u003cbr\u003e\n\n\u003cdiv align=\"center\"\u003e\u003cimg src=\"docs/assets/map.gif\" width=\"40%\"/\u003e\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n\u003cb\u003eMap Making:\u003c/b\u003e\nWith vehicle pose, we can construct a map by fusing model predictions over time\n\u003c/div\u003e\n\u003cbr\u003e\n\n\u003cdiv align=\"center\"\u003e\u003cimg src=\"docs/assets/attention.gif\" width=\"75%\"/\u003e\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n\u003cb\u003eCross-view Attention:\u003c/b\u003e\nFor a given map-view location, we show which image patches are being attended to\n\u003c/div\u003e\n\u003cbr\u003e\n\n## \u003cdiv align=\"center\"\u003e**Installation**\u003c/div\u003e\n\n```bash\n# Clone repo\ngit clone https://github.com/bradyz/cross_view_transformers.git\n\ncd cross_view_transformers\n\n# Setup conda environment\nconda create -y --name cvt python=3.8\n\nconda activate cvt\nconda install -y pytorch torchvision cudatoolkit=11.3 -c pytorch\n\n# Install dependencies\npip install -r requirements.txt\npip install -e .\n```\n\n## \u003cdiv align=\"center\"\u003e**Data**\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\u003cimg src=\"docs/assets/view_data.gif\" width=\"75%\"/\u003e\u003c/div\u003e\n\u003cbr\u003e\n\nDocumentation:\n* [Dataset setup](docs/dataset_setup.md)\n* [Label generation](docs/label_generation.md) (optional)\n\n\u003cbr/\u003e\n\nDownload the original datasets and our generated map-view labels\n\n| | Dataset | Labels |\n| :-- | :-- | :-- |\n| nuScenes | [keyframes + map expansion](https://www.nuscenes.org/nuscenes#download) (60 GB) | [cvt_labels_nuscenes.tar.gz](https://www.cs.utexas.edu/~bzhou/cvt/cvt_labels_nuscenes.tar.gz) (361 MB) |\n| Argoverse 1.1 | [3D tracking](https://www.argoverse.org/av1.html#download-link) | coming soon™ |\n\n\u003cbr/\u003e\n\nThe structure of the extracted data should look like the following\n\n```\n/datasets/\n├─ nuscenes/\n│  ├─ v1.0-trainval/\n│  ├─ v1.0-mini/\n│  ├─ samples/\n│  ├─ sweeps/\n│  └─ maps/\n│     ├─ basemap/\n│     └─ expansion/\n└─ cvt_labels_nuscenes/\n   ├─ scene-0001/\n   ├─ scene-0001.json\n   ├─ ...\n   ├─ scene-1000/\n   └─ scene-1000.json\n```\n\nWhen everything is setup correctly, check out the dataset with\n\n```bash\npython3 scripts/view_data.py \\\n  data=nuscenes \\\n  data.dataset_dir=/media/datasets/nuscenes \\\n  data.labels_dir=/media/datasets/cvt_labels_nuscenes \\\n  data.version=v1.0-mini \\\n  visualization=nuscenes_viz \\\n  +split=val\n```\n\n# \u003cdiv align=\"center\"\u003e**Training**\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003ca href=\"https://www.pytorchlightning.ai\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/docs/source/_static/images/logo.png\" width=\"25%\"\u003e\n\u003c/a\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://wandb.ai/site\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/wandb/client/master/.github/wb-logo-lightbg.png\" width=\"25%\"\u003e\n\u003c/a\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://hydra.cc\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/facebookresearch/hydra/master/website/static/img/Hydra-Readme-logo2.svg\" width=\"15%\"\u003e\n\u003c/a\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n\nAn average job of 50k training iterations takes ~8 hours.  \nOur models were trained using 4 GPU jobs, but also can be trained on single GPU.\n\nTo train a model,\n\n```bash\npython3 scripts/train.py \\\n  +experiment=cvt_nuscenes_vehicle\n  data.dataset_dir=/media/datasets/nuscenes \\\n  data.labels_dir=/media/datasets/cvt_labels_nuscenes\n```\n\nFor more information, see\n\n* `config/config.yaml` - base config\n* `config/model/cvt.yaml` - model architecture\n* `config/experiment/cvt_nuscenes_vehicle.yaml` - additional overrides\n\n## \u003cdiv align=\"center\"\u003e**Additional Information**\u003c/div\u003e\n\n### **Awesome Related Repos**\n\n* https://github.com/wayveai/fiery\n* https://github.com/nv-tlabs/lift-splat-shoot\n* https://github.com/tom-roddick/mono-semantic-maps\n\n### **License**\n\nThis project is released under the [MIT license](LICENSE)\n\n### **Citation**\n\nIf you find this project useful for your research, please use the following BibTeX entry.\n\n```bibtex\n@inproceedings{zhou2022cross,\n    title={Cross-view Transformers for real-time Map-view Semantic Segmentation},\n    author={Zhou, Brady and Kr{\\\"a}henb{\\\"u}hl, Philipp},\n    booktitle={CVPR},\n    year={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbradyz%2Fcross_view_transformers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbradyz%2Fcross_view_transformers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbradyz%2Fcross_view_transformers/lists"}