{"id":13442003,"url":"https://github.com/OpenDriveLab/OpenScene","last_synced_at":"2025-03-20T13:31:54.939Z","repository":{"id":186049928,"uuid":"667662690","full_name":"OpenDriveLab/OpenScene","owner":"OpenDriveLab","description":"3D Occupancy Prediction Benchmark in Autonomous Driving","archived":false,"fork":false,"pushed_at":"2024-05-27T05:13:32.000Z","size":44274,"stargazers_count":339,"open_issues_count":6,"forks_count":21,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-03-13T17:12:50.361Z","etag":null,"topics":["3d-occupancy","autonomous-driving","foundation-model"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenDriveLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.txt","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["OpenDriveLab"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2023-07-18T03:11:06.000Z","updated_at":"2025-03-12T19:12:55.000Z","dependencies_parsed_at":"2024-03-27T09:25:35.613Z","dependency_job_id":"7aa28481-bf48-4997-ae39-b5e24eb3b7ce","html_url":"https://github.com/OpenDriveLab/OpenScene","commit_stats":null,"previous_names":["opendrivelab/openscene"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FOpenScene","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FOpenScene/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FOpenScene/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FOpenScene/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenDriveLab","download_url":"https://codeload.github.com/OpenDriveLab/OpenScene/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244619148,"owners_count":20482369,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-occupancy","autonomous-driving","foundation-model"],"created_at":"2024-07-31T03:01:40.544Z","updated_at":"2025-03-20T13:31:54.932Z","avatar_url":"https://github.com/OpenDriveLab.png","language":"Python","funding_links":["https://github.com/sponsors/OpenDriveLab"],"categories":["Python","Papers","Datasets"],"sub_categories":["2024"],"readme":"\n\u003cdiv id=\"top\" align=\"center\"\u003e\n\n# OpenScene: Autonomous Grand Challenge Toolkits\n\n**The large-scale dataset used for the `End-to-End Driving` and `Predictive World Model` tracks for the [CVPR 2024 Autonomous Grand Challenge](https://opendrivelab.com/challenge2024).**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/challenge.jpeg\" width=\"900px\" \u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\n## News\n\n- **`2024/03/18`** We updated the [test metadata](/docs/getting_started.md#test-set) with box annotations, please re-download it.\n- **`2024/03/01`** OpenScene `v1.1` released, [change log](/docs/getting_started.md#openscene-v11).\n- **`2024/03/01`** We are hosting **CVPR 2024 Autonomous Grand Challenge**.\n\n## Table of Contents\n\n1. [Track: End-to-End Driving at Scale](#navsim)\n2. [Track: Predictive World Model](#worldmodel)\n3. [Dataset: OpenScene](#dataset)\n4. [License and Citation](#license-and-citation)\n5. [Related Resources](#resources)\n\n## Track: End-to-End Driving at Scale \u003ca name=\"navsim\"\u003e\u003c/a\u003e\n\u003cdiv id=\"top\" align=\"center\"\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/e2e_banner.png\" width=\"900px\" \u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\n\u003e - Official website: :globe_with_meridians: [AGC2024](https://opendrivelab.com/challenge2024/#end_to_end_driving_at_scale)\n\u003e - Evaluation server: :hugs: [Hugging Face](https://huggingface.co/spaces/AGC2024-P/e2e-driving-2024)\n\u003e - Development Kit: :ringed_planet: [NAVSIM](https://github.com/autonomousvision/navsim)\n\n- [Problem Formulation](#navsim-baseline)\n- [Evaluation: PDM Score](#navsim-eval)\n- [Submission](#navsim-submission)\n\nNAVSIM gathers simulation-based metrics (such as progress and time to collision) for end-to-end driving by unrolling simplified bird's eye view abstractions of scenes for a short simulation horizon. It operates under the condition that the policy has no influence on the environment, which enables efficient, open-loop metric computation while being better aligned with closed-loop evaluations than traditional displacement errors.\n\n### Problem Formulation \u003ca name=\"navsim-baseline\"\u003e\u003c/a\u003e\nGiven sensor inputs (multi-view images from 8 cameras, LiDAR, ego states, and discrete navigation commands) for a 2-second history, the end-to-end planner must output a safe trajectory for the ego vehicle to navigate along for the next 4 seconds. More information is available in the [NAVSIM docs](https://github.com/kashyap7x/navsim/blob/internal_main/docs/agents.md).\n\n### Evaluation: PDM Score \u003ca name=\"navsim-eval\"\u003e\u003c/a\u003e\nFair comparisons are challenging in the open-loop planning literature, due to metrics of narrow scope or inconsistent definitions between different projects. The PDM Score is a combination of six sub-metrics, which provides a comprehensive analysis of different aspects of driving performance. Five of these sub-metrics are discrete-valued, and one is continuous. All metrics are computed after a 4-second non-reactive simulation of the planner output: background actors follow their recorded future trajectories, and the ego vehicle moves based on an LQR controller. More information is available in the [NAVSIM docs](https://github.com/kashyap7x/navsim/blob/internal_main/docs/metrics.md).\n\n### Submission \u003ca name=\"navsim-submission\"\u003e\u003c/a\u003e\nThe warm-up phase of the challenge has begun! For further details, please check the [Hugging Face](https://huggingface.co/spaces/AGC2024-P/e2e-driving-warmup) warmup server! The main [evaluation server](https://huggingface.co/spaces/AGC2024-P/e2e-driving-2024) will be open soon!\n\n## Track: Predictive World Model \u003ca name=\"worldmodel\"\u003e\u003c/a\u003e\n\u003cdiv id=\"top\" align=\"center\"\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/pred_banner.png\" width=\"900px\" \u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\n\u003e - Official website: :globe_with_meridians: [AGC2024](https://opendrivelab.com/challenge2024/#predictive_world_model)\n\u003e - Evaluation server: :hugs: [Hugging Face](https://huggingface.co/spaces/AGC2024-P/predictive-world-model-2024)\n\n- [Problem Formulation](#worldmodel-baseline)\n- [Evaluation: Chamfer Distance](#worldmodel-eval)\n- [Submission](#worldmodel-submission)\n\nServing as an abstract spatio-temporal representation of reality, the world model can predict future states based on the current state. The learning process of world models has the potential to provide a pre-trained foundation model for autonomous driving. Given vision-only inputs, the neural network outputs point clouds in the future to testify its predictive capability of the world.\n\n### Problem Formulation \u003ca name=\"worldmodel-baseline\"\u003e\u003c/a\u003e\nGiven an visual observation of the world for the past 3 seconds, predict the point clouds in the future 3 seconds based on the designated\nfuture ego-vehicle pose. In other words,\ngiven historical images in 3 seconds and corresponding history ego-vehicle pose information (from -2.5s to 0s, 6 frames under 2 Hz),\nthe participants are required to forecast future point clouds\nin 3 seconds (from 0.5s to 3s, 6 frames under 2Hz) with specified future ego-poses.\n\nAll output point clouds should be aligned to the LiDAR coordinates of the ego-vehicle in the `n` timestamp, which spans a\nrange of 1 to 6 given predicting 6 future frames.\n\nWe then evaluate the predicted future point clouds by querying rays. We will provide a set of query rays for testing propose,\nand `the participants are required to estimate depth along each ray for rendering point clouds. An example of submission \nwill be provided soon.` Our evaluation toolkit will render\npoint clouds according to ray directions and provided depths by participants, and compute chamfer distance for points within\nthe range from -51.2m to 51.2m on the X- and Y-axis as the criterion.\n\nFor more details, please refer to [ViDAR](https://github.com/OpenDriveLab/ViDAR).\n\n### Evaluation: Chamfer Distance \u003ca name=\"worldmodel-eval\"\u003e\u003c/a\u003e\nChamfer Distance is used for measuring the similarity of two point sets, which represent shapes or outlines of two scenens.\nIt compares the similarity between predicted and ground-truth shapes by calculating the average nearest-neighbor distance between\npoints in one set to points in the other set, and vice versa.\n\nFor this challenge, we will compare chamfer distance between predicted point clouds and ground-truth point clouds for points\nwithin the range of -51.2m to 51.2m. Participants are required to provide depths of specified ray directions. Our evaluation\nsystem will render point clouds by ray directions and provided depth for chamfer distance evaluation.\n\n### Submission \u003ca name=\"worldmodel-submission\"\u003e\u003c/a\u003e\nThe evaluation server at [Hugging Face](https://huggingface.co/spaces/AGC2024-P/predictive-world-model-2024) will be open around `late March`!\n\n## Dataset: OpenScene \u003ca name=\"dataset\"\u003e\u003c/a\u003e\n\n\u003cdiv id=\"top\" align=\"center\"\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/OpenScene.gif\" width=\"900px\" \u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\n\u003e - [Medium Blog](https://medium.com/@opendrivelab/introducing-openscene-the-largest-benchmark-for-occupancy-prediction-in-autonomous-driving-74cfc5bbe7b6) | [Zhihu](https://zhuanlan.zhihu.com/p/647953862) (in Chinese)\n\u003e - Point of contact: [contact@opendrivelab.com](mailto:contact@opendrivelab.com)\n\n### Description\nOpenScene is a compact redistribution of the large-scale [nuPlan](https://www.nuscenes.org/nuplan#challenge) dataset, retaining only relevant annotations and sensor data at 2Hz. This reduces the dataset size by a factor of \u003e10. We cover a wide span of over **120 hours**, and provide additional **occupancy labels** collected in various cities, from `Boston`, `Pittsburgh`, `Las Vegas` to `Singapore`.\n\nThe stats of the dataset are summarized [here](docs/dataset_stats.md).\n\n\u003ccenter\u003e\n  \n|  Dataset  | Original Database |      Sensor Data (hr)    |   Flow | Semantic Categories                               |\n|:---------:|:-----------------:|:--------------------:|:------:|:--------------------------------------------:|\n| [MonoScene](https://github.com/astra-vision/MonoScene)  |      NYUv2 / SemanticKITTI     | 5 / 6  |  :x:     | 10 / 19   |\n| [Occ3D](https://github.com/Tsinghua-MARS-Lab/Occ3D)   |      nuScenes / Waymo    | 5.5 / 5.7 |  :x:    | 16 / 14 |\n| [Occupancy-for-nuScenes](https://github.com/FANG-MING/occupancy-for-nuscenes)   |      nuScenes     | 5.5  |  :x:     | 16  |\n| [SurroundOcc](https://github.com/weiyithu/SurroundOcc)   |      nuScenes     | 5.5  |   :x:    | 16  |\n| [OpenOccupancy](https://github.com/JeffWang987/OpenOccupancy)   |      nuScenes     | 5.5  |  :x:     | 16  |\n| [SSCBench](https://github.com/ai4ce/SSCBench)   |      KITTI-360 / nuScenes / Waymo     | 1.8 / 4.7 / 5.6  |   :x:     | 19 / 16 / 14  |\n| [OccNet](https://github.com/OpenDriveLab/OccNet)   |      nuScenes     | 5.5  |   :x:     | 16   |\n| **OpenScene** |       nuPlan      | **:boom: 120**  |   **:heavy_check_mark:**    | **`TODO`** |\n\n\u003c/center\u003e\n\n\u003e - The time span of LiDAR frames accumulated for each occupancy annotation is **20** seconds.\n\u003e - Flow: the annotation of motion direction and velocity for each occupancy grid.\n\u003e - `TODO`: Full semantic labels of grids would be released in future version\n\n### Getting Started\n- [Download Data](/docs/getting_started.md#download-data)\n- [Prepare Dataset](/docs/getting_started.md#prepare-dataset)\n\n## License and Citation \u003ca name=\"license-and-citation\"\u003e\u003c/a\u003e\n\u003e Our dataset is based on the [nuPlan Dataset](https://www.nuscenes.org/nuplan) and therefore we distribute the data under [Creative Commons Attribution-NonCommercial-ShareAlike](https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode) license and [nuPlan Dataset License Agreement for Non-Commercial Use](https://www.nuscenes.org/terms-of-use). You are free to share and adapt the data, but have to give appropriate credit and may not use the work for commercial purposes.\nAll code within this repository is under [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).\n\nPlease consider citing our paper if the project helps your research with the following BibTex:\n\n```bibtex\n@article{yang2023vidar,\n  title={Visual Point Cloud Forecasting enables Scalable Autonomous Driving},\n  author={Yang, Zetong and Chen, Li and Sun, Yanan and Li, Hongyang},\n  journal={arXiv preprint arXiv:2312.17655},\n  year={2023}\n}\n\n@misc{openscene2023,\n      title = {OpenScene: The Largest Up-to-Date 3D Occupancy Prediction Benchmark in Autonomous Driving},\n      author = {OpenScene Contributors},\n      howpublished={\\url{https://github.com/OpenDriveLab/OpenScene}},\n      year = {2023}\n}\n\n@article{sima2023_occnet,\n      title={Scene as Occupancy}, \n      author={Chonghao Sima and Wenwen Tong and Tai Wang and Li Chen and Silei Wu and Hanming Deng  and Yi Gu and Lewei Lu and Ping Luo and Dahua Lin and Hongyang Li},\n      year={2023},\n      eprint={2306.02851},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n## Related Resources  \u003ca name=\"resources\"\u003e\u003c/a\u003e\n[![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\n- [DriveAGI](https://github.com/OpenDriveLab/DriveAGI)  | [OpenLane-V2](https://github.com/OpenDriveLab/OpenLane-V2) | [DriveLM](https://github.com/OpenDriveLab/DriveLM)\n- [Survey on Bird's-eye-view Perception](https://github.com/OpenDriveLab/BEVPerception-Survey-Recipe) | [BEVFormer](https://github.com/fundamentalvision/BEVFormer) |  [OccNet](https://github.com/OpenDriveLab/OccNet)\n\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenDriveLab%2FOpenScene","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FOpenDriveLab%2FOpenScene","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenDriveLab%2FOpenScene/lists"}