{"id":26671126,"url":"https://github.com/scaleapi/pandaset-devkit","last_synced_at":"2025-04-09T19:16:57.751Z","repository":{"id":37224387,"uuid":"191463061","full_name":"scaleapi/pandaset-devkit","owner":"scaleapi","description":null,"archived":false,"fork":false,"pushed_at":"2023-08-14T22:14:06.000Z","size":21288,"stargazers_count":263,"open_issues_count":55,"forks_count":69,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-09T19:16:52.690Z","etag":null,"topics":["cuboid-annotations","devkit","hacktoberfest","lidar","lidar-point-clouds","sensor-data"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scaleapi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-06-11T23:16:52.000Z","updated_at":"2025-03-26T01:16:10.000Z","dependencies_parsed_at":"2023-01-25T08:16:30.931Z","dependency_job_id":"b4c745fe-3d84-412b-8f17-d60ae94b67ec","html_url":"https://github.com/scaleapi/pandaset-devkit","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scaleapi%2Fpandaset-devkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scaleapi%2Fpandaset-devkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scaleapi%2Fpandaset-devkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scaleapi%2Fpandaset-devkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scaleapi","download_url":"https://codeload.github.com/scaleapi/pandaset-devkit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248094988,"owners_count":21046770,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cuboid-annotations","devkit","hacktoberfest","lidar","lidar-point-clouds","sensor-data"],"created_at":"2025-03-25T23:43:55.972Z","updated_at":"2025-04-09T19:16:57.697Z","avatar_url":"https://github.com/scaleapi.png","language":"Jupyter Notebook","readme":"# pandaset-devkit\n\n![Header Animation](../assets/animations/semseg-photo-labels.gif)\n\n\n## Overview\n\nWelcome to the repository of the [PandaSet](https://pandaset.org/ \"Pandaset Official Website\") Devkit.\n\n## Dataset\n### Download\n\nTo download the dataset, please visit the official [PandaSet](https://pandaset.org/ \"Pandaset Official Website\") webpage and sign up through the form.\nYou will then be forwarded to a page with download links to the raw data and annotations.\n\n### Unpack\n\nUnpack the archive into any directory on your hard disk. The path will be referenced in usage of `pandaset-devkit` later, and does not have to be in the same directory as your scripts.\n\n### Structure\n\n#### Files \u0026 Folders\n\n```text\n.\n├── LICENSE.txt\n├── annotations\n│   ├── cuboids\n│   │   ├── 00.pkl.gz\n│   │   .\n│   │   .\n│   │   .\n│   │   └── 79.pkl.gz\n│   └── semseg  // Semantic Segmentation is available for specific scenes\n│       ├── 00.pkl.gz\n│       .\n│       .\n│       .\n│       ├── 79.pkl.gz\n│       └── classes.json\n├── camera\n│   ├── back_camera\n│   │   ├── 00.jpg\n│   │   .\n│   │   .\n│   │   .\n│   │   ├── 79.jpg\n│   │   ├── intrinsics.json\n│   │   ├── poses.json\n│   │   └── timestamps.json\n│   ├── front_camera\n│   │   └── ...\n│   ├── front_left_camera\n│   │   └── ...\n│   ├── front_right_camera\n│   │   └── ...\n│   ├── left_camera\n│   │   └── ...\n│   └── right_camera\n│       └── ...\n├── lidar\n│   ├── 00.pkl.gz\n│   .\n│   .\n│   .\n│   ├── 79.pkl.gz\n│   ├── poses.json\n│   └── timestamps.json\n└── meta\n    ├── gps.json\n    └── timestamps.json\n```\n\n## Instructions\n\n### Setup\n\n1. Create a Python\u003e=3.6 environment with `pip` installed.\n2. Clone the repository `git clone git@github.com:scaleapi/pandaset-devkit.git`\n3. `cd` into `pandaset-devkit/python`\n4. Execute `pip install .`\n\nThe `pandaset-devkit` is now installed in your Python\u003e=3.6 environment and can be used.\n\n### Usage\n\nTo get familiar with the API you can point directly to the downloaded dataset.\n\n#### Initialization\nFirst, we need to create a `DataSet` object that searches for sequences.\n```\n\u003e\u003e\u003e from pandaset import DataSet\n\u003e\u003e\u003e dataset = DataSet('/data/pandaset')\n```\nAfterwards we can list all the sequence IDs that have been found in the data folder.\n```\n\u003e\u003e\u003e print(dataset.sequences())\n['002',...]\n```\n\nSince semantic segmentation annotations are not always available for scenes, we can filter to get only scenes that have both semantic segmentation as well as cuboid annotations.\n```\n\u003e\u003e\u003e print(dataset.sequences(with_semseg=True))\n['002',...]\n```\n\nNow, we access a specific sequence by choosing its key from the previously returned list, in this case sequence ID `'002'`\n```\n\u003e\u003e\u003e seq002 = dataset['002']\n```\n\n\n\nAPI Reference: [DataSet class](https://scaleapi.github.io/pandaset-devkit/dataset.html#pandaset.dataset.DataSet)\n\n#### Loading\nThe devkit will automatically search the sequence directory for available sensor data, metadata and annotations and prepare the directory to be loaded explicitly. At this point no point clouds or images have been loaded into memory.\nTo execute the loading of sensor data and metadata into memory, we simply call the `load()` method on the sequence object. This will load all available sensor data and metadata. \n```\n\u003e\u003e\u003e seq002.load()\n```\n\nIf only certain data is required for analysis, there are more specific methods available, which can also be chained to each other.\n```\n\u003e\u003e\u003e seq002.load_lidar().load_cuboids()\n```\n\nAPI Reference: [Sequence class](https://scaleapi.github.io/pandaset-devkit/sequence.html#pandaset.sequence.Sequence)\n\n#### Data Access\n\n##### LiDAR\nThe LiDAR point clouds are stored as [pandas.DataFrames](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas.DataFrame) and therefore you are able to leverage their extensive API for data manipulation. This includes the simple return as a [numpy.ndarray](https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.html).\n```\n\u003e\u003e\u003e pc0 = seq002.lidar[0]\n\u003e\u003e\u003e print(pc0)\n                 x           y         z     i             t  d\nindex                                                          \n0       -75.131138  -79.331690  3.511804   7.0  1.557540e+09  0\n1      -112.588306 -118.666002  1.423499  31.0  1.557540e+09  0\n2       -42.085902  -44.384891  0.593491   7.0  1.557540e+09  0\n3       -27.329435  -28.795053 -0.403781   0.0  1.557540e+09  0\n4        -6.196208   -6.621082  1.130009   3.0  1.557540e+09  0\n            ...         ...       ...   ...           ... ..\n166763   27.670526   17.159726  3.778677  25.0  1.557540e+09  1\n166764   27.703935   17.114063  3.780626  27.0  1.557540e+09  1\n166765   27.560664   16.955518  3.767948  18.0  1.557540e+09  1\n166766   27.384433   16.783824  3.752670  22.0  1.557540e+09  1\n166767   27.228821   16.626038  3.739154  20.0  1.557540e+09  1\n[166768 rows x 6 columns]\n```\n```\n\u003e\u003e\u003e pc0_np = seq002.lidar[0].values  # Returns the first LiDAR frame in the sequence as an numpy ndarray\n\u003e\u003e\u003e print(pc0_np)\n[[-7.51311379e+01 -7.93316897e+01  3.51180427e+00  7.00000000e+00\n   1.55753996e+09  0.00000000e+00]\n [-1.12588306e+02 -1.18666002e+02  1.42349938e+00  3.10000000e+01\n   1.55753996e+09  0.00000000e+00]\n [-4.20859017e+01 -4.43848908e+01  5.93490847e-01  7.00000000e+00\n   1.55753996e+09  0.00000000e+00]\n ...\n [ 2.75606640e+01  1.69555183e+01  3.76794770e+00  1.80000000e+01\n   1.55753996e+09  1.00000000e+00]\n [ 2.73844334e+01  1.67838237e+01  3.75266969e+00  2.20000000e+01\n   1.55753996e+09  1.00000000e+00]\n [ 2.72288210e+01  1.66260378e+01  3.73915448e+00  2.00000000e+01\n   1.55753996e+09  1.00000000e+00]]\n```\n\nThe LiDAR points are stored in a world coordinate system; therefore it is not required to transform them using the vehicle's pose graph. This allows you to query all LiDAR frames in the sequence or a certain sampling rate and simply visualize them using your preferred library.\n\nInstead of using always all of the point clouds available, it is also possible to simply slice the `lidar` property as one is used from python lists.\n```\n\u003e\u003e\u003e pc_all = seq002.lidar[:]  # Returns all LiDAR frames from the sequence\n```\n```\n\u003e\u003e\u003e pc_sampled = seq002.lidar[::2]  # Returns every second LiDAR frame from the sequence\n```\n\nIn addition to the LiDAR points, the `lidar` property also holds the sensor pose (`lidar.poses`) in world coordinate system and timestamp (`lidar.timestamps`) for every LiDAR frame recorded. Both objects can be sliced in the same way as the `lidar` property holding the point clouds.\n```\n\u003e\u003e\u003e sl = slice(None, None, 5)  # Equivalent to [::5]  # Extract every fifth frame including sensor pose and timestamps\n\u003e\u003e\u003e lidar_obj = seq002.lidar\n\u003e\u003e\u003e pcs = lidar_obj[sl]\n\u003e\u003e\u003e poses = lidar_obj.poses[sl]\n\u003e\u003e\u003e timestamps = lidar_obj.timestamps[sl]\n\u003e\u003e\u003e print( len(pcs) == len(poses) == len(timestamps) )\nTrue\n```\n\nThe LiDAR point clouds include by default the points from both the mechanical 360° LiDAR and the front-facing LiDAR. To select only one of the sensors, the `set_sensor` method is available.\n```\n\u003e\u003e\u003e pc0 = s002.lidar[0]\n\u003e\u003e\u003e print(pc0.shape)\n(166768, 6)\n\u003e\u003e\u003e s002.lidar.set_sensor(0)  # set to include only mechanical 360° LiDAR\n\u003e\u003e\u003e pc0_sensor0 = s002.lidar[0]\n\u003e\u003e\u003e print(pc0_sensor0.shape)\n(106169, 6)\n\u003e\u003e\u003e s002.lidar.set_sensor(1)  # set to include only front-facing LiDAR\n\u003e\u003e\u003e pc0_sensor1 = s002.lidar[0]\n\u003e\u003e\u003e print(pc0_sensor1.shape)\n(60599, 6)\n```\nSince the applied filter operation leaves the original row index intact for each point (relevant for joining with `SemanticSegmentation`), one can easily test that no point was left out in filtering:\n```\n\u003e\u003e\u003e import pandas as pd\n\u003e\u003e\u003e pc0_concat = pd.concat([pc0_sensor0, pc0_sensor1])\n\u003e\u003e\u003e print(pc0_concat.shape)\n(166768, 6)\n\u003e\u003e\u003e print(pc0 == pc0_concat)\n           x     y     z     i     t     d\nindex                                     \n0       True  True  True  True  True  True\n1       True  True  True  True  True  True\n2       True  True  True  True  True  True\n3       True  True  True  True  True  True\n4       True  True  True  True  True  True\n      ...   ...   ...   ...   ...   ...\n166763  True  True  True  True  True  True\n166764  True  True  True  True  True  True\n166765  True  True  True  True  True  True\n166766  True  True  True  True  True  True\n166767  True  True  True  True  True  True\n[166768 rows x 6 columns]\n\u003e\u003e\u003e print((~(pc0 == pc0_concat)).sum())  # Counts the number of cells with `False` value, i.e., the ones where original point cloud and concatenated filtered point cloud differentiate\nx    0\ny    0\nz    0\ni    0\nt    0\nd    0\ndtype: int64\n```\n\nAPI Reference: [Lidar class](https://scaleapi.github.io/pandaset-devkit/sensors.html#pandaset.sensors.Lidar)\n\n##### Cameras\nSince the recording vehicle was equipped with multiple cameras, first we need to list which cameras have been used to record the sequence.\n```\n\u003e\u003e\u003e print(seq002.camera.keys())\n['front_camera', 'left_camera', 'back_camera', 'right_camera', 'front_left_camera', 'front_right_camera']\n```\nThe camera count and names should be equal for all sequences.\n\nEach camera name has its recordings loaded as [Pillow Image](https://pillow.readthedocs.io/en/stable/reference/Image.html) object, and can be accessed via normal list slicing. In the following example, we select the first image from the front camera and display it using the Pillow library in Python.\n```\n\u003e\u003e\u003e front_camera = seq002.camera['front_camera']\n\u003e\u003e\u003e img0 = front_camera[0]\n\u003e\u003e\u003e img0.show()\n```\nAfterwards the extensive Pillow Image API can be used for image manipulation, conversion or export.\n\nSimilar to the `Lidar` object, each `Camera` object has properties that hold the camera pose (`camera.poses`) and timestamp (`camera.timestamps`) for every recorded frame, as well as the camera intrinsics (`camera.intrinsics`).\nAgain, the objects can be sliced the same way as the `Camera` object:\n\n```\n\u003e\u003e\u003e sl = slice(None, None, 5)  # Equivalent to [::5]\n\u003e\u003e\u003e camera_obj = seq002.camera['front_camera']\n\u003e\u003e\u003e pcs = camera_obj[sl]\n\u003e\u003e\u003e poses = camera_obj.poses[sl]\n\u003e\u003e\u003e timestamps = camera_obj.timestamps[sl]\n\u003e\u003e\u003e intrinsics = camera_obj.intrinsics \n```\n\nAPI Reference: [Camera class](https://scaleapi.github.io/pandaset-devkit/sensors.html#pandaset.sensors.Camera)\n\n#### Meta\nIn addition to the sensor data, the loaded dataset also contains the following meta information:\n* GPS Positions\n* Timestamps\n\nThese can be directly accessed through the known list slicing operations, and read in their dict format. The following example shows how to get the GPS coordinates of the vehicle on the first frame.\n```\n\u003e\u003e\u003e pose0 = seq002.gps[0]\n\u003e\u003e\u003e print(pose0['lat'])\n37.776089291519924\n\u003e\u003e\u003e print(pose0['long'])\n-122.39931707791749\n```\n\nAPI Reference: [GPS class](https://scaleapi.github.io/pandaset-devkit/meta.html#pandaset.meta.GPS)\n\nAPI Reference: [Timestamps class](https://scaleapi.github.io/pandaset-devkit/meta.html#pandaset.meta.Timestamps)\n\n#### Annotations\n\n##### Cuboids\nThe LiDAR Cuboid annotations are also stored inside the sequence object as a [pandas.DataFrames](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas.DataFrame) for each timestamp.\nThe position coordinates (`position.x`,`position.y`,`position.z`) are located at the center of a cuboid. `dimensions.x` is the width of the cuboid from left to right, `dimensions.y` is the length of the cuboid from front to back and `dimensions.z` is the height of the cuboid from top to bottom.\n\n```\n\u003e\u003e\u003e cuboids0 = seq002.cuboids[0]  # Returns the cuboid annotations for the first LiDAR frame in the sequence\n\u003e\u003e\u003e print(cuboids0.columns)\nIndex(['uuid', 'label', 'yaw', 'stationary', 'camera_used', 'position.x',\n       'position.y', 'position.z', 'dimensions.x', 'dimensions.y',\n       'dimensions.z', 'attributes.object_motion', 'cuboids.sibling_id',\n       'cuboids.sensor_id', 'attributes.rider_status',\n       'attributes.pedestrian_behavior', 'attributes.pedestrian_age'],\n      dtype='object')\n```\n\nAPI Reference: [Cuboids class](https://scaleapi.github.io/pandaset-devkit/annotations.html#pandaset.annotations.Cuboids)\n\n##### Semantic Segmentation\nAnalogous to the cuboid annotations, the Semantic Segmentation can be accessed using the `semseg` property on the sequence object. The index of each Semantic Segmentation data frame corresponds to the index of each LiDAR point cloud data frame, and can be joined using the index.\n```\n\u003e\u003e\u003e semseg0 = seq002.semseg[0]  # Returns the semantic segmentation for the first LiDAR frame in the sequence\n\u003e\u003e\u003e print(semseg0.columns)\nIndex(['class'], dtype='object')\n\u003e\u003e\u003e print(seq002.semseg.classes)\n{'1': 'Smoke', '2': 'Exhaust', '3': 'Spray or rain', '4': 'Reflection', '5': 'Vegetation', '6': 'Ground', '7': 'Road', '8': 'Lane Line Marking', '9': 'Stop Line Marking', '10': 'Other Road Marking', '11': 'Sidewalk', '12': 'Driveway', '13': 'Car', '14': 'Pickup Truck', '15': 'Medium-sized Truck', '16': 'Semi-truck', '17': 'Towed Object', '18': 'Motorcycle', '19': 'Other Vehicle - Construction Vehicle', '20': 'Other Vehicle - Uncommon', '21': 'Other Vehicle - Pedicab', '22': 'Emergency Vehicle', '23': 'Bus', '24': 'Personal Mobility Device', '25': 'Motorized Scooter', '26': 'Bicycle', '27': 'Train', '28': 'Trolley', '29': 'Tram / Subway', '30': 'Pedestrian', '31': 'Pedestrian with Object', '32': 'Animals - Bird', '33': 'Animals - Other', '34': 'Pylons', '35': 'Road Barriers', '36': 'Signs', '37': 'Cones', '38': 'Construction Signs', '39': 'Temporary Construction Barriers', '40': 'Rolling Containers', '41': 'Building', '42': 'Other Static Object'}\n```\n\nAPI Reference: [SemanticSegmentation class](https://scaleapi.github.io/pandaset-devkit/annotations.html#pandaset.annotations.SemanticSegmentation)\n\n\n\n![Header Animation](../assets/static/montage-semseg-projection.jpg)","funding_links":[],"categories":["Datasets"],"sub_categories":["Sensor and Acuator Interfaces"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscaleapi%2Fpandaset-devkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscaleapi%2Fpandaset-devkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscaleapi%2Fpandaset-devkit/lists"}