https://github.com/thu-lyj-lab/ss3dm-exporter
[NeurIPS 2024] Data exporter for SS3DM: Benchmarking Street-View Surface Reconstruction with a Synthetic 3D Mesh Dataset
https://github.com/thu-lyj-lab/ss3dm-exporter
autonomous-driving carla-simulator streetview surface-reconstruction
Last synced: 2 months ago
JSON representation
[NeurIPS 2024] Data exporter for SS3DM: Benchmarking Street-View Surface Reconstruction with a Synthetic 3D Mesh Dataset
- Host: GitHub
- URL: https://github.com/thu-lyj-lab/ss3dm-exporter
- Owner: THU-LYJ-Lab
- Created: 2023-06-20T02:47:40.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-11-08T06:54:42.000Z (about 1 year ago)
- Last Synced: 2024-11-08T07:35:40.787Z (about 1 year ago)
- Topics: autonomous-driving, carla-simulator, streetview, surface-reconstruction
- Language: Python
- Homepage: https://ss3dm.top/
- Size: 702 KB
- Stars: 7
- Watchers: 0
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## SS3DM Exporter Codebase
[Our paper](https://arxiv.org/abs/2410.21739) describes our data export based on CARLA. For more information, see the official CARLA documentation: [Official Documentation](https://carla.readthedocs.io/en/latest/).
[[`HomePage`](https://ss3dm.top/)][`Paper`](https://arxiv.org/abs/2410.21739)][`Data Download`: [`GoogleDrive`](https://drive.google.com/drive/folders/1-0xeAUYR6PicWK9eWmslVii3jGtyhL8Z) [`HuggingFace`](https://huggingface.co/datasets/SS3DM/SS3DM-Dataset/tree/main)]
> SS3DM: Benchmarking Street-View Surface Reconstruction with a Synthetic 3D Mesh Dataset
[Yubin Hu*](https://github.com/AlbertHuyb), Kairui Wen*, Heng Zhou, [Xiaoyang Guo](https://xy-guo.github.io/), Yong-Jin Liu
NeurIPS 2024 Track on Dataset and Benchmark
## News
🌟 [2024/11/07] We've uploaded the data to [GoogleDrive](https://drive.google.com/drive/folders/1-0xeAUYR6PicWK9eWmslVii3jGtyhL8Z) and [HuggingFace](https://huggingface.co/SS3DM)!
🥳 [2024/09/26] Our [paper](https://arxiv.org/abs/2410.21739) is accepted by NeurIPS 2024 Track on Dataset and Benchmark.
## Quick Start
1. Install and configure CARLA. For installing CARLA's Python library, refer to the [official documentation](https://carla.readthedocs.io/en/latest/). We recommend using Python 3.7.
2. Install the prerequisite libraries used in the code.
```bash
pip3 install -r requirements.txt
```
3. Open the CARLA simulator, choose one of the cities, load it, and then click "Run" in the top right corner. Here, we will use Town02 as an example.
4. In the `path.txt` of our codebase, select a path (you can also define your own path), and modify `export/export_car_manager.py` with the `route_points` and `transform`. Set `route_points` to the coordinates of all points along the path, and set `transform` to the coordinates and rotation corresponding to the starting point. For example, if we choose the `Town_02 300frames` path, the modified code snippet should look like this:
```python
route_points = [
carla.Location(x=4610.736328 / 100, y=28746.634766 / 100, z=22.363468 / 100),
carla.Location(x=4589.283203 / 100, y=20432.091797 / 100, z=22.353996 / 100)
]
transform = carla.Transform(carla.Location(x=4610.736328 / 100, y=28746.634766 / 100, z=22.363468 / 100), carla.Rotation(pitch=0.025818, yaw=-89.975494, roll=0.000184))
```
5. Modify the `frame_count` in `export/export.py` to the desired number of frames, which in this example is 300 frames. This variable is located at line 114 of the code.
6. Run `export/export.py` and wait for the export to complete. The data will be exported to the directory where `export.py` is located.
7. (Optional) If you need to convert the data to Streetsurf format, copy `parse2streetsurf.py` from `data_processing` to the exported data's `data/` directory and run it to convert the data format.
## Default Organization of Exported Data
By default, the exported data in `data/` will contain 8 folders, each storing the following types of data:
- The `depths` folder contains depth data captured by the depth camera, stored in `exr` format. For specific parameters of the depth camera, please refer to our paper. Each frame's data is aligned with the RGB camera.
- The `images` folder contains image data captured by the RGB camera.
- The `insseg` folder contains instance segmentation data from the instance annotation camera, where each color in the image represents a different entity. The same entity will have the same color across different frames from different cameras. Each frame's data is aligned with the RGB camera.
- The `lidars` folder contains point cloud data from the LiDAR, stored in `npz` format. For specific parameters, please refer to our paper or the `lidar2ply.py` script.
- The `poses` folder contains the positional information for each frame's camera, including the rotation matrix from Camera 5 (front camera) to the world coordinate system and the rotation matrices from different cameras to the front camera.
- The `semseg` folder contains image data from the semantic segmentation camera; refer to the CARLA official documentation for the semantic meanings corresponding to colors. Each frame's data is aligned with the RGB camera.
- The `vehicles` and `walkers` folders contain information about other vehicles and pedestrians. This section of the information is not yet fully processed and will be supplemented in future updates.
## Explanation of Code Functionality
- ### Coordinate System Conversion
Our dataset may involve multiple coordinate systems; therefore, we provide coordinate transformation scripts located in the `coord_transform` folder.
`fbx2obj_axis.py` is used to convert the coordinate system of the fbx files exported from CARLA (after being converted to `obj` files) to the world coordinate system we use.
- ### Data Export
The scripts related to data export are located in the `export` folder.
We provide `path.txt`, which records all routes included in our dataset (Note: the frame count may not be accurate, as we made minor adjustments based on the route length);
We provide `export.py` for exporting data. When running it, please ensure that you have correctly configured CARLA, and it will export the data to the `data/` folder in the same directory.
If you need to modify the route and frame count in `export.py`, please edit `export_car_manager.py` and change `frame_count` in `export.py`.
We provide `fbx2obj.py`, located in the `misc` folder. This script converts the `fbx` files exported from CARLA to `obj` files. Please note that CARLA does not export materials when exporting `fbx` files, so you will need to handle the materials to some extent.
- ### Data Processing
To facilitate the processing of the dataset, we provide data processing scripts located in the `misc` folder.
`parse2streetsurf.py` is used to convert data exported from CARLA into training data formatted like Streetsurf.
Currently, it has two modes: when `convert_flag` is `False`, place it in the `data` folder and run it to convert the data in the current folder;
When `convert_flag` is `True`, it can recursively convert all data in the current subfolder. Please note that we have not debugged this functionality, so we do not recommend enabling this option.
- ### Data Visualization
We also provide data visualization scripts to help visualize some of the data, located in the `misc` folder.
`lidar2ply.py` is used to convert point clouds stored in `npz` format in our dataset to `ply` format for easier viewing;
`semseg2png.py` is used to convert instance segmentation data stored in `npz` format in our dataset to `png` format for visualization.