https://github.com/cea-list/tri3d
A unified interface to various 3D driving datasets
https://github.com/cea-list/tri3d
ai autonomous-driving data-science datasets geometry kitti-dataset machine-learning nuscenes waymo-open-dataset
Last synced: about 1 month ago
JSON representation
A unified interface to various 3D driving datasets
- Host: GitHub
- URL: https://github.com/cea-list/tri3d
- Owner: CEA-LIST
- License: other
- Created: 2024-10-17T09:26:00.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2024-10-30T17:10:59.000Z (12 months ago)
- Last Synced: 2024-10-30T18:20:49.534Z (12 months ago)
- Topics: ai, autonomous-driving, data-science, datasets, geometry, kitti-dataset, machine-learning, nuscenes, waymo-open-dataset
- Language: Python
- Homepage: https://cea-list.github.io/tri3d/
- Size: 9.46 MB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# Introduction
Tri3D facilitates the utilization of 3D driving datasets by providing:
- A common coordinate and data encoding convention.
- A common API to read data form various datasets.
- Fast non-sequential access to any sensor sample at any frame index.
- A convenient access to geometric transformations between sensors and
timestamps.
- Plotting utilities.
As of now, Tri3D supports the following datasets:
- [NuScenes](https://www.nuscenes.org/nuscenes)
- [ONCE Dataset](https://once-for-auto-driving.github.io)
- [Semantic KITTI](https://semantic-kitti.org)
- [Waymo open dataset](https://waymo.com/open)
- [Zenseact Open Dataset](https://zod.zenseact.com/frames)
## Conventions
The following conventions are adopted across all datasets :
- An object **position** designates the center of its bounding box.
- In the **local coordinate system of an object**, the x axis points
forward, y leftward, and z updward.
- **Length, width and height** are the dimensions of the object along
x, y and z axes respectively.
- In **camera sensor coordinates**, x points rightward, y points
downward, z point forward.
- In **image coordinates**, x is the pixel column index starting from
the left, y is the pixel row index starting from the top, z is the
depth in meters along the optical axis.
- In **lidar coordinates**, x points in the same direction as the ego
car, y points leftward, z points upward.
The differences with raw dataset conventions are documented in each
datasets class.
Tri3D accounts for timestamps, delays and the movement of the ego car
which carries the sensors. For examples, if the boxes of a given dataset
are annotated at the timestamps of the LiDAR, retrieving the boxes
relative to a camera sensor
(ex: `boxes(seq=0, frame=5, sensor="cam")`) will interpolate the box trajectories at the timestamp of
frame 5 for the camera, and it will also return the object poses
relatively to the position of that camera at this timestamp.
## Dataset API
All datasets implement a common interface which provides access to data
samples such as:
- Sensor frames.
- Acquisition timestamps.
- Sensor poses.
- Camera images.
- Lidar point clouds.
- 3D box annotations.
Moreover, a powerful `.alignment()` function can compute the geometric transformation between any pair of frame and sensors coordinates.
The [tutorial notebook](docs/source/example.ipynb) goes through most of these functions.
Tri3D datasets are somewhat low-level, ie. they do not enforce the
notion of sweep or keyframe where a sample of each sensor around a
timestamp is assembled in a tuple. Instead, datasets expose all
available samples of each sensor indexed by a per-sensor frame index.
Notably, some sensors may work at a higher frequency and contain more
samples than others for a given recording.
When available, keyframes and timestamps are exposed so that coherent
tuples of samples can be rebuilt easily. Moreover, geometric
transformations and pose interpolation functions are provided to
facilitate the creation of new keyframes.
## Geometric transformations
Tri3D provides a small library which facilitates the creation and
manipulation of typical 3D geometric transformations: translation,
rotation, affine, camera projections.
Transformations have a shared interface which supports:
- **Batching**: list of transformations can be grouped together, and
broadcasting rules are supported.
- **Application** to 3D points: applying the transformations to
points, again, boadcasting rules are implemented.
- **Composition**: it is possible to chain transformation together.
- **Inversion**: the inverse of a transformation is readily available.
## Object and sensor poses
In Tri3D, the position and orientation of objects (or sensors) in a
coordinate system are stored as geometric transformation. For instance,
the pose of a camera in a lidar coordinate system is formulated as the
composition of the camera rotation and translation relative to the
lidar. Coincidently, this is also the geometric transformation which
takes points in the camera coordinate system and returns them in the
lidar coordinate system.
For sensors, the sensor poses are accessible via the `.poses()` method.
For object annotations, it is provided by the `Box.transform` attribute.
For example, if we have the pose of a box in lidar coordinates and the
pose of the lidar in camera coordinates, then the position of a point 10
meters in front of that box is given by:
```py
seq = 0
frame = 4
# retrieve poses
box = dataset.boxes(seq, frame, coords="lidar")[0]
box2lidar = box.transform
lidar2cam = dataset.poses(seq, sensor="camera", coords="lidar")[frame]
# compute the position of a point 10m in front of the box, in camera coordinates.
xyz = (lidar2cam @ box2lidar).apply([10., 0, 0])
```