https://github.com/stepankonev/StandardE2E
A framework for unified end-to-end autonomous driving datasets processing
https://github.com/stepankonev/StandardE2E
agroverse argoverse2 autonomous-driving autonomous-vehicles comma2k19 datasets end-to-end-autonomous-driving multi-modal-sensor-data navsim nuscenes pytorch self-driving-car truckdrive view-of-delft waymo waymo-open-dataset wayve world-model
Last synced: about 1 month ago
JSON representation
A framework for unified end-to-end autonomous driving datasets processing
- Host: GitHub
- URL: https://github.com/stepankonev/StandardE2E
- Owner: stepankonev
- License: mit
- Created: 2025-12-27T18:48:33.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2026-06-15T12:49:49.000Z (about 2 months ago)
- Last Synced: 2026-06-15T13:24:18.126Z (about 2 months ago)
- Topics: agroverse, argoverse2, autonomous-driving, autonomous-vehicles, comma2k19, datasets, end-to-end-autonomous-driving, multi-modal-sensor-data, navsim, nuscenes, pytorch, self-driving-car, truckdrive, view-of-delft, waymo, waymo-open-dataset, wayve, world-model
- Language: Jupyter Notebook
- Homepage: https://arxiv.org/abs/2606.04271
- Size: 2.09 MB
- Stars: 22
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Citation: CITATION.cff
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
- awesome-end-to-end-autonomous-driving - StandardE2E: A Framework for Unified End-to-End Autonomous Driving Datasets Processing
README
A framework for unified end-to-end autonomous driving datasets processing
[](https://github.com/stepankonev/StandardE2E/actions/workflows/tests.yml)
[](https://codecov.io/github/stepankonev/StandardE2E)
[](https://github.com/psf/black)
[](http://mypy-lang.org/)
[](https://github.com/stepankonev/StandardE2E)
[](https://standarde2e.readthedocs.io/en/latest/)
[](https://discord.gg/vJnQNcQGQ8)
StandardE2E provides a consistent interface for preprocessing, loading, and training with multimodal data from various end-to-end autonomous driving datasets. It standardizes the complex process of working with different dataset formats, allowing researchers to focus on model development rather than data engineering.
---
## ๐ Documentation
- Latest docs: https://standarde2e.readthedocs.io/en/latest/
## ๐ฆ Installation
### Option 1: From PyPI (Recommended for Users)
```bash
pip install standard-e2e
```
### Option 2: Development with uv (recommended)
```bash
# Install uv: https://docs.astral.sh/uv/
git clone https://github.com/stepankonev/StandardE2E.git
cd StandardE2E
uv sync --all-extras # installs deps and dev deps from uv.lock
uv run pytest tests/ # run tests
```
### Option 3: Manual development (pip/conda)
```bash
conda create -n standard_e2e python=3.12
conda activate standard_e2e
pip install -e ".[dev]"
```
## Plan for E2E Autonomous Driving Datasets Support
| Dataset | Cameras | Lidar | HD Map | Detections | Driving Command | Preference Trajectories |
|---------|---------|-------|--------|------------|-----------------|-------------------------|
| [Waymo End-to-end](https://waymo.com/open/data/e2e/)  |  | โ | โ | โ | โ
| โ
|
| [Waymo Perception](https://waymo.com/open/data/perception/)  |  | โ
| โ
| โ
| โ | โ |
| [Navsim](https://github.com/autonomousvision/navsim/blob/main/docs/splits.md)  |  | โ
| โ
| โ
| โ
| โ |
| [WayveScenes101](https://wayve.ai/science/wayvescenes101)  |  | ยน | โ | โ | โ | โ |
| [Argoverse 2 Sensor](https://www.argoverse.org/av2.html#sensor-link)  |  | โ
| โ
| โ
| โ | โ |
| [Argoverse 2 Lidar](https://www.argoverse.org/av2.html#lidar-link)  | โ | โ
| โ
| โ | โ | โ |
| [comma2k19](https://github.com/commaai/comma2k19)  | ยฒ | โ | โ | โ | โ | โ |
| [TruckDrive](https://torc-ai.github.io/TruckDrive/)  | ยณ | โ
| โ | โ
| โ | โ |
| [View-of-Delft](https://intelligent-vehicles.org/datasets/view-of-delft/)  | โด | โ
| โ | โ
| โ | โ |
| [nuScenes](https://www.nuscenes.org/nuscenes)  | โต | โ
| โ
| โ
| โ | โ |
| [Argoverse 2 Map Change](https://www.argoverse.org/av2.html#mapchange-link)  | โ
| โ
| โ
| โ | โ | โ |
| [KITTI](https://www.cvlibs.net/datasets/kitti/)  |  | โ
| โ | โ | โ | โ |
| [KITTI-360](https://www.cvlibs.net/datasets/kitti-360/)  |  + 2 x  | โ
| โ | โ | โ | โ |
| [Waymo Motion Prediction](https://waymo.com/open/data/motion/)  | โ | โ
| โ
| โ
| โ | โ |
| [Argoverse 2 Motion Forecasting [?]](https://www.argoverse.org/av2.html#forecasting-link)  | โ | โ | โ
| โ
| โ | โ |
ยน WayveScenes101 ships no sensor lidar; the `lidar_pc` modality is populated from the per-scene **COLMAP SfM** point cloud, transformed into each frame's ego frame and range-clipped so it flows through the lidar adapters. It is photogrammetric (sparse, up-to-scale), not a sensor measurement. Cameras are 5 fisheye views (forward + side arc); the ego past/future trajectory is derived from the COLMAP poses. Scenes must be extracted from the distributed `scene_.zip` archives before processing.
ยฒ comma2k19 is a single forward-facing 20 Hz camera (comma EON, 1164ร874, pinhole) with a fused GNSS/IMU ego pose; the ego past/future trajectory is derived from the global ECEF poses and `global_position` additionally carries the ego **speed**. It ships no lidar, HD map, 3D boxes, or driving command. Segments must be extracted from the distributed `Chunk_*.zip` archives before processing (as with WayveScenes101); each `video.hevc` is then decoded frame-by-frame, forward-only. `global_position` is expressed in a per-segment local frame (ECEF-axis-aligned, origin at the segment start), so absolute X/Y/Z and heading are segment-relative; the ego-relative past/future trajectories are unaffected. Native rate is 20 Hz โ use `--frame_stride` to subsample and bound output volume.
ยณ TruckDrive is a long-range highway **heavy-truck** dataset (Torc Robotics / Princeton, CVPR 2026). Its 8 MP surround rig has **11 cameras** โ more than the eight canonical `CameraDirection` slots โ so each camera is mapped to the canonical surround member matching its facing (`FRONT`, `FRONT_LEFT`, `SIDE_LEFT`, `REAR_LEFT`, โฆ) wherever one fits, with dedicated members added only for the genuinely-extra views the eight slots can't name (the forward telephoto pair `FRONT_{LEFT,RIGHT}_NARROW` and the rear-facing side pair `SIDE_{LEFT,RIGHT}_BACK`). `lidar_pc` is the seven-sensor **Aeva FMCW** joint cloud (xyz kept, in the ego frame); `detections_3d` are the tracked 3D boxes in the ego frame, with the ego vehicle's own cab/trailer and `DontCare` groups excluded per the paper taxonomy. The ego pose (PPK + LiDAR-SLAM) drives the past/future trajectory. Short-range Ouster lidar, 4D radar, accumulated GT depth and lane lines have no StandardE2E target yet and are not ingested. Frames are matched across sensors by their synchronization key. **The dataset ships as per-scene, per-modality zips and must be extracted first** โ use [`scripts/extract_truckdrive.sh`](scripts/extract_truckdrive.sh), or [`scripts/prepare_dataset_truckdrive.sh`](scripts/prepare_dataset_truckdrive.sh) to extract and preprocess in one step.
โด View-of-Delft (VoD) (TU Delft, IEEE RA-L 2022) is a compact **urban** dataset whose distinctive sensor is a **3+1D radar**. StandardE2E ingests its single **front camera** (`CameraDirection.FRONT`, 1936ร1216 pinhole), the 64-layer **Velodyne** `lidar_pc` (xyz in the ego/`velodyne` frame; per-point reflectance dropped) and KITTI-format `detections_3d` mapped from camera coordinates into the ego frame โ each of VoD's 13 classes folded into the coarse `DetectionType` taxonomy (the two-wheeler family `bicycle`/`Cyclist`/`rider`/`moped_scooter`/`motor` โ `BICYCLE`; `Car`/`truck`/`vehicle_other` โ `VEHICLE`; static/ambiguous boxes โ `UNKNOWN`; `DontCare` dropped). Box yaw is VoD's KITTI rotation about the LiDAR **โZ** axis (camera-x zero-reference), so the ego-frame heading is `โ(rotation + ฯ/2)`. One **frame = one keyframe**; one **segment = one recording scene** (`delft_*`), grouped via the official scene table so the per-segment past/future ego trajectory (from the per-frame `mapToCamera` pose) never spans two recordings. The **3+1D radar** (`radar` / `radar_3frames` / `radar_5frames`) has no StandardE2E modality yet and is not ingested; per-frame **timestamps are synthesised** at the 10 Hz LiDAR-lead rate (the detection release ships none); the **test** split has sensor data but no labels (so no detections). **The dataset ships as zips and must be extracted first** โ use [`scripts/extract_vod.sh`](scripts/extract_vod.sh) (lidar tree only, with an optional track-id overlay), or [`scripts/prepare_dataset_vod.sh`](scripts/prepare_dataset_vod.sh) to extract and preprocess in one step.
โต nuScenes (Motional, CVPR 2020) is the de-facto surround-view E2E / BEV benchmark: 1000 ~20 s scenes, a **6-camera** surround rig (1600ร900), a 32-beam `LIDAR_TOP` and densely annotated 3D boxes at 2 Hz keyframes (one **frame = one keyframe sample**, one **segment = one scene**). The six `CAM_*` channels map onto the canonical `CameraDirection` members; `lidar_pc` is the `LIDAR_TOP` cloud (xyz, in the ego frame); `detections_3d` are the `sample_annotation` boxes transformed from the global frame into the ego frame, each `category_name` folded into the coarse `DetectionType` taxonomy (nuScenes doesn't annotate the ego vehicle, so nothing is excluded). The ego pose drives the past/future trajectory. **HD map**: the vector **map-expansion** (lane centers recovered from the arcline paths, lane/road dividers, crossings, walkways, stop lines, drivable area, intersections) is translated to the unified `MapElementType` taxonomy in the ego frame and rasterised to BEV by `HDMapBEVAdapter`, **when** the separate `nuScenes-map-expansion-v1.3` pack is unzipped into `/maps/` (absent from the base download; the HD map is skipped if it isn't there). `--split` is an official nuScenes label that also selects the metadata version (`mini_train`/`mini_val` โ `v1.0-mini`, `train`/`val` โ `v1.0-trainval`, `test` โ `v1.0-test`); the **test** split ships no annotations. nuScenes is read **directly from the JSON tables** โ the `nuscenes-devkit` is *not* a runtime dependency (it pins `numpy<2`, conflicting with this project's numpy 2.x), so the split scene-lists and the lane-arcline discretization are vendored from it (Apache-2.0). The 5 radars have no StandardE2E target yet. A **partially-downloaded** trainval converts cleanly โ scenes whose sensor blob isn't on disk yet are skipped. The release ships as `.tgz` archives and **must be extracted first** โ use [`scripts/extract_nuscenes.sh`](scripts/extract_nuscenes.sh), or [`scripts/prepare_dataset_nuscenes.sh`](scripts/prepare_dataset_nuscenes.sh) to extract and preprocess in one step.
## ๐ Key Features
- **Unified Dataset Interface**: Work with multiple datasets through a single API
- **Multimodal Support**: Cameras, LiDAR (point cloud + BEV histogram), HD maps (BEV raster), trajectories, detections and more
- **Flexible Preprocessing**: Configurable pipelines with standardization and augmentation
- **Lazy modality loading**: Configured adapters declare what they consume; no decoding work is done for modalities no adapter reads
- **Trajectory Management**: Advanced handling of time-series vehicle data
- **PyTorch Integration**: Ready-to-use datasets and dataloaders
## ๐ Quick Start & Examples
### Notebooks
- [intro_tutorial.ipynb](notebooks/intro_tutorial.ipynb) - Introduction to StandardE2E framework
- [containers.ipynb](notebooks/containers.ipynb) - Working with data containers
- [multi_dataset_training_and_filtering.ipynb](notebooks/multi_dataset_training_and_filtering.ipynb) - Multi-dataset training and filtering
- [creating_custom_adapter.ipynb](notebooks/creating_custom_adapter.ipynb) - Creating custom dataset adapters
### Code Examples
Run from the project root so `uv run` uses the project environment. If you use pip/conda instead, activate your env and use `python` in place of `uv run python`.
1. **Preprocess Waymo End-to-end dataset** - Convert raw dataset to standardized format ([`dataset_preprocessing.py`](examples/dataset_preprocessing.py))
```bash
uv run python examples/dataset_preprocessing.py \
--e2e_dataset_path E2E_DATASET_PATH \
--split {training,val,test} \
--processed_data_path PROCESSED_DATA_PATH
```
2. **Train your model** - End-to-end training with multimodal data ([`very_simple_training.py`](examples/very_simple_training.py)). This example illustrates iteration over the preprocessed dataset. Also, in this example for validation we use 2 DataLoaders - full validation split and filtered validation split that only contains samples with preferred trajectories.
```bash
uv run python examples/very_simple_training.py --processed_data_path PROCESSED_DATA_PATH
```
3. **Create a unified DataLoader**: This example shows how to process 2 different datasets within same DataLoader. First, please do preprocessing for `Waymo E2E` and `Waymo Perception` datasets in order to utilize them in the DataLoader with the script ([`prepare_datasets_waymo_e2e_perception.sh`](scripts/prepare_datasets_waymo_e2e_perception.sh)).
The script [`creating_unified_dataloader.py`](examples/creating_unified_dataloader.py) created a unified dataloader that iterates over both `Waymo E2E` and `Waymo Perception` in one epoch providing consistent data structure.
```bash
uv run python examples/creating_unified_dataloader.py --processed_data_path PROCESSED_DATA_PATH
```
4. **Add a new dataset adapter** - Guide for adding support for new datasets ([`adding_new_dataset.md`](standard_e2e/caching/src_datasets/adding_new_dataset.md))
## ๐ License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## ๐ Citation
If you find this project useful, you can support it by giving it a โญ, or by contributing with your PRs / issues / feature requests. Also, if you use this project, you can greatly support it by citing our paper ([arXiv:2606.04271](https://arxiv.org/abs/2606.04271)):
```bibtex
@misc{konev2026standarde2e,
title={StandardE2E: A Unified Framework for End-to-End Autonomous Driving Datasets},
author={Stepan Konev},
year={2026},
eprint={2606.04271},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2606.04271}
}
```
and using the badge [](https://github.com/stepankonev/StandardE2E)
Markdown
```markdown
[](https://github.com/stepankonev/StandardE2E)
```
HTML
```html
```