https://github.com/debanjan06/volumetric-ptv3

Edge-optimized 3D semantic segmentation pipeline for airborne LiDAR point clouds (DALES dataset) utilizing a 128-channel Deep Residual Network, 8-bit Quantization-Aware Training (QAT), and optimized static ONNX graph compilation.
https://github.com/debanjan06/volumetric-ptv3

3d-point-cloud airborne-lidar-data computer-vision deep-learning edge-ai feature-engineering geospatial-data model-quantization onnx pytorch quantization-aware-training residual-networks semantic-segmentation

Last synced: about 1 month ago
JSON representation

Host: GitHub
URL: https://github.com/debanjan06/volumetric-ptv3
Owner: debanjan06
License: mit
Created: 2026-05-17T09:00:42.000Z (2 months ago)
Default Branch: main
Last Pushed: 2026-05-31T13:37:27.000Z (about 2 months ago)
Last Synced: 2026-05-31T15:14:53.633Z (about 2 months ago)
Topics: 3d-point-cloud, airborne-lidar-data, computer-vision, deep-learning, edge-ai, feature-engineering, geospatial-data, model-quantization, onnx, pytorch, quantization-aware-training, residual-networks, semantic-segmentation
Language: Python
Homepage:
Size: 20.3 MB
Stars: 0
Watchers: 0
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

<<<<<<< Updated upstream
# Volumetric-PTv3: Edge-Optimized 3D Semantic Segmentation Pipeline

This repository contains an end-to-end deep learning and model optimization pipeline engineered for large-scale semantic segmentation of airborne LiDAR point clouds using the DALES (Dayton Annotated LiDAR Earth Scan) dataset. The project demonstrates a complete production-level machine learning workflow: transforming raw spatial coordinates into context-aware 14D feature spaces, training a 128-channel Deep Residual Network under strict 8-bit Quantization-Aware Training (QAT) constraints, and compiling the dynamic execution graph into a platform-agnostic static ONNX binary optimized for edge NPU deployment.

## Interactive 3D Urban Segmentation Validation

The pipeline features an automated scene reconstruction engine that ingests dense, out-of-sample validation blocks, streams low-latency inference, and stitches discrete coordinate tiles into a contiguous physical environment.

[![Click to Open Interactive 3D Canvas](https://github.com/debanjan06/volumetric-ptv3/blob/main/demo/urban_scene.png?raw=true)](https://htmlpreview.github.io/?https://github.com/debanjan06/volumetric-ptv3/blob/main/demo/final_complete_urban_scene.html)

*Click the preview image above to launch the live, interactive 3D visualization canvas. Once the browser page opens, you can click and drag to rotate, pan, zoom, and turn the urban point cloud upside down directly in your browser window.*

### Semantic Label Classification Key
* Ground and Asphalt Streets: Class 1
* Vegetation and Tree Canopies: Class 2
* Residential Buildings and Roof Layouts: Class 5

---

## Technical Pipeline Optimization Matrix

* **Total Dataset Volume:** ~15 GB raw airborne LiDAR data / 505+ Million spatial points (Dayton urban canopy scan)
* **Evaluation Scale:** 2,883,584 dense point vectors across 11 out-of-sample test assets

---

## Core Engineering Implementations

### 1. Spatial Bottleneck Resolution via 14D Context Vectors
Processing sparse 3D coordinates point-by-point strips away critical topological relationships, leading to representation collapse. Furthermore, file-isolated scaling shifts relative coordinate boundaries arbitrarily from tile to tile, causing severe out-of-sample overfitting.
* **Global Standardization:** Re-engineered data ingestion to scale points across the global boundaries of the entire dataset, maintaining strict coordinate uniformity across independent files.
* **Neighborhood Aggregation:** Integrated an optimized CPU data-loader step using `scipy.spatial.cKDTree`. For every point, the engine calculates structural context across its $k$-nearest neighbors ($k=8$), expanding the raw inputs into a context-aware **14D spatial vector** capturing local roughness, multi-scale height distributions, and density variations.

### 2. Architectural Capacity Expansion via Deep Residual Blocks
To extract complex geometric classification boundaries from the 14D input vectors, the model architecture was upgraded from a shallow linear setup to a **128-channel Deep Residual Network with explicit skip connections**.
* **Gradient Preservation:** Stacking `ResidualQATBlock` modules allows backpropagation signals to travel directly through identity shortcuts, preventing gradient degradation across deeper layers.
* **Loss Optimization:** Driven by standard, unweighted Cross-Entropy Loss over 464 training batches, the network dropped its structural error floor from an initial 2.66 down to a highly stable convergence limit of **0.32495**.

### 3. Low-Precision Edge Optimization via 8-Bit QAT
To satisfy real-world deployment constraints on edge devices with limited power and computational budgets, low-precision constraints were implemented directly into the training loop.
* **Precision Constraints:** Embedded continuous fake-quantization operators into the layer graphs to simulate 8-bit integer truncation bounds `[-128, 127]` during the backward pass. This enables the model weights to adapt to low-precision rounding noise, minimizing precision drop during final serialization.

### 4. Production-Grade Graph Compilation via ONNX
To cross the bridge from dynamic Python frameworks to fixed hardware execution, the system features an automated static serialization compiler.
* **Operator Fusion:** Traces model parameters using `torch.onnx.export` (Opset 17), freezing weights, calculating constant operator folding routines, and permanently fusing Batch Normalization layers into linear scaling operations to maximize hardware processing speed.
* **Dynamic Dimensions:** Configured dynamic axis tracking across the point stream payload (`input_point_stream`), enabling the output binary to natively scale its execution graph to handle varying point-cloud densities per block during real-world data collection.

---

## Repository Structural Mapping

```text
volumetric-ptv3/
├── config/
│ └── architecture.yaml # Hyperparameter and network layout configurations
├── src/
│ ├── preprocess.py # Initial point cloud cleaning and preparation utilities
│ ├── data_prep.py # Global dataset standardization & k-NN neighborhood processing
│ ├── layers.py # Custom quantization-aware layer wrappers and components
│ ├── train.py # 128-channel Deep Residual model definition & training engine
│ ├── evaluate.py # Independent validation pass & dense mIoU calculation
│ ├── export.py # Internal integer-scaling and latency benchmarking harness
│ ├── onnx_export.py # Static graph serialization, folding, and compilation
│ └── demo_visualizer.py # Plotly 3D HTML asset generation engine
├── dist/
│ └── volumetric_ptv3_production.onnx # Compiled platform-agnostic production model graph
├── demo/
│ └── final_complete_urban_scene.html # Fully stitched interactive 3D neighborhood visualization
├── requirements.txt # Python environment baseline dependencies
├── .gitignore # Version control tracking exclusions
├── LICENSE # Open-source distribution parameters
└── README.md # Production documentation and benchmarking metrics
```

---

## Standalone Execution Sequence

### Environment Dependencies Installation

```bash
pip install -r requirements.txt
```

### 1. Ingest Data and Launch Training Pipeline

```bash
python src/train.py
```

### 2. Verify Performance Independently via Checkpoint Loading

```bash
python src/evaluate.py
```

### 3. Compile Optimized Network to Production ONNX Graph

```bash
python src/onnx_export.py
```
## Hardware Profiling & Edge Deployment Telemetry

To validate deployment readiness for intelligent infrastructure applications, the static ONNX graph (Opset 16) was compiled and profiled on physical **Qualcomm Cloud AI 100** hardware via the Qualcomm Developer Network Sandbox.

### Hardware Execution Profiles

### Layer-by-Layer Execution Insights
The proprietary platform compiler (`qaic-compiler`) successfully achieved 100% hardware layer mapping. The 128-channel linear projections and hard-coded residual shortcuts executed natively within the internal hardware vector processing units, avoiding any slow fallback operations to CPU emulation.
## Qualcomm Cloud AI 100 / DragonWing Deployment Readiness
The architecture has been completely optimized and exported for hardware-accelerated edge runtimes. By isolating the network from dynamic tracing dependencies, the graph is structured for seamless compilation into a QAIC Program Binary (.qpc).
- **Target Platform:** Qualcomm Cloud AI 100 Ultra / Standard
- **Graph Format:** Static ONNX Opset 16 (Frozen 1x8192x14 Matrix)
- **Optimization Matrix:** Native 128-Channel Deep Residual Shortcuts with 8-bit Quantization-Aware Training (QAT) simulation rules.

## Dataset Metrics & Reproducibility Pipeline

### 1. Source Material & Ingestion Bounds
The architecture is verified using the **Dayton Aerial LiDAR Elevation Subsets (DALES)** dataset, an open-access, heavily benchmarked 3D semantic segmentation resource collected via airborne laser swaths.
* **Licensing:** Creative Commons Attribution 4.0 International (CC BY 4.0).
* **Ingestion Process:** 1. Register and request raw asset access via the official University of Dayton DALES repository portal.
2. Download the geographic tile collections (`.las` format strings).
3. Structure the asset arrays locally matching the path trees: `data/raw/train/` and `data/raw/test/`.

### 2. Experiment Tracking & Logging Core
The training execution pipeline uses unified metrics hooks to stream model convergence parameters. To review the live learning rate graphs, gradient distributions, and step-wise loss minimization trajectories, access the project tracking dashboard:
* **Logging Integration:** Weights & Biases (W&B) Framework APIs.
* **Core Monitored Targets:** Cross-Entropy Training Loss Floor, Per-Class Intersection over Union (IoU), and Step-Wise Hardware Memory Footprint Scaling.

### 3. Model Ablation Analysis
To confirm the absolute performance impact of each structural addition, the network went through systematic optimization phases under frozen hyperparameter constraints:

---

## Community Asset Hub (HuggingFace Integration)

The final trained weights, parameter blocks, and 8-bit simulation configurations for the 128-channel Deep Residual network iteration are hosted publicly on the HuggingFace Model Hub for immediate download, testing, and deployment integration.

* **Repository Handle:** `debanjan06/volumetric-ptv3-qat-8bit`
* **Asset Payload:** Fully tracked model weights checkpoint (`volumetric_ptv3_qat_8bit.pth`) and optimized compiled graph layer configurations.
### Compute Profile Metrics
- **Total Operational Runtime:** 15 Hours (Full training run + multi-stage spatial validation sweeps)
- **Optimization Overhead:** Includes active INT8 tensor range calibration constraints.

## References & Literature Baseline

Our architecture translates core breakthroughs from the following foundational works into an edge-optimized deployment pipeline:

1. **Point Transformer V3:** Utilized for sparse spatial feature extraction blocks across irregular 3D coordinate distributions.
2. **Deep Residual Learning (ResNet):** Shortcut mapping Topologies adapted to sustain stable gradient flow across the 128-channel dense feature layers.
3. **Quantization-Aware Training (QAT):** Implemented simulated integer-arithmetic optimization loops to guarantee low-latency edge deployment viability.
=======
# Volumetric-PTv3: Edge-Optimized 3D Semantic Segmentation Pipeline